West Virginia Home Visitation Program Evaluation of Professional Development … · 2015-06-01 ·...

West Virginia Home Visitation Program Evaluation of Professional Development and Community Collaboration Prepared for Office of Maternal, Child and Family Health September 2014

Prepared by

Hornby Zeller Associates, Inc. 373 Broadway

South Portland, ME 04106

West Virginia Home Visitation Program Office of Maternal, Child and Family Health U.S. Department of Health and Human Services Maternal, Infant, and Early Childhood Home Visiting Development Grant No. D89MC23160-02-02 from United States Affordable Care Act

Evaluation of Professional Development and Community Collaboration September 2014

Prepared by Hornby Zeller Associates, Inc. 373 Broadway South Portland, ME 04106 (207) 773-9529

Contents Introduction .....................................................................................................................................1

Purpose of This Report ......................................................................................................................5

Evaluation Design and Methodology .................................................................................................7

Methodology for Professional Development ................................................................................ 10

Staff Survey .............................................................................................................................. 10

Interviews ................................................................................................................................ 11

Methodology for Community Collaboration ............................................................................... 13

Community Partner Survey ...................................................................................................... 13

Document Review ................................................................................................................... 14

Key Informant and Staff Interviews .......................................................................................... 14

Results ............................................................................................................................................ 15

Demographics of Home Visiting Staff .......................................................................................... 15

Staff Development Opportunities: Training and Technical Support ............................................. 18

Training Effectiveness and Unmet Needs ..................................................................................... 20

Reflective Supervision Practices .................................................................................................. 24

Effectiveness of Supervision Overall ............................................................................................ 26

Professional Development Scales ................................................................................................. 27

Job Satisfaction and Job Mastery.............................................................................................. 28

Burnout and Intent to Leave .................................................................................................... 30

Effects of Training, Supervision and Time on the Job ............................................................... 31

Community Collaboration: State-Level Efforts ............................................................................. 32

MIECHV Key Stakeholders Team ............................................................................................. 32

Partners in Community Outreach............................................................................................. 33

Regional Collaboration Meetings ............................................................................................. 34

Other State-Level Efforts to Promote Collaboration ................................................................. 35

Community Collaboration: Local-Level Activities ........................................................................ 35

Local Service Continuum ......................................................................................................... 36

Training ................................................................................................................................... 36

Local CQI Efforts Related to Collaboration .............................................................................. 36

Community Agency Views of Collaboration ................................................................................. 41

Summary of Program Strengths ....................................................................................................... 43

Recommendations........................................................................................................................... 45

Appendix A: Community Partner Survey ........................................................................................ 49

Appendix B: Staff Survey Questions ................................................................................................ 57

1 | W e s t V i r g i n i a H o m e V i s i t a t i o n F i n a l E v a l u a t i o n R e p o r t

Introduction

Early childhood home visitation is a no-cost service provided to families that choose to participate. Home visiting is typically offered to families in the prenatal stage or to those who have children age five or under. In 2010, with funds authorized by the Affordable Care Act, the Maternal Infant Early Childhood Home Visitation Program (MIECHV) awarded Development Grants to states seeking to enhance the infrastructure of statewide home visitation programs. The MIECHV program provides the means for collaboration at the federal, state, and community levels to improve health and development outcomes for at-risk children through evidence-based home visiting programs. To be eligible, states were required first to participate in a formal statewide needs assessment1 to determine where services were lacking and where populations were at highest risk for poor health and child development outcomes. Once the needs assessments were completed and approved, states were required to identify their communities with highest need as well as the capacity for home visitation to address federally-identified priorities by either a) selecting one or more home visiting programs from a list of approved evidence-based models, or b) identifying a proposed model that could be considered an innovative or promising approach. While all states following these guidelines received a set amount of funding, those that wanted to improve or expand their programs were invited to apply for either Development Grants or Expansion Grants. The West Virginia Department of Health and Human Resources (DHHR), Office of Maternal, Child and Family Health (OMCFH) is the designated lead agency for home visitation programs and elected to apply for a MIECHV Development Grant which it subsequently received. The focus was to expand the role of professional development and community collaboration strategies to develop and improve the program statewide.

1 West Virginia’s full Statewide Needs Assessment completed September, 2010 is accessible through DHHR’s website at http://www.wvdhhr.org/wvhomevisitation/needs.asp.

http://www.wvdhhr.org/wvhomevisitation/needs.asp


West Virginia’s main goal was to improve the quality of home visiting services to high-risk populations through professional development opportunities, ongoing technical assistance and program monitoring within a cohesive early care and education system.

There are seven home visitation models in West Virginia; at the time of this report, the models are: Early Head Start, Healthy Families America, Healthy Start/HAPI, Maternal Infant Health Outreach Workers, Parents as Teachers, Right From the Start and Save the Children. Every county has at least one program in place. However, only three of those were included on the approved list of evidence-based programs. OMCFH has identified the models that are eligible for MIECHV support:

Early Head Start (EHS)

Healthy Families America (HFA)

Parents as Teachers (PAT) In addition, the Maternal Infant Health Outreach Workers (MIHOW) program is West Virginia’s proposed promising practice model that participated in a randomized control trial in an effort to be recognized as evidence-based. Consequently the federal Development Grant supports all four of these models. Following the MIECHV program objectives, West Virginia’s main goal is to improve the quality of its home visiting services to high-risk populations through professional development opportunities, ongoing technical assistance, and program monitoring within a cohesive early care and education system. By strengthening the state’s early care and education infrastructure, West Virginia hopes to improve services and positively influence child and family outcomes. For this OMCFH project, the state’s efforts were intended to:

Improve and monitor the quality of trainings,

Increase access to trainings,

Develop cross-training opportunities with partners,

Introduce specialized training series, and

Improve staff supervision. West Virginia’s OMCFH expects that direct service staff at contracted agencies will use knowledge gained through trainings in their day-to-day work, stay in their positions longer and report greater satisfaction with their work and supervision.


Community collaboration is viewed as a critical component of providing services to families since West Virginia is a largely rural state without an abundance of resources.

The state plan also prioritized collaborative efforts with the notion that increased knowledge of available services, cross-program referrals and cross-program trainings will enhance programing efficiency and contribute to improving child and family outcomes. The West Virginia Home Visitation Program (WVHVP) has worked to:

Encourage active collaboration and shared learning among multiple models through state and local continuous quality improvement (CQI) activities;

Coordinate trainings for all staff across home visiting models for maternal depression screening, injury prevention, preconception care, child abuse prevention and domestic violence; and

Increase capacity to integrate home visitation services into an early childhood system.

Community collaboration is viewed as a critical component of providing services to families since West Virginia is a largely rural state without an abundance of resources. Programs and providers serving families during the prenatal, infancy or early childhood stages attempt to collaborate and conserve resources and reduce duplication of efforts. This approach has also been shown to be more effective and family-friendly.2

2 James Bell Associates. (2011). Evaluation Brief: Evaluating Inter-organizational Collaborations. Arlington, VA.


Purpose of This Report

The purpose of this report is to present the final results and recommendations from the evaluation of professional development and community collaboration activities under the MIECHV grant. This report is issued by OMCFH’s evaluator, Hornby Zeller Associates, Inc. (HZA). Included here is an overview of the evaluation design and methods used for both parts of the study of 1) professional development, and 2) collaboration. All data collection and related activities described here were performed between November 2012 and the end of May 2014.3 These findings include significant emerging themes and trends identified in the data collected, as well as recommendations for program improvement and possible refinements to the statewide home visitation system. HZA will highlight findings that may be helpful for the state to consider as it prepares for sustainability beyond the funding cycle as well as possible program expansion.

3 A preliminary Evaluation Summary Report was released in May 2013, approximately six months into the first year’s evaluation activities.

Professional Development

Community Collaboration


The statewide evaluation of home visitation services examined West Virginia’s two inter-related areas of focus: professional development and community collaboration.

Evaluation Design and Methodology

The evaluation plan was approved by HRSA November 1, 2012; marking the official start of the first phase data collection effort. From November 2012 through May 2013 the evaluation plan of activities included: in-person site visits and staff interviews; administration, data collection and analysis of the Home Visiting Staff Survey; orientation and distribution of the Group Functioning Scale; collection of minutes and notes kept during Partners in Community Outreach (PiCO)4 meetings; review of Points of Contact forms collected by state epidemiologist; administration of the Community Partner Survey; and participation in quarterly Stakeholder Meetings. The second phase of data collection continued from May of 2013 through May 2014, with activities including: two rounds of in-person site visits and staff interviews; telephone interviews with most of the MIECHV Stakeholders; two rounds of administration, data collection and analysis of the Home Visiting Staff Survey; collection of minutes and notes kept during Partners in Community Outreach (PiCO) meetings; review of points of contact forms collected by state epidemiologist; administration of the Community Partner Survey; and attendance at two statewide Home Visiting Conferences. The statewide evaluation of home visitation services examined West Virginia’s two inter-related areas of focus: professional development and community collaboration. The professional development component was analyzed to determine whether the variables found in the research literature correlating with job satisfaction, burnout and retention in prior studies hold true in West Virginia.

4 Partners in Community Outreach (PiCO), funded by the Claude Worthington Benedum Foundation with in-kind support from OMCFH, is a program of TEAM for WV Children, Inc. and the coalition of HFA, MIHOW and PAT programs in West Virginia. PiCO provides technical support, training and collaboration opportunities for home visiting staff.


The research questions as stated in the State’s Evaluation Plan are as follows.

1. How does participation in West Virginia’s professional development and reflective supervision efforts correlate with the following? a. Job satisfaction b. Burnout c. Intent to leave d. Sense of job mastery

2. Which of the following variables are positively associated with greater job satisfaction, less burnout, reduced

intent to leave, and positive sense of job mastery? a. Demographic characteristics b. Education and experience c. Caseload d. Home visitation model e. Work environment

3. What additional efforts can produce greater job satisfaction, less burnout, reduced intent to leave and

positive sense of job mastery? The community collaboration component looked at how state-level efforts influenced local provider’s work. Examples are coordinated cross-model trainings, development of cross-program standards of practice (Core Competencies), and facilitated Regional Collaboration Meetings where providers across multiple sectors could come together. The research questions are as follows.

1. How have state-level coordination efforts improved program management, efficiencies, service continuum and climate across early childhood programs?

2. How have cross-agency trainings (e.g., domestic violence, maternal depression screening, injury prevention, preconception counseling, child abuse prevention, and SIDS/SUID) been used to foster collaboration?

3. How have the roles and perceptions of partner agencies changed towards home visitation? 4. How do the high-risk counties differ in their collaboration efforts and what factors might account for

differences? The logic model shown on the following page provides an overview of the evaluation design. It should be noted that the state undertook other activities as part of its MIECHV Development Grant; however, with the interest expressed by the OMCFH and the assistance of the technical assistance contractor, HZA focused this particular evaluation on the two initiatives of Professional Development and Community Collaboration.


West Virginia Home Visitation Logic Model, 2012


Because no single instrument was specific to home visitation, nor ideal for gathering the breadth of information needed for this study, a new instrument was developed for West Virginia.

Methodology for Professional Development

To answer the research questions related to professional development, the evaluation used four data collection processes: in-person interviews with direct service and supervisory staff; staff surveys for all levels including an exit survey for those who left their positions; site visits; and a review of documentation relevant to professional development efforts. Together, the data was used to measure:

1. the number of training hours home visiting staff completed, 2. the types of training they attended, 3. the number and types of conferences or professional

meetings staff attended, 4. home visiting staff participation in work-related planning

committees or similar workgroups, 5. the extent and quality of the supervision they received, and 6. home visiting staff satisfaction with their job.

The first five of these show the extent to which DHHR’s activities have resulted in actual staff development opportunities looking back over the course of the project. The last question then sets the stage for measuring the impact of participation in staff development activities on job satisfaction, including analysis of whether that impact is different for home visiting staff in different situations.

Staff Survey HZA researched and considered many instruments currently used in similar workplace environments. Because no single instrument was specific to home visitation, nor ideal for gathering the breadth of information needed for this study, a new instrument was developed for West Virginia. The survey contains targeted sets of questions that were developed after thorough review of other known and tested instruments. For instance, to measure job satisfaction, the research team drew from Paul Spector’s Job Satisfaction Survey (JSS) from 1994. While the full set of questions was not included, the Staff Survey contains 23 of the original 36 questions. A similar process was used in developing the sections addressing burnout, intent to leave and job mastery.


The final instrument has eight sections and 85 items covering topics such as supervision and support, training needs and opportunities, and workplace climate, to name a few. The number of questions varied according to the relevance to the home visiting staff’s position. That is, supervisors could be asked more questions about their management system, decision-making process for hiring and how formal agreements are made, questions that may not be relevant to home visitors. (See Appendix B for the Staff Survey questions.) The survey was administered online at three points in time. Initially, between December 2012 and early January 2013; then between July and August 2013; and again at the end of the project period between January and March 2014. At each data collection point, an email link to the web-based survey was provided to all home visiting staff including supervisors and managers. The survey reportedly took an average of about fifteen minutes to complete. (A full description of the survey can be found in the Home Visitation Evaluation Plan.)

Table 1: Surveys Returned by Evaluation Timeframe

Survey Timeframe Number Percent

1: Dec 2012–Jan 2013 69/102 67%

2: July–Aug 2013 62/106 58%

3: Jan–March 2014 78/107 73%

Average return rate 66%

Descriptive and inferential statistical techniques were used to identify important relationships among the factors tested. Correlation analysis was used to test the relationship of the interventions to job satisfaction, burnout, intent to leave, and job mastery, factoring in the roles of environmental variables such as education and experience of the home visitor. For example, the analysis explored whether the amount and type of training received by a home visitor related to their sense of job mastery. To accomplish this, seven different scales were created which scored and averaged the responses to all the survey questions related to a single indicator. The scales covered the topics of training quality, job satisfaction, intent to leave, burnout, job mastery, supervision type and supervision quality.

Interviews For this evaluation, a total of 108 interviews were conducted with home visitors and supervisors or managers during three separate site visits over the course of the grant.

Table 2: Interviews Conducted by Evaluation Timeframe

Site Visits and Interviews Conducted

Round 1 November 2012 42 interviews

Round 2 July 2013 26 interviews

Round 3 March 2014 40 interviews


Participants were randomly selected from a group of all 113 current staff5 to assure objectivity, though participation was voluntary. Once participation agreements were signed, interviews included in-depth discussion of the following topics: training quantity and quality; type and relevance of training or other professional development activities; sense of competence or mastery; reflective supervision and support; and work with other community agencies or organizations. Interview responses were summarized by topic and coded on a three-point scale. (See criteria displayed in Table 3.)

Table 3: Interview Response Criteria

Interview Topic Criteria for Responses

0 1 2

Training Offered: Quantity

Insufficient opportunities; not able to attend

Aware of opportunities but may not attend

Sufficient opportunities; attended training

Training Offered:

Quality

Poor quality; not helpful or relevant

Somewhat good quality, needs some improvement

High or good quality; helpful

Activities or Efforts Specific to High-Risk Population

Not aware of state’s efforts to address risk factors of population

Awareness of some efforts, needs more attention or work

Aware of efforts in most topics related to risk factors

Job Mastery Training has not helped in job

Training has helped somewhat with sense of competence

Training has helped achieve competence

Reflective Supervision: Practice

Elements of reflective supervision are not used; unaware of what it is

Some elements of reflective supervision used; sometimes used

Most or all elements of reflective supervision always used

Reflective Supervision: Quality

Indicated support was lacking or insufficient

Supervision is somewhat helpful

Supervision is supportive and helpful to staff

Collaboration in High-Risk Areas

Described negative relationships w/ partners; does not work with other state programs

Described neutral or mixed feelings about partners in community; works w/ 1-2 state programs

Affirmed working well w/partners; works with other state programs

5 For most of the evaluation, the program had 113 staff, though at times this number declined slightly when there was turnover or paid-leave for staff.


The evaluation examines how agencies relate to one another, and how the activities and services provided across the agencies changed.

Methodology for Community Collaboration

This part of the project informs West Virginia of the extent to which collaborative efforts are effective in promoting partnerships and improving the prenatal to age five continuum of services. At the state level, the evaluation examines how agencies relate to one another both formally and informally, and the resulting impact on outcomes. For this factor, we looked at West Virginia’s process for establishment or enhancement of: policies and practice guidance; climate and culture; community links; personnel and human resources; information-sharing; and quality improvement in the service continuum for families of children prenatal to five years old. At the community level, the evaluation examines how the activities and services provided across the agencies changed the home visiting agencies along with the collaborative partners. The study looked at changes in formal and informal partnerships; gaps in services for families in the prenatal to five year stage; effectiveness of services to high-risk areas; and how collaborative efforts reduced duplication of effort or otherwise saved resources. This component employs a variety of tools, some of which allowed quantification of results and comparisons in at least two points in time. Both descriptive statistics and qualitative techniques were used for data analysis and to identify important relationships among the factors related to collaboration. Within the ten high-risk counties, each home visiting agency was included for interviews and participation in the Staff Survey. Other data collection efforts described below included: the Community Partner Survey, analysis of point-of-contact data, a review of relevant documentation from Stakeholders Team meetings, and key informant interviews.

Community Partner Survey Home visiting agencies were asked to provide contact information for collaborative partners who were then sent an electronic link to the Community Partner Survey. The research team then cross-referenced those with data from the DHHR-developed Points of Contact Forms,6 completed a simple internet search for each county, and consulted with members of the leadership team who had knowledge of the typical partners across the state. About 200 names were gathered in total, 179 of which had valid contact information or email addresses; these individuals were sent standard email requests and multiple follow-up reminders to complete the Community Partner Survey. (See Appendix A for the questions asked on the Community Partner Survey.) The return rate was 32 percent.

6 Points of Contact Forms are used as part of the Federal MIECHV grant for program-level tracking of collateral contacts, submitted quarterly.


Document Review To supplement the information gathered from staff and stakeholders, the evaluation included a review of Regional Collaboration Meeting minutes and attendance forms, Home Visiting Conference evaluation results, other relevant meeting minutes, agency reports and publications, work plans, committee reports and similar materials. These materials provided a basis for capturing the collaborative process and understanding the context in which decisions were made.

Key Informant and Staff Interviews The West Virginia Home Visitation Stakeholders group and the Early Childhood Advisory Council members were interviewed at the end of the project period, between April and May 2014. A total of 21 individuals out of a possible 28 (75%) completed a telephone interview reflecting on various aspects of the Stakeholders Group. Researchers used a structured, open-ended interview protocol to explore the topics associated with state- and community-level change that were fully defined in the evaluation plan. Directors and program managers of the home visiting agencies in all of the high-risk counties are also considered “key informants” for this study. These individuals were interviewed in person (along with the other staff) over three rounds of site visits and interviews completed November 2012, July 2013 and March 2014. Analysis of all interviews included open and thematic coding completed by trained reviewers. Each interview was transcribed (when they were recorded) by an outside reviewer, and verified for accuracy by the original interviewers. All records were then reviewed and coded by two independent evaluators, using a third evaluator where reconciliation of differences was necessary.


“Home visitors need more training in personal and professional development; basically learning how to be professionals… Something to help them put it all together.”

Results

The information presented in this report is based on all of the activities above, provided data were obtained by June 1, 2014.

Demographics of Home Visiting Staff

The staff survey offered to all managers and direct home visitors yielded 209 combined responses from 21 programs covering 31 counties. At the time of this report, no exit surveys had been received, although about five people have reportedly left or retired from home visiting. Since it was possible for a staff person to complete the survey more than once, duplicates were removed and the most recent survey data were used. The 108 randomly-selected interviews completed between November 2012 and March 2014 were conducted in person at each program’s office location by a team of four HZA research staff. A semi-structured interview allowed program staff to expand on their responses as they thought further about themes related to community collaboration or professional development. Most people responding to the surveys and participating in interviews identified with the Parents as Teachers model, as shown in Table 4.Surveys were also completed by Early Head Start home visitors, though their participation was not required for the final evaluation and are not included in the table.

Table 4: Program Models

Participating Programs Interview Survey

Parents as Teachers (PAT)

61% 55%

Healthy Families America (HFA)

16% 20%

Maternal Infant Health Outreach Workers (MIHOW)

23% 19%


Table 5: Participant Demographics Table 5 shows the demographics of people responding to the survey who represent two-thirds of the home visiting staff in West Virginia. The vast majority were women with a wide range of years of experience in the field of early education, home visitation or similar. Over 92 percent had children of their own and twenty-five percent also had grandchildren, an important factor to home visiting programs such as MIHOW, because they seek to hire staff who have children of their own as one criteria of employment. The typical home visitor in West Virginia is married with children, between the ages of 40 and 50. As shown in Figure 1, more staff members had been in their current position for five years or more (36 %) than any other category. On the other hand, a quarter of staff were fairly new—having been on the job for 12 months or less—followed by twenty percent who have been in their positions between one and two years at the time of the survey. Similarly, most of the respondents identified as home visitors (66%), with 34 percent identifying as supervisors or program managers. Both the survey data and the interview results show a significant number of new home visiting staff across the state, although most participants reported having some kind of relevant experience. All other demographic characteristics remained the same over the course of the evaluation.

Figure 1: Survey Participants: Length of Time in Position

1 year or less, 26%

1-2 yrs, 20%

2-4 yrs, 14%

4-5 yrs, 5%

Over 5 yrs, 36%

Demographics of Survey Participants

Percent

Gender

Female 97%

Male 3%

Relationship Status

Single 10.0%

Divorced 7.5%

Married or Partnering 78.5%

Widowed 3.7%

Have Children?

Yes 92.5%

No 7.5%

Age Group

<30 Years Old 10.5%

31–40 Years Old 28.6%

41–50 Years Old 35.2%

51+ Years Old 25.7%


For many people home visiting is not a full-time job. In fact, slightly less than half of the respondents (43%) said they worked for home visiting full time. An additional 34 percent said they were part time. (The remaining respondents either did not answer the question or said “other” for their work with the home visiting program.) Similar to the results provided in the first-year analysis, all of the home visiting models had fairly solid representation among the three groups: direct service staff, those that have both caseload and management duties and those who are exclusively supervisors. This was consistent across both the interviews and the surveys. However, there was some variation by curriculum model (or program type). As demonstrated in Figure 2, Healthy Families was the most highly represented model among home visitors and supervisors interviewed. Note that no supervisors with caseloads were represented in the Healthy Families model.

Figure 2: Interviewee Roles by Program Model

When examining the education of home visiting staff, as shown in Figure 3 on the following page, almost twice as many staff had some type of Bachelor’s Degree than those with a high school degree. Over one fifth have a related BA (23%, very similar to the first round survey responses), while just under a fifth had an unrelated degree (18%, slightly lower than the first round responses). Consistent with previous survey participants, 13 percent of all staff had a Master’s Degree as the highest level of education. The survey and interview results were consistent.

71%

29%

0%

60%

20% 20%

66%

23%

11% 0%

20%

40%

60%

80%

100%

Home Visitor Supervisor Supervisor w/ Caseload

HFA MIHOW PAT


Figure 3: Staff Education Levels

Staff Development Opportunities: Training and Technical Support

West Virginia DHHR has diligently focused on providing numerous training opportunities covering a range of topics important to In-Home Family Education (IHFE) over the past 18 months. Topics have included: domestic violence, child abuse and neglect, the Adverse Childhood Experiences Study (ACEs), early literacy, social-emotional development, postpartum depression, data collection and documentation, continuous quality improvement, non-profit management, and home visitor safety. Awareness of these topics increased from nine percent to 85 percent from round one interviews to round three (Figure 4). Similarly, the number of interviewees who had been trained in most or all of these areas increased by about 10 percentage points over the same period (from 69% to 80%), also noted in Figure 4.

Figure 4: Staff Efforts to Promote Training in Specific Topics

21%

18%

20%

26%

2%

13% High School

Associate's

Bachelor's (not related)

Bachelor's (related)

Master's (not related)

Master's (related)

9%

58%

85%

69%

35%

80%

0%

20%

40%

60%

80%

100%

Round 1 Round 2 Round 3

Aware of Stateefforts to addresstopics

Trained in most orall topics fromState efforts


“The amount of documentation our parent educators have to do is a challenge. Assessments are important, but the frequency is affecting the quality of the program and their relationships with parents. So in the end, documentation becomes sloppy or unreliable because they just want to get it done for a deadline.”

Home visiting staff said they attended at least one statewide training in the past year; in most cases, these trainings were paid for or subsidized by DHHR and the MIECHV grant. The average number of hours of in-person training this past year was 59, though the actual number of hours per person varied greatly across the state with the range being between three and 200 hours. Staff participated in far fewer hours of web-based training, with an average of 10 hours last year. The overwhelming majority (over 95%) of direct service and supervisory staff stated they were aware of or had attended trainings that were open to other programs or home visiting models beyond those funded by the MIECHV Development Grant. This reflects West Virginia’s efforts to promote resource-sharing as a means of increasing staff capacity to work with high-risk groups, regardless of the location or agency at which staff work. Home visitors and supervisors alike also said that the training topics and the number of opportunities for new staff to attend were a good start to increasing levels of knowledge. Many people stated that the site visits and direct technical support provided by OMCFH on documentation and federal reporting requirements was extremely helpful. However, these comments were often followed by the staff person conceding that he or she was overwhelmed by additional paperwork requirements under the new MIECHV grant. This trend did not change by the end of the evaluation, though OMCFH did secure a vendor to provide an electronic data collection system in effort to improve record-keeping and allow for greater consistency in reporting across home visiting models.


Training Effectiveness and Unmet Needs

Home visiting staff have identified a range of topics where they would like more training. There were 58 written survey responses with a wide range of suggestions for new training topics. The most popular responses have been categorized in Figure 5.

Figure 5: Training Needs as Indicated by Staff Survey

At the start of the project, home visitors indicated a need for intermediate and advanced trainings, with just under half (45%) stating they needed more (as shown in Figure 6). The need for more advanced trainings continued to increase over the evaluation period, and was consistent regardless of whether the staff person was a supervisor (67%) or home visitor (73%). As staff continue in their positions and also continue to serve the highest risk populations, the level of training perceived as necessary has consistently escalated as they gained more knowledge and experience. It could be said, too, that a lack of available resources in rural communities places a greater burden on home visitation staff to know and do more; the unmet needs from a staff’s perspective may not fall to OMCFH to solve. This is a larger issue related to collaboration and availability of resources to serving vulnerable populations.

One staff person stated, “The training for new staff has been great… From a supervisory perspective, I think there is still ample need for the state to provide more advanced training to promote the professionalism of the field.” Home visitors continue to request professional development activities that go beyond basic training or review of previously covered material, although interviewees stated that, “most trainings are basic…we need more advanced training that aren’t so repetitive. Anything that a ‘new’ home visitor would find helpful is no longer applicable to me.” Staff

Mental Health & Infant MH

22%

Program Management

16%

Infant & Child Development

16%

Working with Disabilites

14%

Curricula-related Training

8%

Domestic Violence 8%

Substance Abuse 8%

Assisting with Basic Needs

8%


members at both the direct service and supervisory levels were most interested in opportunities that provide them with useful skills as well as information.

Figure 6: Availability of Intermediate or Advanced Training

Staff development opportunities were provided not only as in-person trainings, but also in alternative formats such as webinars or online sessions. Webinars provided a means to overcome the obstacle of scheduling difficulties and extensive travel time and cost. The majority of interviewees (83%) participated in at least one webinar in the past year. In particular, many respondents said that they looked forward to the “Lunch and Learn” sessions hosted regularly by the state office. There were no comments indicating that technology or internet access were barriers to participation. Interviewees shared that the state office was very responsive to their requests for specific topics and since the sessions were short and relevant to their work, they benefited from calling in. In summary, the webinar sessions that were planned based on home visitors’ needs and interests were well-received and an efficient and economical way of sharing information across the state.

Another focus of the evaluation is to determine whether or not the professional development efforts help staff to meet job requirements. Figure 7 shows the responses to specific topics on which home visitation staff has requested training in the past.

45%

20%

59%

18%

96%

4%

0%

20%

40%

60%

80%

100%

Higher Level Training isNeeded

Higher Level Training isAvailable

Round 1

Round 2

Round 3


Figure 7. Impact of Training on Home Visitors’ Abilities to Work with Special Populations:

The training has prepared me to work with families by learning more about…

The figure shows that home visitors felt most competent in working with families on children’s social and emotional growth and children’s literacy, two areas that were the focus of trainings that were offered. Home visitors felt least prepared by the training they attended for working with families who have low birth weight babies and serving families who have mental illness. Starting at the first phase of the study, and reiterated at the conclusion of the evaluation, staff rarely had overtly negative remarks about training; rather, they described challenges with logistics such as: notice of and timing of training; length of training (a few reported difficulty in getting away for the required 1-2 days); distances involved in traveling to sessions; and disappointment when descriptions did not “match” the content covered. It was common to hear seasoned staff express frustration when presentations were redundant or repetitive. In the last year of the evaluation, supervisors articulated a need for guidance to staff in professional ethics, which was a topic that had not been discussed early on in the project. The OMCFH staff, Early Childhood Advisory Council (ECAC), Partners in Community Outreach (PiCO), Prevent Child Abuse West Virginia and Head Start and Early Head Start Association worked with Collective Impact, LLC to successfully organize and host two statewide Home Visitation Conferences, in May 2013 and then again in May 2014. The 2014 Conference evaluation results were examined to see if attendees, in general had a positive experience and to glean what aspects were most valuable to those who provided feedback after the conference. Just under a quarter of those that attended provided feedback (84 out of 340 participants) via an anonymous two-page form that was then summarized by Collective Impact, LLC. According to this summary, the conference was a good experience (for 47 participants or 59%) and the overall quality of workshops was good (50 participants or 63%) The question which had the highest number of negative responses was Variety of Session Topics; 25 percent of respondents indicated “fair or poor” and when asked “What one thing would most improve the conference?” there were 32 written responses, ten (or 31%) of which included suggestions for greater variety in topics, with a few suggesting an offer of different levels (e.g., beginner/intermediate/advanced).

7.5%

3.3%

1.0%

2.1%

1.1%

1.6%

1.6%

21.4%

17.6%

7.9%

4.2%

6.6%

3.3%

2.6%

15.0%

13.2%

5.2%

4.2%

2.7%

4.4%

3.6%

40.5%

45.6%

50.8%

49.5%

48.1%

46.2%

44.3%

15.6%

20.3%

35.1%

40.0%

41.5%

44.5%

47.9%

0% 20% 40% 60% 80% 100%

Low birthweight babies

Mental illness

Substance use or alcohol abuse

Abuse, neglect, or domestic violence

Children with developmental disabilities

Child development & literacy

Children’s social emotional growth

Strongly Disagree Disagree No Opinion Agree Strongly Agree


The list of suggestions for future topics was consistent with finding from the interviews with staff where they discussed training interests and unmet needs. Figure 8 shows the topics suggested after the conference, grouped into six general types. Two obvious differences were noted between the training needs indicated through the statewide evaluation before the conference, and the training suggestions made following the conference. First, there were no suggestions for mental health-related topics following the conference, although it was the largest category from the statewide survey. In addition, there were no training suggestions related to working with adults with developmental delays or disabilities, even though this was also a recurring response provided in the statewide survey and interviews.

Figure 8: Suggested Training Topics from Home Visiting Conference, 2014

From the statewide evaluation of professional development, the results showed strong correlations between those who had a positive rating of training (i.e., the degree to which they felt training “prepared” them to work with high-risk groups) and job satisfaction. Those who said the training they received was effective also reported high job satisfaction. Figure 9 shows how helpful training was perceived to be based on the staff person’s length of time in his or her position. The survey results show little variation in the length of time a home visitor has been on the job and whether she considered training helpful. Across the board home visitors said training has helped them to do their jobs. In the interviews the more experienced home visitors and supervisors often expressed that trainings were repetitive and even a waste of time. They stated the need for more education and training opportunities related to the troubling and complex issues presented by families seen by home visitors. Numerous staff members expressed the desire to learn more about what to do when presented with situations related to substance use (particularly prescription drug abuse), mental illness, and complex family dynamics (such as with multigenerational households or teen parents). Home visitors were interested in gaining a better understanding of these topics, beyond reviewing the basic information about the subject.

Home Visiting Practices & Curricula

28%

Prenatal & Child Development

20%

Risk Factors & ACEs

20%

Community Resources

12%

Substance Use 10%

Other 10%


Here again, the themes discovered in the beginning of the evaluation about the need for more advanced training on topics such as substance abuse and mental illness hold true during the last phase of the evaluation. However, many of the MIECHV-funded efforts took time to put in place, and the entire project period is relatively short for organizing significant activities statewide.

Figure 9: Perceived Helpfulness of Training Based on Job Tenure

The overwhelming majority of new home visitors found any and all trainings to be helpful. General observations included statements like, “All training is useful because information is power,” and “Training makes it possible to see things from another perspective.” Staff responses also spoke to specific needs: “Trainings really need to provide a skill, not just information. It needs to be hands-on. We want the opportunity to practice skills” and “The DV trainings over the past year have allowed me to better help families and has given me a sense of confidence.” In summary, while numerous opportunities for professional development are being offered and/or paid for by the state, and are perceived as good opportunities for new staff, they will need to be adjusted to address the more advanced and complex needs of more seasoned professionals.

Reflective Supervision Practices

West Virginia requires all managers to complete reflective supervision training as part of the MIECHV Development Grant. The goals of reflection are grounded in a relationship-based approach to supporting staff rather than the once typical top-down or hierarchical model. This philosophy is strength-based, process-oriented, culturally sensitive and often described as a parallel process. The critical elements of reflective supervision include regularly scheduled, in-person meetings between staff and supervisors, which are designed to help home visitors to reflect on their actions when working with families and then begin to apply what they learn in their work in the homes. These sessions also provide an opportunity to discuss concerns and develop corrective plans if needed. Open lines of communication are intended to help to develop strong relationships and retain staff members who become increasingly competent and confident in their roles.7

7 In addition to required training in reflective supervision, the WVHVP adheres to the home visiting model developers’

recommendations and requirements for supervisor qualifications in regard to education, number of home visitors supervised, training length and timing, and format and frequency of observations of home visitors. For instance, the PAT requirement is that staff members working more than part time meet monthly, one-on-one, for a minimum of two hours. PAT also requires a

0% 4%

0% 0% 6%

19% 25%

8%

40%

0%

81%

71%

92%

60%

94%

0%

20%

40%

60%

80%

100%

1 year or less 2-4 years 5-7 years 8-10 years Over 10 years

Has not helped Has somewhat helped Has helped in job


All supervisory respondents had participated in reflective supervision training, compared to a quarter of the home visitors. Nearly 70 percent of supervisors reported that they implement most or all elements of reflective supervision, while just over half (58%) of home visitors indicated during the interview that most or all aspects of reflective supervision are used in their supervision sessions. The use of reflective supervision varied by home visiting model, as noted in Figure 10. Every interviewee from HFA indicated that most or all elements of reflective supervision were used in their agency throughout the entire grant period. By the end of the evaluation, 76 percent of PAT participants used reflective supervision; similarly, 74 percent of MIHOW interviewees indicated reflective supervision was used.

Figure 10: Use of Reflective Supervision by Home Visiting Model

An important element in reflective supervision is the frequency of supervisory sessions themselves, which should be at least monthly. Figure 11 shows that only 12 percent of home visiting staff say they received supervision less than monthly.

Figure 11: Frequency of Supervisory Meetings

minimum of two hours per month for all-staff team meetings (which is in addition to the one-on-one sessions). This is not necessarily the case for HFA or MIHOW.

1% 11%

42%

10%

36%

0%

10%

20%

30%

40%

50%

No Meeting Occasionally Monthly Every 2 Weeks Weekly

0% 0%

100%

16%

32%

53%

14%

33%

54%

0%

20%

40%

60%

80%

100%

Elements are notused

Some elements aresometimes used

Most/all elements arealways used

HFA

MIHOW

PAT


In general, West Virginia’s strong focus on reflective practice across programs has led to an increase in implementing reflective supervision, the method considered to be best practice among the home visiting professional community. While most interviewees said that they had supportive work environments throughout the project, those that indicated it was not supportive decreased from eight percent down to zero between the start and end of the project, as shown in Figure 12.

Figure 12. Supportive Work Environment

Most staff members had the opportunity to discuss specific cases or situations with their supervisors and troubleshoot problems or brainstorm solutions. Not all programs have one-on-one supervision sessions, with 12% meeting only occasionally or not at all; however some supervisors stated they would like to make time for formal and regularly scheduled one-on-one supervision in addition to group sessions. Consistent with the intent of reflective supervision, by the end of the project, all interviewees described very supportive work environments in which their colleagues and supervisors were readily available to help address concerns, either in person or through other means such as email or text messaging.

Effectiveness of Supervision Overall

As shown in Figure 13 below, home visitors rated their supervisors fairly highly, with the overwhelming majority expressing agreement or strong agreement with the ten positively-framed statements. The index created to examine the overall quality of supervision and support as perceived by the home visitors revealed a mean score of 4.43 out of a possible 5. The analysis of supervision, based on both the surveys and interviews, suggests that staff are generally supported and valued by their supervisor. Likewise, the majority reported other positive aspects of their managers, such as being approachable, helpful and knowledgeable about home visiting-related issues; all of which are influential in the home visitors’ sense of job satisfaction.

8% 0%

28% 20%

65%

80%

0%

20%

40%

60%

80%

100%

Start of Project End of Project

Not supportive

Sometimes supportive

Always supportive

“Reflective supervision allows for positive, productive meetings that allow everyone to think outside the box…it allows the supervisee to work through their difficulties without the fear that they will be judged.”


Figure 13: Rating of Direct Supervisors

Moreover, those who rated their supervisors as high quality also reported high levels of job satisfaction. Not surprisingly, a low quality rating of supervision and support was moderately, (but still significantly) related to a greater intent to leave the job altogether. There was no significant relationship between the staff person’s education level or age and his or her overall job satisfaction.

Professional Development Scales

This section explores broader professional development themes, including any significant correlation or connection between and among variables. Information is again derived from the staff survey. The discussion presents categories assessed such as burnout and job satisfaction. Consistent with the overall evaluation design, the same questions were asked at three intervals during the MIECHV Development Grant cycle. This report provides the final assessment of supervisors’ and home visitors’ overall scores in each of the scales. The analysis examines the eight categories to see how they interact with each other, and whether there are statistically significant relationships. The categories included are:

Staff Burnout

Intent to Leave

Job Satisfaction

Job Mastery

Availability of Training

Perceptions of Quality of Training

Quality of Supervision

Use of Reflective Supervision

1%

2%

2%

2%

2%

2%

2%

1%

2%

4%

5%

8%

8%

8%

8%

2%

4%

5%

6%

6%

42%

40%

38%

34%

39%

35%

37%

33%

30%

28%

52%

50%

51%

50%

47%

60%

58%

61%

62%

62%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Is knowledgeable

Helpful with challenges

Communicates clearly

Strengths-based

Encourages reflection

Is approachable

Encourages teamwork

Advocates for staff

Makes me feel respected and…

Is appreciative of skills and needs

My Supervisor...

Disagree No opinion Agree Strongly Agree


Figure 14 displays the average scores for five of the scales (excludes training and reflective supervision). In general, results are positive and show that staff are mostly satisfied with their supervision, fairly confident in their abilities to do their work, and have a relatively low desire to leave. Results do show a large standard deviation which means there was a wide range of responses and not everyone necessarily identifies with these trends. The lowest deviation from the mean was found in Job Satisfaction, meaning that job satisfaction was more uniformly experienced among home visiting staff than any of the other variables.

Figure 14: Average Scores in Professional Development Scales, 2014

Job Satisfaction and Job Mastery The research questions concerning overall satisfaction and home visitor’s perception of job mastery seek to address what additional efforts are needed by the State or home visiting agencies to produce greater satisfaction and mastery as well as decreased burnout, and a reduced intent to leave. By the end of the study period, the job satisfaction scale was very strongly correlated to supervision quality, (the strength of Pearson’s r here was .851), and likewise job mastery was moderately correlated to satisfaction (Pearson’s r value of .623) and moderately correlated to supervision quality (Pearson’s r of .491). If staff perceived training to be useful they had a higher sense of job satisfaction and job mastery. There was no correlation between the number of training hours per se, however, and either job satisfaction or job mastery. In Figure 15, which displays the final responses to job satisfaction questions, we note that overall, most of the respondents (76%) are satisfied with their job “as-is.” This is lower than the first round results, however, where 82 percent had a positive answer. On the other hand, fewer people looked for new jobs and far fewer planned to leave this year when compared to last.

Figure 15: Job Satisfaction and Intent to Leave

2.12 0.84 3.71 3.96 4.43 0

1

2

3

4

5

Burnout Intent to Leave Job Satisfaction Job Mastery SupervisionQuality

16%

1%

30%

93%

76%

84%

99%

70%

7%

24%

0% 20% 40% 60% 80% 100%

Applied or sent resume

Plan to leave

Looked for new job

Plan to stay for 1-2 yrs

Satisfied with job

YES

NO


Figure 16 provides a visual representation of the relationship between Job Satisfaction and the other factors studied. A blue arrow is drawn between two metrics if their correlation is statistically significant at the 0.05 level or better, and a red arrow indicates statistical significance at the 0.001 level. Factors in yellow are not correlated to other factors. Note that the quality of supervision is the most important factor, both because it shows the strongest correlation to any other item, in this instance high job satisfaction, and because it is correlated with the most other items. These include job mastery and the perceived usefulness of training. Also positively correlated, though not as strong, are perceived usefulness of training and job satisfaction as well as strong sense of job mastery and job satisfaction.

Figure 16. Correlates of Job Satisfaction

High Job Satisfaction

Perceived usefulness of training

Strong job mastery

Degree of reflective

supervision

Low staff burnout

Low intent to leave

High quality of

supervision

Strong Correlation

Weak Correlation

No Correlation


Burnout and Intent to Leave When examining burnout and intent to leave, the inverse of the above results are shown. Specifically, there is a strong correlation between perceived low quality supervision and burnout; low job satisfaction and burnout and weak sense of job mastery and burnout. Even low perceptions of training are correlated with burnout. Use of reflective supervision per se was not associated with burnout, just as it was not associated with job satisfaction, above. However, the survey showed that if the program used reflective supervision the way it was intended, staff were less likely to express feelings of emotional drain and to experience negative physical impacts. The relationships are shown in Figure 17.

Overall the burnout factor was low, second only to intent to leave. Based on the scales created to examine these variables, the average level of burnout was 2.1 (on a scale of 1 to 5 with 1 being low); the correlation to the supervision quality rating was lower at final evaluation, as shown by the decline in Pearson’s r value of -.529 to -.415.8 In this case, a lower score is desirable indicating fewer negative tendencies, or, put another way, staff were even less burnt-out and experienced better quality supervision than before. Not surprisingly, a high burnout score was strongly associated with low job satisfaction and low job mastery. The relationship between burnout and job mastery (Pearson’s r = -0.661) is among the strongest of any scales examined. The strong relationship between burnout and low job satisfaction was shown by Pearson’s r -.758, even more significant than the supervision quality score and a stronger correlation than in the first year. That means that people who had low job satisfaction also expressed feeling burned out. Those staff members also expressed a low level of competence to do their job.

Figure 17. Correlates of Staff Burnout

8 Using the Correlation Coefficient Pearson’s r, a strong correlation is between the number -.6 and -1; the closer the value to 1, the stronger

the correlation. A weak correlation is typically between -.3 and -.2

High Staff Burnout

Perceived usefulness of training

Low job mastery

Degree of reflective

supervision

Low job satisfaction

High intent to leave

Low quality of

supervision

Strong Correlation

Weak Correlation

No Correlation


A similar process was used to examine the specifics related to a home visitor’s intent to leave. Staff who had high scores on the burnout scale were significantly more likely to have a high intent to leave score. Overall, responses to the survey revealed a very low average score for the intent to leave index (.84), but the high standard deviation (1.13) indicates that the results are skewed, with some staff members showing a much higher intent to leave.

Shown here are the average scores for five scales examined (excluding those related to training.) In general, results are positive and show that staff are mostly satisfied with their supervision, fairly confident in their abilities to do their work, and have a relatively low desire to leave

Effects of Training, Supervision and Time on the Job While the research questions set out to determine the influence of hours and types of training on job satisfaction, mastery and intent to leave, the number of hours of training was not significantly correlated with any other measure, when looking at both web-based training and in-person training. There were also no significant findings when examining the impact of reflective supervision on any of the indices. The fact that the quality of supervision was significantly correlated with several other factors—while the degree of reflective supervision, specifically, was not—may reflect the importance of other characteristics of supervisors and their relationship with home visitors, not only the particular model used. Both the interviews and the surveys demonstrated that the quality of support and degree of respect provided by a direct supervisor has a strong influence on a staff person’s sense of satisfaction with their work. As shown in Figure 18, the question of intent to leave was examined by the amount of time a home visitor had been on the job and specifically whether she had searched for a new job or sent out a resume in the past year. We found that those who had been on the job between five and ten years were more likely than others to at least be looking. About a quarter of the staff had also sent out a resume. The second most vulnerable period was the first year on the job when over a third of the staff had done some searching. These results are something to consider when examining retention strategies and where to target them.

Figure 18: Job Search Status by Length in Position

Similarly, staff who indicated they were looking for a job and submitting resumes tended to differ by age. Figure 19 shows that the age group most likely to be looking for another job was the 30 year old or under group, followed by the 41 to 50 year olds. There were no other demographic characteristics that correlated to intent to leave, such as marital status, education level or whether the staff person had children of her own.

36% 28%

53% 57%

20% 18% 17% 24%

29%

7%

0%

20%

40%

60%

80%

100%

1 Year or Less 2-4 Years 5-7 Years 8-10 Years > 10 Years

Searched for Job Sent Resume


Figure 19: Job Search Status by Age Group

Community Collaboration: State-Level Efforts

By the conclusion of the project, collaborative efforts had increased and it is clear that collaboration and forming healthy community partnerships continues to be a priority of the state leadership team and local providers alike. While the MIECHV Stakeholders Group did not regularly convene, many of its members also participated in other groups such as Partners in Community Outreach (PiCO), which effectively contributed to strengthening collaborative efforts under the MIECHV Development Grant.

MIECHV Key Stakeholders Team The final year of the evaluation included more in-depth analysis of the state-level collaboration efforts. The MIECHV Stakeholders and Early Childhood Advisory Council (ECAC) had a successful beginning together with the formation of a diverse group representing all aspects of providers serving families of young children in West Virginia. During the first phase, the State established a working plan for a formal Home Visitation Stakeholders Team, although based on interviews conducted with members at the end of year one and again during the final phase of the evaluation, this plan was not implemented as intended. Facilitated meetings were to be held quarterly with full representation from not only all home visitation model providers, but also invested community professionals in partner organizations. Examples of the relevant partner groups are: Prevent Child Abuse West Virginia, the Department of Education, the Head Start Director’s Association, Child Care, and all Early Childhood Advisory Council members. This group was set up initially to influence collaboration positively and inform the state plan for professional development and continuous quality improvement (CQI). It began by bringing members together to share details about each program and mission in connection with the State Home Visitation Program. Those who were available participated in two separate tours of home visiting sites in different regions of the state to learn first-hand about the experiences of service providers. Although Stakeholders convened on a couple of occasions, there was not a regular, predictable schedule, and when meetings did occur, the majority of those interviewed described conflicts with prior existing commitments. Interviewees were not clear about the group’s intended purpose, see Figure 20, and often referenced activities and meetings held either by PiCO or the ECAC; they also shared accomplishments from the Core Competency Workgroup or the Early Childhood Planning Taskforce rather than those of the Stakeholders Team. While these other groups are somewhat similar and in many cases there is overlapping membership, the Stakeholders’ inability to articulate the intent and purpose, combined with general inactivity speaks to the need to revisit the plan for informing the State Home Visitation Program.

44.4% 33.3% 35.1% 32.0%

22.2%

10.0%

18.9% 20.0%

0%

20%

40%

60%

80%

100%

<30 Years Old 30-40 Years Old 41-50 Years Old 51+ Years

Searched for Job Sent Resume


The overlap in membership in other existing groups contributed to a lack of clarity about this group. Just under half of those interviewed (10 out of 21 members, or 48%) could make the distinction and tended to have more hands-on involvement with state-level activities such as working on the Core Competencies, Resource Directories and the Help Me Grow service. Those that did not understand the purpose of the group also said that the group met too infrequently and did not have a consistent agenda or goal; they tended to question if they were “truly involved” and used phrases like, “I feel out of the loop” and “early on I was included, but now I am sort of feeling like I was left off the list.”

Figure 20: Perception of Stakeholder’s Group Purpose Among Members, 2014

Some members of the Stakeholders Team were involved with an extensive review and revision of the Core Competencies for all programs to use as guidance in implementing their programs with fidelity. In partnership with the State Early Childhood Advisory Council, the next phase of this project will include the dissemination and implementation of the Core Competencies.

Partners in Community Outreach The PiCO group has been in place since 1999, and with the MIECHV Development Grant was designated by OMCFH to be the coordinating organization for all grantees, with membership from each of the three home visitation groups (HFA, MIHOW and PAT). It is perhaps the most consistent and solid resource available for home visiting staff, and has provided valuable insight on the connections among the state, the major home visitation programs and efforts of their members. PiCO regularly provides resources for staff development and networking at both the management and direct staff level. While this evaluation does not formally assess the degrees of effectiveness of this group, PiCO plays an important part in developing community collaboration.

Yes 24%

No 38%

Not Sure of Purose

38%

Did the Stakeholders' Group Fulfill its Purpose?


For instance, PiCO has taken action toward providing community linkages to address many of the state’s objectives under the grant including:

Working with and serving on the Early Childhood Advisory Council and Stakeholders Team;

Working with and serving on the Advisory Panel for the Our Babies, Safe and Sound project;

Voting on the West Virginia Campaign to Raise Children Out of Poverty key issues;

Participating in the development of Home Visiting Core Knowledge and Competencies;

Attending and assisting with the state Home Visiting Conference;

Attending the Perinatal and Child Health Summit;

Communicating priorities for home visitation with partners by generating, for example, articles on In-Home Family Education to the Sisters of Saint Joseph Health and Wellness Foundation and the West Virginia Nurses Association; and

Providing consistent and ongoing communications and meeting facilitation for each home visiting program across the state.

Regional Collaboration Meetings With assistance from West Virginia’s OMCFH and Bureau of Children and Families, all home visiting staff had the opportunity to participate in the newly-formed Regional Collaboration Meetings. The initial set of four regional meetings was facilitated by an outside consulting firm at the first statewide Home Visiting Conference in May of 2013.The feedback was extremely positive and helped providers and administrators at all levels better understand the breadth of family services in place across each region. Participants said that they learned what services other providers are able to offer families and the face-to-face meeting allowed them to discuss new referral strategies first-hand. The second set of Regional Meetings took place in March of 2014, with the goal of maintaining a regular schedule that works for each region’s members. Meetings were facilitated by the same team of consultants from the first round, relieving the attendees of the burden of maintaining organization and record-keeping requirements. In general, the Regional Meetings contributed to the State goal of increased efficiency by reducing duplicative efforts and increasing understanding and collaboration between organizations. Bringing the local-level groups together positively impacted service provision, according to attendees, they also help the state meet their goal of strengthening the statewide home visiting system. Future success will be contingent upon convening meetings that include invested service providers. When asked about the factors that contribute to forming successful connections among professionals, one state leader said, “It’s all about building relationships—just like we encourage within families between parent and child… We use the same philosophy to build those connections between programs and families, staff and supervisors, between agencies and sharing information and resources.”

Regional Meetings contributed to the State

goal of increased efficiency by reducing

duplicative efforts and increasing

understanding and collaboration between

organizations.


Other State-Level Efforts to Promote Collaboration The state worked diligently with the federal technical assistance provider to establish plans for continuous quality improvement (CQI); the major goal being to coordinate services and resources to children and families more effectively. One effort that began after the first year of the MIECHV Development Grant was the Help Me Grow referral service. This is a free service that connects families with critical developmental resources for their children birth through five years. The goal of Help Me Grow was to identify children at risk and link them to the help they need. This is a program used by others states which West Virginia OMCFH decided to adopt.

According to interviews with the MIECHV Stakeholders, Help Me Grow has been a valuable resource for families to access directly, though less useful for program staff (i.e., home visitors and managers) to access for information. Another effort that may contribute to more effective referrals, communication and record-keeping between different home visiting agencies is the State’s attempt to secure a web-based data tracking system. Many staff and stakeholders discussed the need for this move to a more modern and consistent system to assist with documentation and information sharing. It is anticipated that the new system will be in place by fall of 2014.

Community Collaboration: Local-Level Activities

At the local level, collaboration occurs in two ways: referrals of families to services and participation in community activities. Because the home visitors live in the communities where they work, many have developed personal connections which give them access to services for their client families which might otherwise be less accessible. At a broader level, home visitors and the administrative and supervisory staff also participate in joint efforts, such as task forces, created to address specific community concerns. None of the interviewees ever found it difficult to connect with other providers, however whether it was sometimes or always easy varied by interview round, with the middle round showing the best results (Figure 21).

Figure 21. Ease of Community Collaboration

70%

8%

38%

30%

92%

63%

0%

20%

40%

60%

80%

100%

Round 1 Round 2 Round 3

Sometimes: collaborationdepends on the provider

Easy to partner with otherproviders


“The regional meetings have been more effective at connecting people and helping people understand who is doing what... At the local level, collaboration probably works better because they have a better sense of what’s available in their communities.”

Local Service Continuum Most of the Community Partners surveyed (51 of 58) indicated that it was very easy to communicate with home visiting staff. The most frequent form of collaboration reported was participation in joint meetings, taskforces, or coalitions. Only seven respondents indicated that as a collaborative partner they had not participated in shared events offered by the OMFCH. Sharing information was also a frequent form of communication, with only two respondents indicating that it never occurred. A few respondents provided specific examples of collaboration that was working best. Sharing in community activities was popular among respondents, with some stating that having a shared focus and working together on events worked particularly well, and allowed families to come together and share information with one another. A vast majority of the respondents agreed that collaboration was very important to building a positive community. Some noted that their inclusion in community events over the years allowed for the sharing of more information with families about the services available, indicating that the broader efforts have paid dividends for specific families.

Training Free or very low-cost training opportunities continue to be valuable to staff. Many home visiting staff reported gratitude for the direct support from the State office in providing training on additional requirements for the MIECHV program. Since priority has been placed on serving high risk groups, staff started networking within their local communities to address concerns and needs of families served. Likewise, the State has made arrangements to host joint training including other providers in effort to increase competence as well as collaborative opportunities. Staff members at both direct service and supervisory levels expressed interest in meeting more regularly with local service providers so they could collaborate in working with certain families. Specific examples included serving families who have high-risk pregnancies, working with drug-affected babies, and working with adults who have developmental disabilities and/or mental health needs.

Local CQI Efforts Related to Collaboration Community partners also reported relatively frequent connections with home visiting programs through family referrals. The majority of respondents (70%) have received referrals from their home visiting program partners. Only nine of the 58 respondents (15.5%) indicated that they never made referrals to the home visiting program.


A vast majority of respondents agreed that collaboration was very important in building a positive community.

Over 29% were new partners who began working with IHFE in the past year or two.

While the local connections and partnerships continue to grow between various service providers and the home visiting programs, three areas surfaced during the first year of the evaluation that continue to be important and are possible areas of focus for the Continuous Quality Improvement (CQI) activities, connections to medical services, program visibility and partnering with providers. Connection to medical services: Despite the reports of extensive collaboration, both supervisory and direct service staff from the home visiting programs expressed a desire to have more and better connections with local medical professionals. One issue here is related to the programs’ goal of enrolling families before or immediately following the birth of a baby. Home visitors need referrals from medical staff to know who is pregnant. However, the home visiting staff also want better connections with medical professionals because they recognize that the latter are highly regarded by parents and are therefore valuable prospective partners for the in-home staff when providing guidance on the health and development of the child. That is, if a home visitor expresses a concern about the child’s development and the parent seeks help from a medical professional, the home visitor would like medical professionals to be proactive rather than taking a “wait and see” approach. Staff also indicated that they would like to have better access to mental health and substance treatment professionals. These types of services were mentioned at nearly every interview site as being lacking or having a long wait list. Visibility of home visiting agencies: The data provide somewhat conflicting information about some of the other possible challenges home visiting programs face. For instance, staff indicated that their organizations had limited visibility in the community or were often confused with others, particularly Child Protective Services. However, over 80 percent of community partners reported that the home visitors had high levels of visibility in the community and over 95 percent indicated that the programs have strong reputations. Partnering with other providers: While home visiting staff expressed some concerns about competing with other programs, only four of the 28 community partners with an opinion thought they competed either for clients or for funds. The Staff Survey and the Points of Contact forms completed by the home visitors also showed that home visitors access and partner with a wide variety of service providers. These include the state early intervention providers (called Birth to Three), public schools, health departments, hospitals and clinics, WIC and nutrition services, and churches and faith-based organizations, just to name a few.

A few notable trends were apparent from the qualitative activities of the evaluation: one quarter of the home visitors reported that homeless services and/or shelters were not available, followed by 21 percent citing alcohol or substance abuse services were lacking, and 20 percent said legal services or courts were insufficient to meet the need of their community.


More systematic information comes from both the Staff Survey and the Points of Contact form home visitors complete during the course of their work with families. The survey presented home visiting staff with a list of services and asked them to indicate how readily available each was. Figure 22 shows the results. It is no surprise that programs supported by the federal government such as WIC and food stamps/SNAP are perceived by home visitors as being most accessible.

Figure 22: Availability of Community Programs

83.3%

79.2%

78.1%

77.5%

75.3%

74.0%

65.8%

62.9%

61.6%

58.9%

57.5%

56.2%

53.4%

46.6%

43.1%

42.5%

39.7%

35.6%

30.1%

30.1%

25.0%

20.5%

15.3%

19.4%

22.2%

20.8%

25.0%

25.0%

25.0%

34.7%

34.7%

36.1%

41.7%

44.4%

41.7%

50.0%

52.8%

38.9%

55.6%

56.9%

63.9%

68.1%

63.9%

61.1%

1.4%

1.4%

1.4%

1.4%

9.7%

1.4%

4.2%

5.6%

1.4%

5.6%

4.2%

4.2%

19.4%

5.6%

8.3%

6.9%

2.8%

11.1%

19.4%

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

Food Stamps/SNAP

WIC

Birth to Three/Early Intervention

Child Health Insurance Program (CHIP)

WV DHHR Child Protection Services

Medicaid

Family Resource Centers

Right From the Start

Head Start

Domestic Violence Prevention

Family Planning/Women’s Health

Food Services (food pantries, churches)

Baby Pantry

Mental Health Services

Services for Children with Special Health Needs

Early Head Start

Skills Building/Vocational

Child Care (home or center-based, for infants-preK)

Alcohol or Substance Abuse Counseling or Services

Fuel or Utilities Assistance

Parent Support Groups

Prenatal Classes

Always Available Sometimes/Often Available Never Available


“The people [involved in the IHFE program] are wonderful and loyal and dedicated. As long as the leadership has passion, the relationships will continue to be strong.”

The 454 unduplicated Points of Contact records9 home visiting programs submitted during the last quarter of the evaluation period provide a different perspective on service availability but with a similar result. Looking at these contacts, we get a snapshot of the trends in professional connections between home visiting and other providers, that is the number and types of providers contacted by home visiting programs on behalf of families. . Where agencies fall into multiple categories or provide unidentifiable services (i.e., individuals or religious organizations), they are categorized here as part of “other.” Excluding “other,” the most frequent types of referrals were education-based (11%) and health-related (10%), followed by those for basic needs which encompasses food, clothing and shelter unless the need was related directly to a baby, in which case it was classified as baby-related. The overlap of high usage and high availability of services to fulfill basic needs demonstrates consistency of response between the two data sources. Staff members at all locations stated they worked hard to connect families to necessary services, and this was noted as a necessary aspect of their work even before the MIECHV Development Grant. During the Regional Collaboration Meetings, home visiting staff said that in many of the more rural areas services for basic needs continue to be insufficient for the families served. The most commonly reported challenge was reliable and dependable transportation for families. Reportedly very few programs can support families in accessing transportation, making that a barrier to accessing many other supports, including child care, medical appointments, and substance abuse treatment or services. As is well reflected by the variety of services, resources, and contacts displayed in Figure 23, local home visiting collaboration efforts result in families receiving a wide range of necessary support addressing topics such as education and child health and development and nutrition information. Additionally, of the 21 home visiting programs required to submit contact information, there were a total of 60 “formal” Memorandums of Understanding (MOU), another example of the strong connections that exist on the local level.

9 Home visitors record information quarterly about the actual contacts they have made referring a family to specific services. The service itself is

not recorded, but the type of agency sometimes provides an indication of the kind of service.


Figure 23: Community Contacts by Category or Type

11%

10%

9%

8%

7%

7%

5%

5%

4%

4%

4%

3%

3%

3%

3%

2%

2%

2%

2%

2%

2%

2%

1%

1% 21% 41%

Education

Medical/Health Related

Other

Basic Needs & WIC

Advocacy/Lobbying

Home Visiting

Birth to Three

Family Resource Center

DHHR

Legal

Mental Health

Family Resource Network

Child Protective Ser

Faith-based Support

Library

Resource

Baby Related

Child Care

Substance Abuse

Head Start

Homeless Shelter

Right from the Start

Domestic Violence

Child Advocacy Center

Crisis Services

Disability Support

In order to meet the needs of families living in

high-risk situations, programs have been

resourceful in collaborating on providing

services as shown here with the large variety

of service types. It would appear than no one

program has the burden of providing services

exclusively.

However, those with the greatest amount of

contacts are in two distinct areas:

Education

Medical & Health


Community partners valued the shared focus of the various agencies, the tangible support that the agencies have provided one another and the joint training activities.

Community Agency Views of Collaboration

Overall, collaboration appeared to be one of the strong points of the home visiting program, although it is difficult to tell if these strong connections were already in place before the MIECHV Development Grant, given the solid leadership of PiCO and the relatively small number of professionals across the State office that keep each other informed. Community partners valued the shared focus of the various agencies, the tangible support that the agencies have provided one another and the joint training activities. In fact, when asked what could improve their relationships with the home visiting programs, the majority of respondents (86%) who commented on this question (more than half the respondents) said that nothing needed to be done. Comments alluded to the importance of continuing to build on the foundation already established in local communities. Community partners agreed that strong partnerships begin with positive individual connections among professionals working in the same area. The one thing respondents did state would help improve their relationship was more time to collaborate, specifically when serving specific, shared families. For example, staff are not always aware of other providers that may be coming into the home. This theme emerged from Regional Collaboration Meetings as well as from home visiting staff specifically. According to supervisory staff, conversations about reducing duplication and sharing appropriate information would help assure more efficient and effective services are provided. The biggest barrier to coordinating services between programs was time and money. Community agencies lack the time and the funding to reach the level of collaboration they would like to have. Asked to respond with their opinions towards home visiting programs (on a scale of 1 as low to 5 as high), community agencies have highly favorable views on home visiting programs, as noted in Figure 24.Community agencies view home visiting programs as having good reputations, being effective in helping families, and providing quality information, to name just a few.


80% of community agencies said it was

very easy to communicate with home visiting staff.

45% also said they had worked with

home visiting for 5 years or more.

40% said they contacted home visiting

service monthly; 38% said weekly or

more often.

Figure 24: Community Agency Views of Collaboration

15.2%

18.2%

17.8%

14.0%

14.9%

15.6%

19.0%

15.6%

2.2%

6.8%

4.4%

4.7%

6.4%

2.2%

16.7%

2.2%

82.6%

75.0%

77.8%

81.4%

78.7%

82.2%

64.3%

82.2%

0% 20% 40% 60% 80% 100%

Quality of information

Collaboration between your organization and HV

Efforts of the HV program to communicate

Effectiveness of HV in helping families

Responsiveness of HV

Quality of overall service provided to families by HV

Visibility of HV in your community

Reputation of HV in your community

Low Rating

ModerateRating

High Rating


The most successful examples of community collaboration come from individuals who displayed solid interest in working outside of their own agencies.

Summary of Program Strengths

At the conclusion of this evaluation that began close to two years ago, leaders of West Virginia’s Home Visitation Program are in a position to examine the results of their concerted effort to improve opportunities for professional development and community collaboration as they prepare to expand services to families with greatest risk. The State has been successful in many areas as described throughout this report: it has increased access to training; partnered with community providers to offer topics relevant to staff needs and concerns; promoted best practices in supervision systems; and supported programs in their efforts to develop stronger connections with local-level organizations. The quality of staff supervision stands out as a critical factor in ensuring job satisfaction and reducing plans to leave the their positions. The support provided to all staff to participate in free and low-cost training has been very beneficial and has contributed to an increase in training hours across models, where once this may have depended on limited funds. Similar to the first year’s results, many home visitors and supervisors continued to express the desire for more intermediate and advanced training with a practical focus on how to handle the complex situations that families experience. Staff are interested in building on the foundational knowledge they have received about multiple risk factors and the ACEs. Some staff have begun exploring enrolling in certification or credentialing programs to that end. At the end of the evaluation period, some supervisors indicated that they would like to see training or mentoring opportunities that would promote professionalism of the field. While an increasing number of programs reportedly use reflective supervision, this does not always include time devoted specifically to professional or workplace ethics. This could be an opportunity for OMCFH and their Stakeholders to consider in planning the next Home Visitation Conference and in structuring the professional development system as a long-term goal of the CQI plan. A concern that was expressed on the staff survey and during interviews was the challenge experienced with new paperwork requirements and the time-consuming nature of the additional documentation, much or all stemming from new federal requirements. Program staff (including some in supervisory positions) still do not fully understand where the data goes or how the information gathered is used. Many staff members described how helpful the leadership was in providing first-hand guidance and suggestions for handling data collection requirements, though many still felt stretched between completing the requirements and conducting regularly scheduled visits with families.


The data collection landscape will change with the introduction of a new data system this year. OMCFH has been working diligently with a contractor on setting up a web-based data collection system. This will assist in coordinating data as required by the federal MIECHV benchmarks, but will also be useful in reporting other outcomes relevant to West Virginia. Home visiting programs are very comfortable networking across models. Any sense of competitiveness is missing. On the contrary, programs were eager to meet with providers from other models for useful and efficient management strategies. The Regional Collaboration Meetings were positively received and well-attended when the plans for meetings were sent out well in advance. Those that were less-attended had been scheduled with less notice and/or cancelled and rescheduled due to weather. Supervisors trained on reflective supervision continue to demonstrate commitment to the process. They have made efforts to follow more formalized processes such as meeting regularly and having agendas, both of which are suggested by the reflective supervision model. While many home visitors themselves were not as familiar with the reflective supervision concepts per se, as a group they felt supported by their supervisors. Community partners providing feedback to the evaluators were unanimously positive about home visiting services in their community. They perceived home visitors to be collaborative and helpful to families. The most successful examples of community collaboration seemed to come from individuals who displayed solid interest in working outside of their own agencies. While it is important that MOU’s are secured and tracked, collaboration that is effective and long-lasting ultimately stems from positive connections among individuals; this has become apparent during this evaluation process.


Recommendations

This section provides six recommendations for West Virginia’s OMCFH and Stakeholders Team to review as they continue to define professional development systems, promote state and local-level collaboration, and consider expanding home visitation services. 1. Organize a formal venue to release the final Core Competencies for Home Visitation. The State worked diligently to revise the standards that all home visiting programs in West Virginia could use as part of their self-assessment to adhering to evidence-based guidelines relevant to each curriculum. From this standpoint, an official “kick-off” or formal review of the Core Competencies, including all Stakeholders and home visiting professionals, would aid in the State’s interest of strengthening partner relationships and community collaboration. This type of event, similar to others such as the Home Visitation Conference and the Help Me Grow Kick-Off, will unite providers in the common goal of ensuring that the highest quality services are provided to families. In addition, this gives the leadership an opportunity to review the Core Competencies and outline expectations for program use and adherence. Going forward, Core Competencies could also be reviewed during the biannual site visits, which were greatly appreciated by home visiting supervisors and staff. 2. Continue to provide training relevant to home visiting profession, particularly higher-level skill-based

training. West Virginia OMCFH has increased the opportunities and access to more training specifically designed for this field. OMCFH has gone to extensive lengths to assure trainings are either a reasonable cost and/or free to participants. Likewise, these opportunities have been made available to community partners so that the information can be beneficial to others as well. Building on the foundational training such as that offered through the State Home Visitation Conference, OMCFH should work toward offering more intermediate or advanced level training to build upon existing knowledge and expertise. This can be done through “tracks” at conferences and/or multi-level training where pre-requisites must be completed before progressing. Furthermore, these advanced trainings can specifically address the topic areas that staff have expressed needing, including working with families who have serious mental illness, serious depression or substance use and working with families who have babies affected by drug use and children with special needs. For staff members who already have extensive knowledge and experience, the State could also consider supporting home visitors through formal coursework and working toward earning a specific credential. Staff who feel competent in their roles are more likely to feel fulfilled and ideally, the State will retain a cadre of highly-trained professionals. 3. Continue to emphasize Reflective Supervision and Reflective Practice. Quality of supervision was shown in this study to be the most important factor correlating with job satisfaction and an intent to stay with home visiting. Given that the vast majority of supervisors reported implementing “most or all” elements of reflective supervision, it is clear that the emphasis OMCFH has put on training and support to that end has been beneficial. If the goal is to have all programs successfully using this model of reflective practice, the State could continue to work with supervisors on the essential requirements of supervision and ways to support staff in difficult situations. In addition, training should be required for direct service staff as well, with the idea that if both parties who are engaging in this process fully understand the intent and purpose of reflection, they will be more successful in applying the same principles when working with families. Assuming that the basic elements of reflective practice can easily be reviewed though brief two-to-three hour interactive sessions, the State should consider outlining (and offering) training required as part of either home visitor


orientation or annual updates on related topics such as conducting case reviews, motivational interviewing, using a strengths-based approach, maintaining professional boundaries and developing cultural competence. 4. Convene regularly-scheduled Regional Collaboration Meetings. As part of the State’s effort to improve community linkages, reduce duplication of services and maximize resources, it successfully organized the first Regional Collaboration Meetings at the Home Visitation Conference in Charleston. These meetings were facilitated by an outside provider and were viewed by staff to be a positive experience in terms of fostering local-level relationships and mutual understanding of program purpose. OMCFH could continue to lead this effort by arranging for quarterly or biannual meetings to include regional home visiting programs and their partners, fitting nicely into their goal of coordinating efforts to improve efficiencies and a healthy working environment across early childhood programs. Some of the topics to address are: How to best coordinate services to families involved with multiple agencies; how home visitors can be aware of the presence of other agencies; how to work with the medical profession both to increase prenatal referrals and to avoid, wherever possible, the “wait and see” attitude towards the early identification of issues by home visitors; and how to foster access to needed services such as transportation. 5. Use MIECHV Stakeholders as advisors to inform major home visiting activities. West Virginia is a geographically large state with a relatively small workforce. There are highly-competent professionals recognized for their knowledge, experience and expertise who are asked to participate in many different aspects of serving children and families. As a result, the homogenous group composition can contribute to blurred boundaries between programs. This can also be challenging when a program’s goal is to expand and be more inclusive of outside perspectives. As an example, the MIECHV Development Grant funded three home visiting models in particular. It was widely recognized that these programs work very closely with other home visiting model providers, thus it made sense to involve representation from each model in the Stakeholder’s Team. Apart from the initial meeting, the Stakeholder’s meetings were viewed by its members as optional, especially since they were likely to see each other in different venues, making it seemingly possible to fulfill multiple obligations through one meeting. However, not every member was involved with the other meetings, so the second unintended effect was that members felt excluded when in fact meetings did not take place. Interviews revealed that not only was there disconnect within the group, but there was also a lack of purpose and a feeling of disorganization for those who did not have overlapping responsibilities. To avoid this confusion going further, Stakeholder Meetings could:

a. be scheduled at regular intervals with sufficient advanced notice; b. use web-based technology for alternating in-person and conference call style meeting; c. include a standing agenda, attendance forms, meeting minutes and action steps for members to

reference after each meeting; d. consider assigning an outside facilitator that is reliably available to keep all documentation and

disseminate to members; e. not be combined with other existing meetings (except in special circumstances such as with joint ECAC

meetings where the agenda specifically requires attendance of both groups); f. involve CQI and evaluation representatives to close the loop between program inputs and expected

outcomes; and g. be required or incentivized for members to attend.


6. Incorporate evaluation results into State-level CQI Plans. Under the MIECHV requirements, each state must develop and refine its formal process for examining efforts, data and relevant information to fit into the Plan, Do Check Act cycle of quality assessment and programmatic improvement. To formally and objectively examine the activities designed to improve professional development and collaborative systems, West Virginia’s OMCFH retained an outside evaluation and consulting firm that could carry out a multi-year study which would guide the state in its CQI activities. Using this report to guide state-level CQI efforts, some suggestions for upcoming activities using self-reflection and data analysis are briefly described here.

a. Conduct a review of programs and geographic areas that were included in the evaluation to determine

who, if anyone was missing. For instance, did all areas of the state have adequate representation? Which models could be included going forward? Were there providers who should have had input but did not?

b. Discuss findings in each research topic area with CQI leadership team and MIECHV Stakeholders. This step involves reviewing federally-required benchmark data with epidemiologists to determine which, if any, have been affected by the project efforts. How do the results of the professional development and collaboration study relate to the outcomes of families and children enrolled? Is there anything the State anticipated would change that did not? Were there surprises in the results that can inform decision-making or advocacy work?


Appendix A: Community Partner Survey

Hornby Zeller Associates, Inc. © 2013 Page 1

West Virginia Home Visiting Program

Community Partner Survey 2012-2013

In-Home Family Education (IHFE) in West Virginia relies heavily on its partners

and community organizations to best serve the children and families enrolled in

the home visiting program. West Virginia DHHS would like to find out more about

the local connections. Your program was identified as a collaborative partner to

an In-Home Family Education site. Please take a few moments to complete the

following questionnaire. We are looking for honest feedback, but your

participation is voluntary. All responses are kept confidential and will not be

shared with anyone outside of Hornby Zeller Associates’ evaluation team.

1. Name of your organization _________________________________________

2. What is your role in this organization?

Administrative Staff

Director

Home Visitor/Family Educator

Program Manager/Supervisor

Other (specify) __________________________

3. What counties do you serve? __________________________________________

4. How would you categorize your organization? (Choose only those that best apply)

Adult Ed or Vocational Program

Advocacy Group

Birth to Three (early intervention)

Child Care Provider/Center-Based Early Education

Child Protective Services (CPS)

College or University

Community Action Program (CAP)

Community Health Center

Community Mental Health Provider

Early Head Start (home-based, center-based or combo)

Family Leadership First

Family Planning Program

Family Resource Center

Family Resource Network

Head Start

Homeless Shelter or Services

Housing Services

Hospital or Medical Center

Law Enforcement or Public Safety Program

Partners in Prevention Grantee

PCP: Family Practice

PCP: OB/GYN

PCP: Pediatrics

Public Health

Public School


Substance Abuse Services or Program

WIC Program

Other (specify) __________________


5. How long have you worked for this organization?

Less than 1 year

1-3 years

3-5 years

More than 5 years

6. Is your program located in the same building with IHFE?

Yes

No

Don’t know

7. Is your program part of the same agency as the IHFE program?

Yes

No

Don’t know

8. How long has your organization worked with the IHFE program?

Less than 1 year

1-2 years

3-4 years

5 or more years

Don’t know

9. Does your organization have a formal Memorandum of Understanding (MOU)

with IHFE?

Yes

No

Don’t know

10. In the past year, how often did you (or someone at your organization) have

contact with staff at the IHFE program?

More than once a week Weekly Monthly

Quarterly

Once or twice in the year

No contact this year

11. How easy is it to communicate with the IHFE staff? (by phone, email or other)

Very

Somewhat

Not easy

Have not tried


12. How would you categorize your relationship with the IHFE program?

(You may select more than one.)

Funding Source

Fiscal Agent

Training Provider/Educator

Provider of Services for Children

Provider of Services for Adults

Provider of Services for Families

Referral Provider

Referral Recipient (IHFE sends families to you)

No relationship

Other relationship (specify) ________________________________________

13. In the past year, to what extent has your program worked with IHFE on

following activities?

A lot Some

Not at

all

Don’t

know

a. Participated in joint staff training

b. Provided training for the program

c. Sent funds to the program

d. Received funds from the program

e. Shared resources (explain):

f. Referred families to them

g. Received referrals from them

h. Provided space or materials to the program

i. Shared information with them

j. Participated in joint meeting, taskforce, coalition

k. Volunteered (or provided volunteers)

l. Co-sponsored or co-facilitated activity or event

m. Other (explain):

14. Please share your opinion of the IHFE program in the following areas, using a

score between 1(LOW rating) and 5(HIGH rating)

1

Low

2

3

Moderate

4

5

High Unknown

Quality of information from IHFE

Your understanding of the IHFE services/program

Collaboration between your

organization and the IHFE program


Efforts of the IHFE program to communicate with your organization

Effectiveness of IHFE in helping families

Responsiveness of IHFE to your organization

Quality of overall service provided to families by IHFE

Visibility of IHFE in your community

Reputation of IHFE in your community

Continues on next page →


15. How much do you agree or disagree with the following statements?

Agree

Somewhat Agree Neutral

Somewhat Disagree Disagree

No Opinion

Working with this program has

been positive

I think it is important to

collaborate with this program

I feel families would be better

served by some other program

The IHFE staff are adequately

trained to work with families

I trust the program to refer

families to us

I feel we are competing for clients

I feel we are competing for funds

I believe there is a need for IHFE

services in my community

I feel that IHFE should offer universal services

I feel IHFE should focus its efforts on certain families (explain):

16. What are some elements of collaboration with IHFE that have worked

particularly well? _________________________________________________________

___________________________________________________________________________

17. What could be done, if anything, to improve your relationship with IHFE?

______________________________________________________________________________

18. What do you feel are the current barriers to collaborating or coordinating

services with IHFE programs? _____________________________________________

____________________________________________________________________________

19. What is one thing you would suggest the program change or do? ________

______________________________________________________________________________

Have you expressed your concerns directly to the program?

Yes

No, I have not because _________________________________________________

___________________________________________________________________________

Thank you for your participation. Your feedback will help us thoroughly

evaluate the effectiveness of the West Virginia Home Visiting Program.


Appendix B: Staff Survey Questions

West Virginia Home Visiting Program

2012 Staff Survey

The West Virginia Home Visiting Program (WVHVP) is working to improve how you as home visiting staff

feel about your work and the support that you receive from your supervisors and managers. Please take a

few minutes to complete this confidential survey which is administered by an outside firm. Your answers will

not be associated with you and will be reported along with the responses of others as a group. Your honest

feedback will directly influence how the WVHVP will focus its efforts in the coming years. This survey is

optional and the answers that you give will not impact your position but it may improve the state of West

Virginia’s home visiting program.

Staff Survey DATE COMPLETED _______________________

SECTION 1

Tell us a little about yourself…

1. The county(ies) where you work: <check all that apply>_____________________________________

2. The home visiting program you work with: ________________________________________ The home visiting model you use: □ MIHOW □ HFA □ PAT □ Early Head Start □ Other ____________________

3.

Your title _________________ How long in this position: _____ months ____ years

Do you work □ Full time or □ Part Time as a home visitor? Do you supervise anyone who is a home visitor? □ Yes □ No If yes, do you supervise full time? □ Yes □ No

Is your job as a home visitor your only job? □ Yes □ No

If not, what is your other job(s)? _____________________

4.

Before this job, did you have other experience working with families and young children? □ Yes □ No If yes, your experience included (check all that apply):

□ home visitor □ case manager □ teacher □ teacher aide □ social worker □ nurse □ early childhood specialist □ child care provider □ other (please specify) ________________________________________________

5.

The highest level of education you have achieved so far: □ High school/GED □ Associate’s Degree □ Bachelor’s Degree in Child Development, Social Work, or related □ Other Bachelor’s Degree □ Master’s Degree in related □ Other Master’s Degree □ Doctoral degree □ Other ( please specify) ______________________________________

6. The year you were born ______

7. Your gender: □ Female □ Male

8. Your relationship status: □ Single □ Divorced □ Married or Partnering □ Widowed

9. Do you have children? □ Yes □ No If yes, check all that apply:

□ Birth □ Adopted □ Step □Foster □ Kinship Care Do you have grandchildren? □ Yes □ No

10. Which of the following best describes your racial/ethnic background (check all that apply): □ African American □ Asian/Pacific Islander □ Native American/Alaskan Native □ Hispanic/Latino □ White/Caucasian □ Other (please specify) _________________

SECTION 2

Tell us about your job...

11.

Do you see yourself staying in this job for the next year or two? □ Yes □ No Do you think you would leave in the next three months? □ Yes □ No Are you satisfied with your job as-is? □ Yes □ No In the past year, have you looked at ads for other jobs? □ Yes □ No Have you sent your resume or put an application in somewhere? □ Yes □ No

12.

If you were to leave your job as a home visitor, why would you leave? □ Higher pay □ Continuing my education □ Better benefits □ Accept a promotion □ Better job opportunity in the same field □Career change □ Don’t like the commute/ required travel □ Program instability □ Conflict with other employees □ Relocation/Move □ Family, health or personal □ Conflict with management □ Don’t think I am making a difference □ Other reason (please describe briefly): ____________________________________________________

13.

Do you participate in work-related committees within your agency? □ Yes □ No If yes, how many hours per month do you spend? _____ How about outside of your agency, do you participate in work-related committees in other places? □ Yes □ No If yes, how many hours per month do you spend? _______

14. What is your current caseload? ________ families. (If you are supervisory ONLY, with no families assigned, please enter 0)

15. How many hours do you work in a typical week? ______ hours/week.

16. What is your annual salary? <include ranges to select from: $0-10,000; $10,0001-20,000, etc.>

17.

What initially attracted you to the home visiting field? Check all that apply: □ Work is Rewarding □ Flexibility in Scheduling □ Prior Experience □ Formal Training

□ Had a Home Visitor Myself □ Believe I Can Make a Difference

□ Other, please specify: ___________________________________________

SECTION 3 Tell us about your training needs and interests…

18.

How many hours of face-to-face, job-related training have you attended in the last 12 months? _______ How many of these hours were NOT recorded in STARS? ________ How many hours of web-based trainings have you received in the last 12 months? __________________ If you have not attended any training, SKIP TO Question #.

Please mark each item with your level of agreement.

Strongly

Disagree

Disagree

No

Opinion

Agree

Strongly Agree

Not Applicable

19. The training prepared me to work with low- birth weight babies and their families.

20. The training prepared me to work with families with substance use or alcohol abuse.

21. The training prepared me to work with families with mental illness.

22. The training helped me understand developmental disabilities or delays.

23. The training prepared me to work with families who experience abuse, neglect, or domestic violence.

24. The training prepared me to work with families on children’s learning and literacy.

25. The training prepared me to work with families on children’s social and emotional growth.

26.

The training provided all the information I need to complete mandatory forms and required assessments.

27. I receive adequate training to do my job.

28. My agency gives me adequate opportunity for professional development.

29. I find it useful to have partner agencies (such as Head Start, WIC, Birth to Three) in the training.

30.

I have participated in Technical Assistance Center on Social Emotional Intervention for Young Children (TACSEI) training.

□ Yes, currently attending □ Yes, completed training □ No

31. I have participated in Center for Early Literacy Learning (CELL) training.

□ Yes, currently attending □ Yes, completed training □ No

32. I have attended _____ professional conferences in the past 12 months.

33. At this time I would like further training in ______________________________________________

SECTION 4A FOR HOME VISITORS ONLY. SUPERVISORY STAFF: SKIP TO SECTION 4B

Tell us about the supervision and support you receive… This section is about your direct supervisor only.

Remember that what you share is kept confidential.

34. Do you have regular meetings with your supervisor?

□ Yes, weekly □ Yes, every other week □ Yes, monthly □ Not regularly, but we do meet □ No, we do not meet

35.

Do you prepare for these meetings by creating an agenda or list of items to discuss?

Does your supervisor bring an agenda?

□ Yes □ No, my supervisor sets the agenda □ No, we just say what is on our mind at that time

□ Yes □ No

36.

Do you know who to go to if you are not satisfied with your supervisor?

Have you ever felt uncomfortable with your supervisor?

□ Yes □ No

□ Yes □ No If yes, why? ___________________________

SECTION 4B FOR SUPERVISORS OR HOME VISITORS WITH SUPERVISORY RESPONSIBILITY ONLY.

Tell us about the supervision and support you provide… This section is about the staff you directly supervise.

Remember that what you share is kept confidential.

34. Do you have regular meetings with your supervisees?

□ Yes, weekly □ Yes, every other week □ Yes, monthly □ Not regularly, but we do meet □ No we do not meet

35. Do you prepare for these meetings by creating an agenda or list of items to discuss?

□ Yes □ No, my supervisees set the agenda □ No, we just say what is on our mind at that time

36. Have you ever felt uncomfortable in your role as supervisor?

□ Yes □ No If yes, why? ___________________________

Q37-46 FOR HOME VISITORS ONLY. SUPERVISORY STAFF: SKIP TO SECTION 5

Tell us about the supervision and support you receive…

This section is about your direct supervisor only.

Please mark each item with your level of agreement. Strongly

Disagree Disagree No

Opinion Agree Strongly Agree

37. My supervisor is knowledgeable about topics related to home visiting.

38. My supervisor is helpful in working through challenges with the families I serve.

39. My supervisor communicates clearly.

40. My supervisor uses a strengths-based approach in our work together.

41. My supervisor encourages me to reflect on my actions and learn from my experiences.

42. I can approach my supervisor with my concerns.

43. My supervisor encourages our team to work together in a positive way.

44. My supervisor is a supportive advocate for home visiting staff.

45. I feel respected and valued by my supervisor.

46. I feel my supervisor is appreciative of individual skills, needs, and interests.

SECTION 5

Tell us a little about the organization you work for…

Please mark each item with your level of agreement.

Strongly Disagree Disagree

No Opinion Agree

Strongly Agree

47. I am paid well for the work I do.

48. I receive good benefits from my agency.

49. I feel respected and valued by my agency.

50. I have enough time to complete my job requirements.

51. I have the tools I need to do my job (e.g., computer, phone, supplies).

52. I can access needed services and resources for families and children I work with.

53. I was given a realistic description of my job before I accepted the position.

54. People who do well in my job have a good chance of getting promoted.

55.. My agency has a good reputation in the community.

56. I am proud to tell people who I work for and what I do.

57. My organization has clear goals and works toward them.

58. Communication within my organization is good.

59. I am clear about my job responsibilities.

60. My organization is innovative in meeting family needs.

61. I like the things I do at work.

62. I have to work harder at my job to compensate for the work of others.

63. My job negatively impacts my physical health and/or energy level.

64. I feel emotionally drained from this work.

65. My work environment is supportive.

66. I believe in the work I am doing.

67. My coworkers support my organization’s goals and mission.

68. I am confident in my ability to work with low-birth weight babies and their families.

69. I am confident in my ability to work with families with substance use or alcohol abuse concerns.

70. I am confident in my ability to work with families who have mental illness.

71. I am confident in my ability to work with families where there is domestic violence.

72. I know what to do or who to contact when working with children who have developmental disabilities or delays.

73. I know what to do or who to contact when working with families who experience abuse or neglect.

SECTION 6

Tell us about the other services available to families in your community…

Please rate how readily available each of these services are to your families.

74.

Never Available

Sometimes Available

Often Available

Always Available

Parent Support Groups

Prenatal Classes Child Care (home or center-based, for infants-pre K

age)

Alcohol or Substance Abuse Counseling or Services

Family Planning/Women’s Health Services

Early Head Start

Head Start

Medicaid

Skills Building/Vocational Services

Family Resource Centers

Mental Health Services Services for Children with Special Health Care

Needs

Fuel or Utilities Assistance

Birth to Three/Early Intervention Services

Food Stamps/SNAP

WIC Food Services (for example, food pantries or lower

cost food from a church)

Domestic Violence Prevention Program


Baby Pantry West Virginia Department of Health and Human

Resources Child Protection Services

Child Health Insurance Program (CHIP)

75.

What would you say are the most pressing problems for the families you serve?

SECTION 7 Tell us about your organization’s relationship with referral sources and partner

agencies…

Please rate how well your agency works with each listed here. Please choose Not Applicable if the service is not available in your area.

76.

Never Work Well

Together

Sometimes Work Well Together

Often Work Well Together

Always Work Well Together

Not Applicable

Birth to Three/Early Intervention Services

Right From the Start (or other nurses)

SECTION 8

Tell us about your overall outlook… Please mark each item with your level of agreement.

Strongly

Disagree Disagree No

Opinion Agree Strongly Agree

77. I have little control over things that happen to me at work.

78. Some of the problems at work I can't seem to solve at all.

Family Resource Centers West Virginia DHHR Child

Protection Services

Head Start/Early Head Start Programs

Clinics including Family Planning/Women’s Health Services

Hospitals, Physicians (or other

medical services)

Pediatric Primary Care

Mental Health Services

Alcohol or Substance Abuse Services

Domestic Violence Prevention Programs

Courts or Legal Services Shelters (homeless, temporary,

other) Schools (including preschool

public or private)

Faith-based Organizations

WIC

Health Department Please use the space below if you would like to explain any of your choices above.

79. There is not much that I can do to change important things at work.

80. I often feel helpless dealing with problems at work.

81. Sometimes I feel like I am being pushed around at work.

82. I can do almost anything that I really set my mind to.

Would you like to say anything more about your work?

83. What aspects of your work are you most comfortable with?

84. What is most challenging for you?

85. What more can the home visiting program do to support you in your job?

Thank you very much for your feedback. If you have other comments about your

home visiting program, please add them below.

PLEASE REMEMBER THIS SURVEY IS CONFIDENTIAL.

West Virginia Home Visitation Program Evaluation of Professional Development … · 2015-06-01 ·...

Documents

Transcript of West Virginia Home Visitation Program Evaluation of Professional Development … · 2015-06-01 ·...