EVALUATION SERVICES Research Priorities and Implementation

Text4Mood Evaluation Report 2016

2

This report was prepared on behalf of Text4Mood for:

Dr. Vincent Agyapong

BSc MB ChB DCP DHSM MSc MRCPsych MCPsychI FRAMI PGDip (Statistics) CSCST MD FRCPC

Associate Clinical Professor and Consultant Psychiatrist, Department of Psychiatry, Faculty of Medicine & Dentistry University

of Alberta, Edmonton & Northern Lights Regional Health Centre, Fort McMurray Alberta, Canada

780-714-4315

This report was prepared by:

Alecia Kallos

Research and Evaluation Coordinator

Evaluation Services

Research Priorities and Implementation

Alberta Health Services,

780-407-4671

[email protected]

Vanessa Gibbons-Reid

Evaluation Lead

Evaluation Services



780-407-4675

[email protected]

Ashley Drobot

Research and Evaluation Coordinator

Evaluation Services



780-407-4880

[email protected]


3

Background ..................................................................................................................................... 4

Methods ....................................................................................................................................... 4

Limitations .................................................................................................................................. 4

Results ......................................................................................................................................... 5

How Respondents Heard about Text4Mood .............................................................................. 5

Zone of Medical Care................................................................................................................... 5

Age of Respondents ......................................................................................................................7

Gender of Respondents ............................................................................................................... 8

Ethnicity of Respondents ............................................................................................................ 8

Respondent Employment Status ................................................................................................ 9

Chronic Mental Health Condition ..............................................................................................10

Chronic Medical Condition ........................................................................................................ 11

Why Respondents Signed up for Text4Mood ............................................................................ 12

Assistance Needed to Sign up for Text4Mood ........................................................................... 13

How Often Respondents Read the Text Messages ..................................................................... 13

What Happened when Respondents Received the Text Messages ............................................ 14

How Often Did the Respondents Understand the Text Messages ............................................. 14

How did the Text Messages make People Feel .......................................................................... 15

Respondent’s Perception of the Text Messages ......................................................................... 16

Level of Agreement about Text4Mood Outcomes ...................................................................... 16

Benefits of Text4Mood ............................................................................................................... 17

Respondent Satisfaction with Text Message Frequency ............................................................ 18

Preferred Text Message Frequency ............................................................................................ 19

Respondent Access of Emergency or Crisis Services ................................................................. 19

Respondent Frequency of Access of Emergency or Crisis Services .......................................... 20

Opinions About Use of Technology-based Services as part of Healthcare ................................ 21

Other Comments to Improve Service ......................................................................................... 21

Overall Survey Highlights about the Text4Mood Program .......................................................... 22

Evaluation Guiding Principles ...................................................................................................... 23


4

The Text4Mood program began in January 2016 to be a support to individual’s overall care plan

by aiming to provide a positive boost during the day. Participants sign up via text message to

receive a supportive daily text message. This report summarizes the findings from a survey

regarding the Text 4 Mood program.

An online survey was sent to 4111 individuals via text message. The survey was available for six

weeks, from March 1, 2016 to April 12, 2016. Individuals who had participated in the program

for at least six weeks received a text message with the survey link as well as a reminder text

message in the final week of the survey. Participants who had been in the program for less than

six weeks only received a text with the survey link once due to the short timeframe of the survey.

Survey questions were subject to interpretation by participants and all data was self-reported.

The online and anonymous nature of the survey also meant that participants may have filled out

the survey more than once.


5

A total of 894 individuals completed the survey giving an overall response rate of 21.7%. The

‘n’ for each individual question is reported alongside the results.

The top three ways respondents heard about the Text4Mood program were through a friend

(22%, n=198), on the news (19%, n=168), and other (18%, n= 161). The least common means of

hearing about the program were from a clinic (3%, n=30), the participant’s doctor or nurse (3%,

n=31), and Twitter (2%, n=20).

The majority of respondents received medical care in Edmonton Zone (42%, n=375), followed

by North Zone (34%, n=301).

22.0% 19.0% 18.0% 14.0% 11.0% 9.0% 6.0% 3.0% 3.0% 2.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Through afriend

On thenews

Other On awebsite

Facebook From aposter

From apamphlet

From myclinic

From mydoctor or

nurse

Twitter

42.0%33.7%

12.8%5.7% 3.7% 2.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Edmonton North Calgary Central Province (notAlberta)

South


6

Of the respondents from North Zone who specified the sites where they received medical care,

the most common sites were Fort McMurray (48%, n=100) and Grande Prairie (31%, n=66).

Only 8% (n=16) of participants received medical care from High Level and surround areas, and

no participants (0%, n=0) received medical care from St John and surrounding areas.

Of the respondents from Central Zone who specified the sites where they received medical care,

the majority received medical in Red Deer and surrounding community (51%, n=20), followed

by Lloydminster and surrounding areas (26%, n=10). Rocky Mountain House and surrounding

communities was the least common site of medical care (5%, n=2).

48.0%

31.0%

13.0%8.0%

0.0%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fort McMurry andsurronding

communities

Grande Prairie andsurroundingcommunities

Peace River andsurroundingcommunities

High Level andsurroundingcommunities

St John andsurroundingcommunities

51.0%

26.0%18.0%

5.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Red Deer andsurroundingcommunities

Lloydminster andsurroundingcommunities

Sherwood Park andsurroundingcommunities

Rocky Mountain Houseand surrounding

communities


7

Of the respondents from South Zone who specified the sites where they received medical care,

the majority of participants received medical care in Lethbridge and surrounding communities

(82%, n=11), while the remaining respondents received medical care in Medicine Hat and

surrounding communities (8%, n=1).

Half of the respondents were 26-45 years old (51%, n=410), while almost one-third were 45-

65 years old (31%, n=253).

92.0%

8.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Lethbridge and surrounding communities Medicine Hat and surrounding communities

2.0%

15.1%

50.7%

31.3%

0.9%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

<15 16-25 26-45 46-65 >75


8

Most respondents were female (83%, n=668).

The majority of respondents were Caucasian (83%, n=679). First Nations, Métis, or Inuit was the

second most reported ethnicity (5%, n=42).

82.5%

17.2%

0.4%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Female Male Other

83.1%

5.1% 4.8% 3.9% 2.0% 1.0% 0.1%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Caucasian First Nations/Metis/ Inuit

Other Asian Prefer Not toDisclose

Black Hispanic


9

Many respondents identified university degree or diploma (Bachelor’s) as their highest level of

education (32%, n=260), followed by college diploma (23%, n=189) and high school (20%, n=161).

Almost half of respondents identified as being employed full-time (49%, n=441), followed by

employed part-time (17%, n=151), and employed as a student (10%, n=89). The least common

employment statuses include those participants who identified as unemployed on government

social benefits (3%, n=28), retired (2%, n=15), while others preferred not to disclose (2%, n=19).

32.0%

23.0% 20.0%12.0%

7.0%3.0% 2.0% 0.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Universitydegree ordiploma

(Bachelor's)

College dipolma High schoolPost Graduatedegree (MSc,

PhD)

Apprenticeshipor trade

certificate/diploma

Junior highschool

Prefer not todisclose

Elementaryschool

Series1 32% 23% 20% 12% 7% 3% 2% 0%

49.0%

17.0%10.0%

6.0% 5.0% 5.0% 3.0% 2.0% 2.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Employedfulltime

Employedpart-time

Student Self-employed Homemaker Unemployedwith noincome

Unemployedon

governmentsocial benefits

Retired Prefer not todisclose


10

The majority of respondents stated that they did not have a chronic mental health condition

(56%, n=454), while 38% (n=307) of participants were diagnosed with a chronic mental health

condition.

* Participants were able to select more than one answer; percentages may be

greater than 100%. A quarter of respondents identified as having depression (n=227). The

next most common mental health conditions among participants included anxiety (20%, n=177)

and trauma or stress-related disorder (7%, n=58).

56.0%

38.0%

7.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

No Yes Unsure

25.4%

19.8%

6.5%

3.8%

2.7%

1.6%

1.0%

0.8%

0.7%

0.6%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Depression

Anxiety

Trauma and Stressor Related

Bipolar

Substance-Related

Personality

Schizophrenia Spectrum

Obssessive Compulsive

Neurodevelopmental

Sleep-Wake


11

The majority of respondents identified as not having a chronic medical diagnosis (66%, n=478),

whereas just over a quarter of respondents stated that they have chronic medical condition

(27%, n=192).


greater than 100%. Of the respondents who had chronic medical condition and who

provided details, 37% (n=70) had a nervous, musculoskeletal or sensory conditions. Other

common conditions included metabolic disorders (16%, n=30), endocrine disorders (14%,

n=26), gastrointestinal disorders (14%, n=26), respiratory disorders (13%, n=25), and

cardiovascular disorders (9%, n=18).

66.0%

27.0%

7.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

No Yes Unsure

36.5%

15.6%

13.5%

13.5%

13.0%

9.4%

5.7%

4.7%

4.2%

3.6%

2.1%

1.0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Nervous, musculoskeletal and sensory

Metabolic

Endocrine

Gastrointestinal

Respiratory

Cardiovascular

Cancers and Tumours

Dermatological

Reproductive

Other

Renal

Infections


12

*Participants were able to select more than one answer; percentages may be

greater than 100%. Approximately half of respondents (52%, n=461) signed up for text for

mood to help elevate their mood, while 49% signed up to help them feel better.

Of those respondents who selected the ‘other’ category, the majority of the respondents signed

up for this program to get support, motivation, inspiration and encouragement and to exercise

positive thinking. They use the program to remind themselves to practice mindfulness and view

life from a different perspective. In addition, a few respondents signed up to address specific

concerns such as, “ground my anxiety,” help with “potential postpartum depression,” and “to

help fight an addiction.”

“To switch my focus to positive thoughts.” “Provides hope.” “Start my day with something positive.”

Other respondents signed up to be able to share positive thoughts or to support their families or

friends that are dealing with mental health issues. A few respondents were curious about the

service and wanted to try it. Finally, some respondents are mental health service providers, such

as psychologists, who signed up to test if the service is appropriate to recommend to their clients

or managers who signed up as a trial prior to referring their staff to the service.

“I work in mental health and wanted to see what the service was like prior to recommending to my clients.” “To see if my patients would benefit from it”

51.6% 49.3%

24.5%19.7%

15.9%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Elevate my mood Help me feel better Help me worry less Novelty/fun Other


13

The majority of respondents did not need assistance signing up for the program (96%, n=694).

The majority of respondents (90%, n=648) said that they ‘always’ read the text messages.

Only 8.1% (n=58) read the text messages ‘often’.

96.0%

4.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

No Yes

90.0%

8.1%1.5% 0.3% 0.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Always Often Sometimes Rarely Never


14


greater than 100%. Two-thirds of respondents (65%, n =584) said that they read the text and

reflected on the message. One-third of respondents (33%, n=295) read the text more than once,

29% (n=258) read the text and took a positive action and 18% (n=160) read the text and took no

action.

Three-quarters of respondents (76%, n=545) said that they ‘always’ understood the text

messages that they received.

65.4%

33.0%28.9%

17.9%

0.9% 0.1%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I read the textand reflected

I returned toread the text for

support morethan once

I read the textand took a

positive action

I read the textand took no

action

I did not read thetext

I read the textand took a

negative action

75.7%

22.9%

1.4% 0.0% 0.0%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Always Mostly Sometimes Rarely Never


15

Seventy-five percent of respondents (n=545) said that the messages made them feel ‘supported’.

The majority of the respondents who chose ‘other’ (18%, n=127) stated that receiving the daily

text message made them feel good. They found the text messages set a positive tone for their

morning, making them feel “motivated,” “happy,” and “inspired.” Some mentioned that they

share them with friends and families to spread the inspiration and thought.

“…they are so positive and uplifting everyone should be having access to this wonderful tool.” “When everything is going bad, that message brightens my day. I look forward to it every day.” “They just help me to get or stay more positive when I am struggling.”

Some respondents highlighted that they had mixed feelings about the text messages. They found

some messages to be inspiring and supportive, while other messages were ineffective and

frustrating. Some respondents found that the text messages did not alter their mood or feelings

in anyway. A few participants found messages that had a religious tone (e.g. talked about God)

to be inappropriate. Others felt that the messages reminded them of their personal challenges

leaving them discouraged.

“I am grateful for them, and basically after reading it, I tucked it away in my notes. Sometimes they made a LOT of sense for that particular day/moment, but I do not really feel like they, in and of themselves saved me. They are like a treat each day, and some days I share them, some days they really hit home and other days they made me feel upset, anxious and guilty. But I DO NOT want to stop getting them!!” “Some are good. Others don't seem to be appropriate for what I thought the service is.”

75.0%

6.5%1.1%

17.5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Supported Made no difference Annoyed Other


16

Overall, respondents always found the Text4Mood messages to be positive (69%, n=501) and

on-topic (45%, n=325). Respondents identified messages as always supportive 62% (n=448) of

the time and always to the point 48% of the time (n=347).

For this question the response categories of ‘strongly agree’ and ‘agree’ were combined into

‘agreement’. The response categories of ‘strongly disagree’ and ‘disagree’ were combined into the

category ‘disagreement’. Survey respondents agreed or strongly agreed that participating in the

program helped them to remember goals (78%, n=562), monitor mood (74%, n=531), and cope

with stress (67%, n=556). Respondents neither agreed nor disagreed (65%, n=474) when it came

to Text4Mood’s ability to manage suicidal thoughts.

69.6%

62.2%

28.3%

32.5%

38.3%

4.9%

13.2%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Positive

Supportive

To the point

On-topic

Always Mostly Sometimes Rarely Never

11.5%42.2%45.1%

48.2%

77.2%

57.0%

23.9%

73.8%

78.1%

19.6%

35.1%

65.8%

22.9%

19.3%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Cope with stress

Cope with loneliness

Manage suicidal thoughts

Monitor my mood

Remember my goals

Agreement Neither agree nor disagree Disagreement


17

For this question, the response categories of ‘strongly agree’ and ‘agree’ were combined into


category ‘disagreement’. In each category, the majority of respondents were in agreement with

each statement. Most respondents felt hopeful in managing issues in their lives (82%, n=587),

followed by feeling on track with life or when everyday stressors come up (73%, n=522).

75.2%

81.7%

56.0%

76.7%

72.5%

19.4%

16.9%

31.3%

20.3%

22.5%

14.7%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Feel connected to a support system

Feel hopeful I can manage issues in my life

Know where to get help for depression or anxiety

Feel that I am the one in charge of managing depression oranxiety

Feel like I know how to stay on track with life or wheneveryday stressors come up



18

For this question, the response categories of ‘strongly agree’ and ‘agree’ were combined into


category ‘disagreement’. The majority of respondents felt Text4Mood improved overall mental

well-being (83%, n=598) and made respondents feel like they could bounce back if they made a

mistake (77%, n=554).

Overall, the majority of respondents were very satisfied or satisfied with the frequency of text

messages received (95%, n=681). Only 1% (n=8) of respondents were dissatisfied.

58.6%

77.0%

70.8%

83.1%

72.1%

36.0%

20.4%

26.0%

15.1%

25.4%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Feel like I am making a change

Feel like I can bounce back if Imake a mistake

Make better choices

Improve my overall mental well-being

Enhance my quality of life


56.0%

39.0%

4.0%1.0% 0.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Very satisfied Satisfied Neither satisfied nordissatisfied

Dissatisfied Very dissatisfied


19

Overall, the majority of respondents preferred to receive supportive text messages once per day

(68%, n=491).

The majority of respondents have not accessed emergency or crisis services for mental health

related problems since signing up for and receiving Text4Mood text messages (94%, n=677).

68.0%

17.0%12.0%

3.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Once a day Twice a day Once every other day Once a week

94.0%

6.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

No Yes


20

The majority of respondents said they never used an emergency or crisis line for mental health

problems (71%, n=514). While 5% (n=38) said they used crisis services less frequently since

signing up for Text4Mood. Only 2% (n=11) of respondents reported more frequent access of

emergency or crisis services.

71.0%

22.0%

5.0% 2.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

I have never used theemergency or crisis line

for mental healthproblems

No change Less frequently More frequently


21

The majority of respondents identified with most certainty in favor of text support for follow-up

care (52%, n=376), followed by text messaging for managing medical appointments (57%,

n=409). Use of technology-based services as part of healthcare was also identified as most

certain for Emails for managing medical appointments (49%, n=355) and web-based

counselling (42%, n=376).

In addition to answering the survey, some respondents provided other comments or emailed

Evaluation Services to add in extra information. Respondents stated that they wished that the

text messages came from the same number each day. Comments included:

“A new number every day that is not in my contacts created worry and a rush of anxiety” “I would like to get a text from one phone number only so that I can assign a distinctive text tone sound”

42.1%

31.2%

25.1%

56.8%

49.3%

52.2%

27.5%

29.4%

24.2%

26.8%

27.5%

28.9%

16.4%

18.5%

24.2%

9.4%

11.9%

12.9%

11.1%

17.5%

22.2%

9.4%

0% 20% 40% 60% 80% 100%

Web-based counselling

Telephone counselling

Consultation by video conferencing

Text messaging for managing medicalappointments

Emails for managing medical appointments

Text support for follow-up care

Most Certainly Probably Not Sure Probably Not Certainly Not


22

The majority of respondents signed up to use the Text4Mood program to elevate

their mood, help them feel better, or to support or motivate themselves and others.

Most respondents always read and understood the text messages and many read

the text and reflected or returned to the text for support. Generally, the texts left people

feeling supported and uplifted.

The text messages helped people to remember their goals, cope with stress, and

feel hopeful. Over two-thirds of respondents said that the text messages enhanced their

quality of life, improved their overall mental well-being, and made them feel like they

could bounce back from mistakes.

Respondents felt that generally, the text messages were on-topic, to the point,

supportive and positive. However, some participants noted that some topics, especially

those around religion, seemed inappropriate.

Most respondents preferred daily text messages. Some mentioned that the early

morning text messages were not ideal. Others also mentioned that receiving the text

messages from multiple numbers was confusing or stressful.

When asked about future use of technology-based services for healthcare,

respondents seemed open to using text for follow-up care and text or email for managing

medical appointments. There were mixed opinions about consultation by video,

telephone, or the internet.


23

Through evaluation excellence, healthcare practitioners are guided to do their best work to

improve practice and the delivery of healthcare services. Evaluation Services (ES) provides an

internal model of evaluation to Alberta Health Services (AHS). While there are organizational

benefits to having an internal evaluation service, this process can be compromised unless clearly

defined guiding principles are understood. Those principles include a commitment to integrity,

collaboration and ethical oversight.

Integrity

To reduce the risk of conflict of interest, Evaluation Services restricts evaluation services

to health-related programs or projects that are governed by operations outside of their

own Research portfolio. Doing so neutralizes potential power dynamics between the

evaluator and primary stakeholders of the evaluation. While the interpretation of

evaluation results by stakeholders is critical to providing context and a deeper

understanding, ES will only report on what the data presents and will not entertain

requests to exclude or adjust findings unless there is evidence that the request is valid.

This helps to safeguard the integrity of the evaluation results.

The evaluation practice is governed by a professional code set by the Canadian

Evaluation Society (2012), our own Survey and Evaluation Services Standards of

Practice (2011), AHS’s Code of Conduct (2013) and provincial legislations1. Evaluation

Services will not engage in inappropriate requests that may violate those standards and

the integrity and reputation of the evaluator and the evaluation.

Collaboration

We recognize that active stakeholder involvement in evaluation planning and decision-

making is essential to success. Collaboration helps to ensure that: evaluation results are

useful; decision-making is evidence informed; there is good stewardship of resources;

and through stakeholder engagement, a culture of evaluation evolves within AHS.

Ethical Oversight

Evaluation Services is committed to employing a systematic approach to ensuring the

highest ethical standard for this evaluation. Evaluation Services is committed to

providing sound methodology and ethical values and behavior at all stages of the

evaluation. Ethical issues will be identified and addressed as they arise. Ethical oversight

is essential to reducing risk to human participants and by protecting the personal and

health information collected and stored for evaluation purposes.

1 Governing provincial legislations include: Health Information Act (HIA), Freedom of Information and Protection of Privacy

(FOIP) and the Alberta Evidence Act (AEA)

EVALUATION SERVICES Research Priorities and Implementation - Mental Health Innovation ... · 2020....

Documents

Transcript of EVALUATION SERVICES Research Priorities and Implementation - Mental Health Innovation ... · 2020....