National patient survey programme - UK Data Service€¦ · The national patient survey programme...

National patient survey programme

NHS Mental Health Service Users

2006

Quick guide

Updated 13th January 2006

UK Data Archive Study Number 5598 - Mental Health Trusts: Community Mental Health Service User Survey, 2006

This document and other documents referenced within are available from the Mental Health surveys website at:

www.healthcarecommission.org.uk/MentalHealthSurvey2006

Contacts Mental Health Surveys Co-ordination Centre PatientDynamics Riverside House 5 Nutfield Lane High Wycombe HP11 2ND

Tel: 01494 560677

E-mail: [email protected]

Updates

Before you start your survey, check that you have the latest version of this document, as there might be some small amendments from time to time. (The date of the last update is on the front page.) In the unlikely event that there are major changes, we will e-mail all trust contacts directly to inform them of the change.

Changes to the procedures out l ined in th is document

It is not permissible to deviate from the agreed protocol as set out in the guidance manual. The terms of the ethical approval do not permit these types of alteration. Furthermore, such alterations might mean that the comparability of the survey would be compromised, and this could affect the calculation of performance indicators. If trusts want to make any minor changes to the method set out in this guidance, they must first check with the Co-ordination Centre that the proposed change would not compromise comparability.

Please direct questions or comments about this guidance to:

Delia Knox

Tel: 01494 560674


CONTENTS 1 In t roduct ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 The Healthcare Commission ...........................................................................1 1.2 How to use this guide .......................................................................................1 1.3 Basic requirements for national Mental Health Survey 2006 .....................2

2 T imetable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Commiss ion ing a su rvey f rom an Approved Cont ractor . . . . . . . 5

3.1 Model contract .................................................................................................5

4 Data pro tect ion and conf ident ia l i ty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4.1 Service users’ names and addresses.............................................................6 4.2 Confidentiality ...................................................................................................6 4.3 Service user anonymity ....................................................................................6 4.4 Storing completed questionnaires.................................................................7

5 E th ica l i s sues , e th ics commi t tees & research governance . . 9

5.1 Ethical approval for the mental health survey ............................................9 5.2 Research governance requirements ............................................................9

6 Pub l ic i s ing the su rvey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

7 Making su re your CPA l i s t i s re l iab le . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

8 Compi l ing a sample o f se rv ice user s fo r the su rvey . . . . . . . . . . 14

8.1 Compile a full list of current service users .................................................. 14 8.2 Data fields to include in the list of service users ....................................... 16 8.3 Taking a sample of service users ................................................................. 17 8.4 Check the sample list .................................................................................... 18 8.5 Ask your Chief Executive to sign off the sample ...................................... 19 8.6 Submit the list to the NHS Strategic Tracing Service (NSTS)..................... 20 8.7 Check the trust’s records for service user deaths .................................... 20 8.8 Organise the service user information into the sample file .................... 21 8.9 Sharing the service user sample file with an Approved Contractor..... 23 8.10 Using the sample file...................................................................................... 23

9 Implement ing the su rvey - p ract ica l i t ies . . . . . . . . . . . . . . . . . . . . . . . . . 24

9.1 Materials .......................................................................................................... 24 9.2 Setting up a FREEPOST address.................................................................... 25 9.3 IMPORTANT: Setting up a PO Box for mail returned undelivered........... 25 9.4 Setting up a FREEPHONE line........................................................................ 25 9.5 Covering letters .............................................................................................. 26 9.6 Multi-language sheet .................................................................................... 27 9.7 Sending out questionnaires.......................................................................... 28 9.8 Booking in questionnaires............................................................................. 29 9.9 Sending out reminders .................................................................................. 29

9.10 Comparing departments or teams within your trust................................ 30

10 Enter ing and coding ques t ionnai re data . . . . . . . . . . . . . . . . . . . . . . . . . . 32

10.1 Checking for errors ........................................................................................ 32 10.2 Submitting data to the Co-ordination Centre.......................................... 33 10.3 Checklist .......................................................................................................... 37

NHS mental health survey guidance 2006 Page 1

1 Introduction

1.1 The Healthcare Commission

The national patient survey programme is now being led by the Healthcare Commission. The Healthcare Commission exists to promote improvements in the quality of healthcare and public health in England. It is committed to making a real difference to the delivery of healthcare and to promoting continuous improvement for the benefit of patients and the public.

The Healthcare Commission has a statutory duty to assess the performance of healthcare organisations in the NHS and award annual ratings of performance, to coordinate inspections and reviews of healthcare organisations carried out by others, and register organisations providing healthcare in the independent sector on an annual basis. The Commission has created an entirely new approach to assessing and reporting on the performance of healthcare organisations - the annual health check – which will examine a much broader range of factors enabling us to focus on what really matters to patients and the public. The information from the survey of mental health service users will provide performance indicators for the annual health check.

1.2 How to use this guide

Trusts have the option of conducting the survey in house or using an Approved Contractor (see Section 3). Whichever you choose, you will need to address the guidance in Sections 1 to 8 of this guide. Sections 9 and 10 cover the practicalities of mailing out the survey, following-up responses and processing the data and submitting them to the Co-ordination Centre. These sections will be most relevant to Approved Contractors, or trusts undertaking the surveys themselves.


1.3 Basic requirements for national Mental Health Survey 2006

For comparisons between and within trusts to be accurate and fair, the surveys must be carried out using standard procedures in all trusts. Those standards are set out in detail later in this document. In summary, they are as follows:

• Tell the Co-ordination Centre who is carrying out your survey (i.e. whether it will be carried out by an Approved Contractor or in-house) by 16th January 2006. Give them contact details of two people in your trust who will be responsible for the survey’s progress (e-mail: [email protected]).

• Send two reminders to non-responders, even if a 40% response rate is already achieved.

• Follow the sampling procedure set out in this guidance. Work closely with the person in the information department who draws the sample, and check carefully that this guidance is followed exactly. See Section 8 - Compiling a sample of service users for the survey.

• Your Chief Executive must sign off the sample before you send out questionnaires, and you must send a signed declaration that the sample has been signed of. This must reach the Co-ordination centre by 10th February 2006.

• Send questionnaires to a random sample of 850 adults aged 16 and over on the trust’s Care Programme Approach (CPA) register. There is no upper limit on the age of service users to be included this year (in previous years, patients aged over 65 were excluded).

• Use the standard 8-page Mental Health Questionnaire 2006 downloaded from the new Mental Health surveys website at: www.healthcarecommission.org.uk/MentalHealthSurvey2006.

• Use only the standard covering letters downloaded from the new Mental Health surveys website at: www.healthcarecommission.org.uk/MentalHealthSurvey2006.

• A clean, checked data set, must be submitted to the Co-ordination Centre in the form outlined in Section 10.2 - Submitting data to the Co-ordination Centre by 5pm on 8th May 2006.

• Two copies each of the paper documents you used, (questionnaire and covering letters for each mailing must be submitted to the Co-ordination Centre in the form outlined in 10.2 - Submitting data to the Co-ordination Centre by 5pm on 8th May 2006.

• Check the data carefully for errors before submitting it to the Co-ordination Centre. See Section 10.1.


• Keep paper copies (or scanned pictures of all of the pages of the questionnaires, including the front and back pages) of all returned questionnaires until 31st August 2006 but do not send these to the Co-ordination Centre.


2 Timetable

Assuming no delays, it is reasonable to allow about 8-11 weeks to run the survey. The timetable below sets out important dates for the 2006 survey. If you commission an Approved Contractor, most of the work will be done by them, but all trusts have to be involved in some of the stages of the process, marked in bold below.

Latest date to complete Task See Section

16/01/2006 Inform Co-ordination Centre who is carrying out the survey and send contact details of 2 trust personnel

3/02/2006 Draw sample of service users 8 3/02/2006 Set up P.O. Box (if carrying out survey in-house) 9.3

3/02/2006 Print questionnaires & covering letters. Ensure you have enough headed paper, envelopes, return envelopes and labels 9.1 & 9.5

3/02/2006 Set up FREEPOST address and FREEPHONE line 9.2 & 9.4

10/02/2006 Ask Chief Executive to sign off sample and submit declaration to Co-ordination Centre

8.5

3/02/2006 Establish system for responding to telephone enquiries 9.4 3/02/2006 Establish system for booking in questionnaires 9.8

3/02/2006 Submit service user sample list to NSTS to check for deceased service users 8.6

24/02/2006 If using an Approved Contractor, supply them with trust headed paper, signature of senior executive and (if appropriate) signed honorary contract

9.1

03/03/2006 Check your own trust’s records again for service user deaths 8.7 03/03/2006 Send out first questionnaires 9.7

Mar-April 2006 Book in returned questionnaires 9.8 Mar-April 2006 Enter questionnaire data 10 Mar-April 2006 Respond to telephone enquiries 9.4

Mar-April 2006 Stick labels on pre-packed numbered questionnaires supplied by Approved Contractor (if NOT using honorary contract) 9.7

17/03/2006 Send out first reminders to non-responders 9.9 End of March

2006 Prepare for peak in phone calls as first reminders received 9.4

31/03/2006 Send out second reminders to non-responders 9.9 28/4/2006 Complete data entry 5/5/2006 Very Important: check data for errors 10.1

8/5/2006 Send data to Co-ordination Centre 10.2

8/5/2006 Send paper documents & checklist to Co-ordination Centre 10.2

Begin analysing trust's results and writing report

Disseminate results to staff, service users and the public


3 Commissioning a survey from an Approved Contractor

The framework agreement set up by the Department of Health covers the core survey process. Approved Contractors are expected to:

• Provide support to trusts on sampling

• Print questionnaires, covering letters and reminders and provide consumables

• Handle returned questionnaires, liaise with trusts on reminders

• Offer support to ensure good response rates. e.g. offer FREEPHONE line

• Enter and clean data entry and submit data to the Co-ordination Centre by the deadline

• Prepare reports for trusts.

Twelve organisations have been approved by the Healthcare Commission to carry out patient and service user surveys. Trusts may commission one of these contractors without further tendering for the survey work. Before committing to a contractor, you are advised to check exactly what is covered within the cost quoted.

Further information about each of these organisations, can be found on the new Mental Health surveys website at www.healthcarecommission.org.uk/MentalHealthSurvey2006.

3.1 Model contract

A model contract has been drawn up by the Healthcare Commission. By signing it, the survey contractor is obliged to keep the information confidential at all times, and to comply with the Data Protection Act 1998. The model contract also ensures that survey contractor staff members sign and abide by the honorary contract (see Section 4). The honorary contract is set up between the trust and those members of the survey contractor staff who will have access to service users’ information. The honorary contract describes how service users’ personal data will be sent to the survey contractor, and how the data can be used. It also ensures that only those members of staff named in the contract will have access to the data.

The model contract in Word format is available on the new Mental Health surveys website.


4 Data protection and confidentiality

You need to comply with the Data Protection Act 1998, and keep service users’ responses confidential. You also need to comply with the NHS Code of Practice on Confidentiality and Caldicott Guidance. As a part of this, you will need to take care that you meet any guarantees of anonymity or confidentiality made in covering letters and on the questionnaire form. It will also be necessary to establish appropriate contractual arrangements with any contactors. Your trust’s Caldicott Guardian and legal advisors will be able to advise you on confidentiality and data protection. This website has further information:

http://www.dh.gov.uk/PolicyAndGuidance/InformationPolicy/PatientConfidentialityAndCaldicottGuardians/fs/en

Further information about the Data Protection Act can be found at:

http://www.informationcommissioner.gov.uk

The Healthcare Commission intends to archive the survey data with the UK Data Archive after the analysis is completed.

4.1 Service users’ names and addresses

If you commission an Approved Contractor to carry out the survey, there are two common methods currently being practised by trusts and contractors, as advised by the Healthcare Commission:

Method 1

1. The trust allocates serial numbers to their list of sampled service users.

2. The contractor delivers pre-packed serial-numbered questionnaire packs to the trust.

3. The trust attaches number-matched address labels to the envelopes and posts them out.

4. Completed questionnaires are returned directly to the contractor and the contractor checks off the Record Numbers on returned questionnaires.

5. At reminder times, the contractor sends pre-packed serial-numbered questionnaire packs to the trust for those service users who have not yet returned questionnaires or opted out.

This process is described in more detail in Section 8 and 9.


Method 2

With the agreement of the trust’s Caldicott Guardian, an honorary contract is set up between the trust and two people who are already employed by the contractor. The contractor staff thus become unpaid employees of the trust (while continuing to be employees of the contractor) during the period in which the survey is carried out.

It is then permissible for the contracted employee to be given service user contact details for the purposes of sending out questionnaires and reminders to service users, and sticking address labels on to envelopes. The external contractor must be registered under the Data Protection Act and appropriate steps must be taken to protect service user confidentiality.

The Healthcare Commission has developed a set of honorary contracts for trusts and survey contractors carrying out the surveys. This is to ensure that the transfer of patient data is done in a way that does not compromise patient confidentiality, and to avoid the need for each trust to develop its own arrangements.

A Healthcare Commission sample honorary contract can be found on the new Mental Health surveys website at:


4.2 Confidential i ty

Any service user survey must respect confidentiality. That is, service users must be given reliable assurances that doctors, nurses, social workers and other trust staff will not be able to identify any individual’s responses. Furthermore, their responses must not be presented to anyone in a way that allows individuals to be identified. For example, if a service user is known to be under the care of a particular team, and his or her year of birth, sex and ethnic group are known from their survey responses, it might be possible to use this information to identify them. We recommend that responses should be aggregated into groups of no less than 30 service users before data are presented to staff.

4.3 Service user anonymity

In-house surveys

It is important to ensure that any claims you make about anonymity are accurate. In most cases where a survey is carried out in-house, it is not accurate to tell service users that their responses will be anonymous. The person who receives the completed questionnaires is usually able to match these responses to service user names and addresses.


Approved Contractors

Anonymity can sometimes be achieved if there is a clear separation between the information seen by an approved contractor and the information held by the trust. Service users’ names and addresses should be seen by trust staff only, while individual responses should be seen by contractor staff only. As long as the response data supplied to trusts do not include Service User Record Numbers and are not provided to trusts in a way that allows individuals to be identified, it can reasonably be claimed that responses are anonymous.

4.4 Storing completed quest ionnaires

Completed questionnaires must be stored in a separate location to lists of service users’ names. Similarly, the electronic file containing the service users’ names and addresses should be stored on a separate computer to that containing the survey data. Any mailing lists of service users’ names and addresses should be destroyed when the mailing process is complete.


5 Ethical issues, ethics committees & research governance

5.1 Ethical approval for the mental health survey

The Co-ordination Centre has obtained Multi-Centre Research Ethics Committee (MREC) approval for the standard Mental Health Questionnaire, the covering letters and reminder letters. You do not, therefore, need to seek ethical approval for this survey. Further information on the ethical approval process can be found at www.corec.org.uk.

The MREC letters can be downloaded from the mental health surveys website.

To comply with the ethical approval, the survey must be carried out according to the guidelines set out in this document.

5.2 Research governance requirements

The Healthcare Commission, as sponsor of this national survey, has taken steps to ensure that principles of research governance and ethics are followed thoroughly. The development of the survey methods, the questionnaire and covering letters to service users, have all been approved by an MREC. The questionnaire and guidance notes on how to conduct the survey are produced by the Mental Health Surveys Co-ordination Centre who are guided by peer reviewed research evidence in this area.

The Department of Health has confirmed to the Healthcare Commission that it would be inappropriate for individual trusts to follow the same local research governance processes as they would if the survey were a study the trust is to sponsor. As this national service user survey has MREC approval and the Healthcare Commission takes responsibility for it as sponsor, this would duplicate work and delay implementation unnecessarily.

Trusts are invited to give permission for the surveys to go ahead after confirming they have the local research governance arrangements to support this study.

The following table has been prepared by the Healthcare Commission. It is based on Section 3.10 of the Research Governance Framework for Health and social care (2nd Edition, 2005, available at http://www.dh.gov.uk/assetRoot/04/12/24/27/04122427.pdf). The left-hand column sets out the responsibilities of organisations providing care and the right-hand columns sets out the arrangements made by the Healthcare Commission for this survey. If you are required to seek approval from your research governance lead, you are advised to present this information to your R&D Manager in support of your request.


Responsibi l i t ies of organisat ions providing care

Research Governance Framework Healthcare Commission patient and service user surveys

• Retain responsibility for the quality of all aspects of participants’ care whether or not some aspects of care are part of a research study.

The survey is carried out on the experiences of service users after they have received the care so this does not apply.

• Be aware and maintain a record of all research undertaken through or within the organisation, including research undertaken by students as part of their training.

All Chief Executives are informed of the proposals of the survey.

• Ensure patients or users and carers are provided with information on research that may affect their care.

The survey does not affect the care of the service users. Anonymised results are used for performance rating and local quality improvement initiatives. Detailed guidance is issued to survey leads regarding the publicity of the results and its impact on care.

• Be aware of current legislation relating to research and ensure that it is implemented effectively within the organisation.

This requirement is not specific to this survey.

• Ensure that all research involving participants for whom they are responsible has ethical approval and that someone with the authority to do so has given written permission on behalf of the care organisation before each study begins.

The Healthcare Commission as sponsors of the study have sought ethics approval from MREC. There is a designated lead for each survey who is appointed by the Chief Executive.

• Ensure that no research with human participants, their organs, tissue or data, begins until an identified sponsor, who understands and accepts the duties set out in this framework, has confirmed it accepts responsibility for that research.

The Healthcare Commission as sponsors have undertaken steps to ensure that all the duties of the sponsors listed in section 3.8 of the Research Governance Framework are followed thoroughly.

• Ensure that written agreements are in place regarding responsibilities for all research involving an external partner, funder and/or sponsor, including agreement with the University or other employer in relation to student supervision.

A detailed guidance is issued to all the trusts, which spells out the responsibilities of all parties involved in the survey.

• Maintain the necessary links with clinical governance and/or best value

The guidance notes very strongly recommend the trusts to maintain these


Research Governance Framework Healthcare Commission patient and service user surveys

processes. links and follow best practice evidence. • Ensure that, whenever they are to

interact with individuals in a way, which has a direct bearing on the quality of their care, non-NHS employed researchers hold honorary NHS contracts and there is clear accountability and understanding of responsibilities.1

In situations where trusts opt to use the services of an external contractor to draw the sample for the survey, the contractor is required to enter into an honorary contract with the trust. These procedures are specifically detailed in the guidance notes.

• Put and keep in place systems to identify and learn from errors and failures.

The Healthcare Commission also undertakes consultations with the trusts in order to ensure that the errors and failures are reported back to the Healthcare Commission. The survey programme is constantly evaluated and reviewed in the light of these.

• Put and keep in place systems to process, address and learn lessons from complaints arising from any research work being undertaken through or within the organisation.

This requirement is not specific to this survey.

• Ensure that significant lessons learnt from complaints and from internal enquiries are communicated to funders, sponsors and other partners.

The Healthcare Commission maintains a helpline facility, which can be used by service users or trusts to report any complaints. Similar arrangements are in place with the NHS Patient Survey Co-ordination Centre who are commissioned by the Healthcare Commission to co-ordinate the patient and service user surveys.

• Ensure that any research-related adverse events are included in reports to the National Patient Safety Agency in line with the standard procedures of the organisation; or to the systems for adverse events reporting in social care.

Not applicable to the service user survey. Patient safety is not compromised, this being a postal survey.

• Permit and assist with any monitoring, auditing or inspection required by relevant authorities.

The results of the surveys are used for performance monitoring and national star rating mechanisms

1 When universities and hospitals employ staff on joint or dual contracts, they are expected to make joint arrangements for accountability and management. See A Review of Appraisal, Disciplinary and Reporting Arrangements for Senior NHS and University Staff with Academic and Clinical Duties, a report to the Secretary of State for Education and Skills by Professor Sir Brian Follett and Michael Paulson-Ellis, September 2001 (The Follett Report).


6 Publicising the Survey

You can heighten awareness of the survey and show the importance your trust places on service user feedback through the following channels:

• posters in outpatients departments

• direct communications with service users before admission or at discharge

• community newsletters

• A press release in the local media

Service users might ask clinical staff, patient liaison officers, or the Chief Executive's office about the survey, even though your covering letters give contact details for the survey managers and the dedicated helpline. Notify front line staff and executive offices that a survey is being conducted, and give them the name and number of a contact person. Survey managers should be prepared to respond to these calls quickly.

Template staff briefings and information for use in press releases can be downloaded from the new Mental Health surveys website.


7 Making sure your CPA list is reliable

For reliable comparisons to be made between trusts, it is essential that the definitions of Enhanced and Standard CPA are applied consistently.

Levels of need

To target resources effectively, and promote a consistent national approach, services are required to deliver the Care Programme Approach according to two levels of need:

• Standard; and

• Enhanced.

The characteristics of people on Standard CPA will include some of the following:

• they require the support or intervention of one agency or discipline or they require only low key support from more than one agency or discipline;

• they are more able to self-manage their mental health problems;

• they have an active informal support network;

• they pose little danger to themselves or others;

• they are more likely to maintain appropriate contact with services.

People on enhanced CPA are likely to have some of the following characteristics:

• they have multiple care needs, including housing, employment etc, requiring inter-agency co-ordination;

• they are only willing to co-operate with one professional or agency but they have multiple care needs;

• they may be in contact with a number of agencies (including the Criminal Justice System),

Account may also need to be taken of the definition of the levels given on page 53 of the National Service Framework for Mental Health (NSF).


8 Compiling a sample of service users for the survey

This section explains how to draw the sample of service users. This task will need to be carried out by a member of staff at the NHS Trust.

N.B. It is essential that the person who draws the sample understands the importance of following these instructions carefully. Also, that person’s line manager must give them the time and support they need to do the task properly.

Please read all of this section before you start to compile your service user list.

8.1 Compile a ful l l is t of current service users

• If you are in any doubt about these sampling instructions, please call the Co-ordination Centre on 01494 560677

• There is no upper limit on the age of service users to be included this year (in previous years, patients aged over 65 were excluded).

• Compile a full list of all service users on either standard or enhanced CPA who are seen between 1st September 2005 and 30th November 2005. Make sure you take out any duplicated names.

o Include all teams covered by your trust.

o If your trust does not already have all service users on one electronic list, you will need to combine all the separate lists so that you have one complete list.

o If some service users’ details are not held electronically, it will be necessary to type in their details, so that all service users’ details are held electronically.

o Remember to include service users on both standard and enhanced CPA.

If you do not have an adequate CPA list:

Call the Co-ordination Centre on 01494 560677

If you do not have adequate information on date of last contact:

Include all service users who have had a care review in the last 12 months – including on-going care in psychiatric outpatients, with a clinical psychologist, social worker or with another mental health professional.


Who to leave out

The following service users should be removed from the list:

• Anyone seen only once for an assessment (For example, those who were seen by a psychiatrist for a single assessment.)

• Any children who are under 16 on the date the sample was drawn

• Any service users who are known to be current inpatients

• Service users who are known to have died

• Service users who do not have a known UK address

• Service users who have not had contact with Mental Health Services in the last 3 months

• Any duplicated names

• Any alias names where the person’s real name is already included in the list.


8.2 Data f ields to include in the l ist of service users

The following information should be kept in a file in Microsoft Excel or Access:

• Service User Record Number2

• Title (Mr, Mrs, Ms, etc.)

• Initials (or First name)

• Surname

• Address Fields 3

• Postcode

• Year of birth

• Gender

• Ethnic group 4

• Day of the month of last contact (1 or 2 digits; e.g. 2 or 30)5

• Month of last contact (1 or 2 digits; e.g. 9 or 10)

• Year of last contact (4 digits; e.g. 2005)

• Whether the service user is on Enhanced or Standard CPA

• Any other details required by the NHS Strategic Tracing Service (NSTS). Wherever possible, this should include the NHS number. 6

• Unit or team7 8

2 This is a series of sequential numbers (for example, 1001 through to 1850). It will be included on address labels and on questionnaires. 3 The address should be held as separate fields (e.g. street, area, town, county, postcode). This should be consistent with the address format required by the NSTS. 4 This field should be included wherever possible. This data is required in order to evaluate non-response from different ethnic groups. This is in keeping with the aims of the Healthcare Commission and Department of Health to be more responsive to all ethnic groups and provide services that take account of their individual requirements. 5 Date fields must be supplied in separate columns. 6 The NHS number can give more accurate matching, especially if addresses are incomplete. It is advisable to liaise with the registered NSTS batch trace user (if this is not the same person who creates the sample list) to ensure that all the required fields are included in the list of service users (see Section 8.5 for more details on using the NSTS). 7 Include this if you plan to compare units within your trust 8 See Section 9 for further details


8.3 Taking a sample of service users

It is likely that your full list will include thousands of service users, but you will need to send questionnaires to only 850 service users. Taking a sample can only take place once you have compiled the full list of eligible service users as described in Section 8.2.

To select the 850 service users for your sample, you need to take a simple random sample of 850 service users from your full list. This means that each eligible service user in the full list (the population) has an equal chance of being included in the sample. The procedure for doing this is as follows:

1. Put the list of service users into a Microsoft Excel file. The first row of this file should be headings (Such as name, Year of birth, CPA level, etc). There should be one row for each service user below that.

2. Put the cursor in cell A1 and insert a blank column (Click Insert, then Columns). Give this column a heading: Random.

3. In cell A2, type =Rand()

When you press Enter, a random number between 0 and 1 will be generated in that cell.

4. Copy the formula from Cell A2 to all the other cells in that column, so that each service user has a random number.

5. You are now ready to sort the list of service users. Before you do this, it is essential that you select all columns in your list, including the header row. The simplest way of ensuring this it to click on the blank square above and to the left of Cell A1. This selects all cells in the worksheet. If you do not select all cells in your list, you could mix up service users’ details, or you could fail to sort some of the service users in the list.

6. Once all cells are highlighted, click Data, then Sort. At the bottom of the Sort dialogue box that comes up on the screen, ensure that <My data has Header row> is selected. At the top of that dialogue box, use the drop-down box under “Sort by” to select Random and click OK.

7. The data will then be sorted by the random number. Do not worry that the random numbers change when you sort them and they do not look as if they are sorted in any order.9 You can now select the top 850 service users in the file, as these will be a random selection of your service users. Note that to do this, you should delete rows below 851 in your file as the first row contains the header information.

9 The random numbers change each time any cell in the worksheet is changed.


Note

You are aiming for a response rate of at least 40%, which means that you should have at least 340 completed questionnaires. You will maximise your response rate by following this guidance carefully. It is not acceptable to try to boost the number of responses you receive by sending out questionnaires to a larger number of service users. The Co-ordination Centre will only be able to accept responses from the 850 service users in your list that have been correctly sampled.

8.4 Check the sample l ist

Once you have compiled your list of 850 service users, it is important to carry out a few final checks before sending the list to the NSTS:

• Remove duplications. Check your list to make sure service users’ names do not appear more than once, and remove any duplicated names. Where possible, link any alias names so that only one name per service user appears in the list.

• Postal addresses. Check again that there are no addresses that are outside the UK.

• Ages. Check again that all service users are aged 16 and over.

• Incomplete information on key name and address fields. Check for any records with incomplete information on key fields (surname and address) and try to find more complete information. If key name and address fields cannot be verified, remove those individuals from the sample. However, do not exclude anyone simply because you do not have a postcode for them, or because you could not get a matched trace with NSTS. Only remove a service user if there is insufficient name or address information for the questionnaire to have a reasonable chance of being delivered. The more cases that are removed at this stage, the poorer the sample coverage and the greater the danger of bias.

• Current inpatients. Check again that none of the service users are known to be current inpatients in your trust (or elsewhere, if known).

• Deceased service users. Check that the service users are alive. (See next section for further details.) Also check that your trust does not have a record of a service user’s death.

• Service user in last 3 months. Check that all service users have been seen in the last 3 months.

• Distribution of service user ages. Check that your sampled service users’ ages cover the full range of expected ages (aged 16 and over). You can do this by plotting the service user ages on a graph.


8.5 Ask your Chief Executive to s ign off the sample

For survey results to be comparable results across trusts, the definitions used in the sampling process must be applied identically across trusts. Therefore, Chief Executives are requested to sign off the sample before questionnaires are sent out. The purpose of this is to ensure that the sample has been drawn correctly and that the definitions of standard and enhanced CPA have been applied appropriately. The Chief Executive is asked to:

• Verify that the CPA status of sample members is complete and correct

• Confirm that the sampling has been undertaken according to the guidance for the survey

• Supply the trust’s implemented definition of enhanced CPA.

A paper copy of the signed declaration should be sent to the Co-ordination Centre at the following address by 10th February 2006.

Mental Health Surveys Co-ordination Centre PatientDynamics Riverside House 5 Nutfield Lane High Wycombe HP11 2ND


8.6 Submit the l ist to the NHS Strategic Tracing Service (NSTS)

Before sending out the questionnaires, the list of service users will also have to be checked for any deceased service users by the NHS Strategic Tracing Service (NSTS).

The NSTS contact details are as follows:

Help desk telephone number: 0121 788 4001

Website: http://www.connectingforhealth.nhs.uk/nsts

The time required to carry out the checks depends partly on how compatible the list you submit to the NSTS is with their system requirement. To avoid delay, check carefully that your list is in the correct format for NSTS.

The file returned from NSTS can be used to identify the records that need to be deleted from the sample file.

Full details on the process of submitting your details to the NHS Strategic Tracing Service (NSTS) can be found in the instruction manual:

Atos Origin NHS Patient Survey File Creation Guide

Note

Tracing services are not fool-proof and even after your service user list has been checked for deaths, some service users may die in the period between running the check and the questionnaire being delivered. You may find that some recently deceased service users remain in your sample. Special sensitivity is required when dealing with telephone calls from bereaved relatives.

8.7 Check the trust ’s records for service user deaths

Other valuable sources of information on service user deaths are your own trust’s records and the records of acute trusts in your area. It is essential that you check that your trust has no record of a service user having died at your trust. If you are able to check with your local acute trusts, you should also do this. Relatives are likely to be particularly offended if they receive a questionnaire or reminder from the trust where their relative died.


8.8 Organise the service user information into the sample f i le

Once the file is returned from the NSTS, you will need to keep the service user information in an electronic spreadsheet or database file, where you can record which questionnaires have been returned. Do not exclude service users just because NSTS could not find a match. At the end of the survey, you will send an anonymised version of this information to the Co-ordination Centre. (See Section 10.2 for details of how to do this.)

Firstly, you will need to add three new columns:

1. Service User Record Number. This field will be a series of consecutive numbers (for example, 1001 through to 1850).

2. The Outcome field will be used to record which questionnaires are returned to the freepost address, or are returned undelivered, or which service users opt out of the survey, etc.

3. The Comments column is useful for recording any additional information that may be provided when someone calls the FREEPHONE to inform you that the respondent has died or is no longer living at this address.

Table 1 shows part of an example Excel file comprising service user details. Only the fields headed in red italics should be included in the file sent to the Co-ordination Centre.

Table 1 – Sample Excel file of service user details

Serv

ice

User

Re

cord

Num

ber

Title

Initi

als

Surn

am

e

Ad

dre

ss1

Ad

dre

ss5

Post

cod

e

Year

of B

irth

Gen

der

Ethn

ic G

roup

Day

of la

st c

onta

ct

Mon

th o

f las

t co

ntac

t Ye

ar o

f of l

ast

cont

act

Leve

l of C

PA

Out

com

e

Com

men

ts

1001 Mrs AM Abbott --

AB1 1YZ 1971 2 1 2 10 2005 1 3 Informed that service user died

1002 Mr EC Ahmed --

AB2 6XZ 1958 1 3 14 11 2005 2 1

--

1849 Ms K Yoo --

AB4 7MX 1987 2 5 21 10 2005 1

1850 Ms F Young --

AB9 5ZX 1946 2 1 30 12 2005 1 1


Notes on Table1

• Service User Record Number. This number is unique for each service user. In the example, the numbers are in ascending order, starting at 1001 at the top of the list, through to 1850 at the bottom. The service user record number will be included on address labels and questionnaires. Later, when questionnaires are returned (whether completed or not), you (or the Approved Contractor) will use these numbers to monitor which service users have returned their questionnaires and to identify non-responders, who will need to be sent reminders. If an Approved Contractor is used, you need to agree with them the range of serial numbers that will be used.

• The Service User Record Number, Title, Initials, Surname, Address fields and Postcode are used for printing out address labels. You can use mail merge in a word processing package for this purpose.

• Year of birth is included in the form NNNN. The service user list should only include service users aged 16 and over.

• Gender should be coded as 1 = male and 2 = female. However, be aware that other systems may use a different coding.

• Ethnic Group should be coded using the broad categories 1 = White; 2 = Mixed; 3 = Asian or Asian British; 4 = Black or Black British; 5 = Chinese; 6 = any other ethnic Group. These are based on the standard categories introduced by the NHS Information Authority from 1st April 2001, but if your trust is using these categories, the data will need to be re-coded to these numeric codes.

Do NOT automatically code unknown ethnic groups as 6 – this group is only for those service users who are known not to belong to any of the other 5 named groups.

• The last contact Day, Month and Year are recorded in separate columns and formatted as general or numeric (rather than as dates).

• Level of CPA records whether the service user is on Standard or Enhanced CPA. It is coded 1 = Standard and 2 = Enhanced.

• The Comments column is useful for recording any additional information that may be provided if anyone informs you that the service user has died or is no longer living at this address.

• The Outcome field should be coded as follows:

1 = Returned useable questionnaire 2 = Returned undelivered by the mail service or service user moved house 3 = Service user died 4 = Service user reported too ill to complete questionnaire, opted out or

returned blank questionnaire 5 = Service user was not eligible to fill in questionnaire 6 = Questionnaire not returned (reason not known)


• The outcome column is left blank at first if the survey has not been returned, so it can be seen that Ms Yoo has not yet returned her questionnaire.

If the survey is being carried out in-house by the trust, you can use the file containing the service user name and address details to record the outcome information. If you are working with an Approved Survey Contractor, you should supply them with a list of record numbers (but service user names and addresses should be removed), against which they can record the outcome codes.

8.9 Sharing the service user sample f i le with an Approved Contractor

If you are working with an Approved Survey Contractor, but not using an honorary contract to share service users’ name and address details, you should supply them with a version of the list shown in Table 1 (with names and addresses removed). The contractor can use this list to record the outcome codes, and you should ensure that the contractor is kept up to date with any information that comes directly to the trust about service user deaths, or requests to be excluded from the survey sample, etc.

8.10 Using the sample f i le

This file has two purposes:

1. It will be used to keep a record of which service users have returned questionnaires so that reminders can be sent to them.

2. The anonymous data in this file (i.e. all the data except service user name and address information) will form part of the file that you will submit to the Co-ordination Centre when the survey is completed.

For confidentiality reasons, do not keep name and address details in the same file as survey response data. You will need to match up the anonymised service user information file with the data file once your survey is completed.


9 Implementing the survey - practicalit ies

9.1 Materials

Quest ionnaires

Each trust must use the standard 8-page Mental Health Questionnaire. This is available on the new Mental Health survey website at:


It is designed to fit on to eight sides of A4 paper and consists of questions, which cover the issues that have been found (through qualitative work with service users) to be the most important to service users.

It is practical to ensure that the number of pages in a questionnaire is a multiple of four so that sheets can be printed double-sided on A3 paper and folded to make an A4 booklet, stapled in the middle. If pages are stapled at the corner, pages are likely to become detached and get lost.

Number of questionnaires

You need to allow for sending out duplicate questionnaires with second reminders. Printing costs can be unnecessarily high if a second print-run is required, so it is worth ensuring that the first print-run is sufficiently large to allow for contingencies. Depending on the response rate to the first mailing, you should need approximately 1,500 copies.

Trust headed paper

You need trust headed paper for covering letters for the first and third mailing. (A reminder slip is used for the second mailing.) You need the same number of sheets of headed paper as questionnaires: i.e. approximately 1,500 sheets.

If an Approved Contractor is being used to carry out the survey work, it is preferable that the paper does not include a telephone number for the trust, as service users should call the contractor’s FREEPHONE line, rather than the trust switchboard

Other i tems

You also need:

• Large envelopes for mailing questionnaires to service user

• Labels for addressing envelopes


• Labels for sender address on reverse of envelopes

• FREEPOST envelopes for return of questionnaires

9.2 Sett ing up a FREEPOST address

A FREEPOST address allows service users to return completed questionnaires at no cost to themselves. After you have paid for the licence, you will only pay for the responses you receive. The FREEPOST address can be printed on the envelopes you send out with the questionnaires. Printed envelopes must comply with Royal Mail guidelines. Details of how to apply for a FREEPOST licence can be found at the Royal Mail website: http://www.royalmail.com. Alternatively, you can call your local Sales Centre on 0845 7950 950.

9.3 IMPORTANT: Sett ing up a PO Box for mail returned undelivered

When sending out questionnaires, you need to put a return address on the envelope so that undelivered mail can be returned without the need to open the envelope. If an Approved Contractor is used, their return address should be put on the envelope. If you are carrying out the survey in-house, you should set up a PO Box, so that the return address does not give an indication that the addressee has been in contact with mental health services. At time of going to press, the cost of running a PO Box is £43.80 for six months, and a further £43.80 to redirect post from a PO Box to another address. You can find out more about these services by contacting Royal Mail on Tel: 08457 740 740.

9.4 Sett ing up a FREEPHONE l ine

The covering letter to service users should include a telephone number for service users to call if they have any questions or complaints about the survey. You might want to set up a FREEPHONE line for this purpose. All staff who are likely to take calls should be properly briefed about the details of the survey, and be aware of the questions or complaints they are likely to receive. If you run the survey in-house, you might want to set up a FREEPHONE line for this purpose. Alternatively, many Approved Contractors offer this service.

Common quest ions and comments

• I have a specific comment, complaint or question about my care or treatment. Who can I contact at the trust?

Service users can be referred to the trust’s PALS, the complaints manager or other appropriate member of staff. Approved contractors should be given the contact details of the PALS office or an appropriate member of trust staff so that they can refer callers to that person.


• The person to whom the questionnaire is addressed is unable to understand the questionnaire.

Relatives or carers may call to pass on this information. In some cases, they may offer to complete the questionnaire for the service user, but this is only advisable if there is a good chance that the responses will be a true reflection of the service user’s views.

• The person to whom the questionnaire is addressed has died.

Even with the use of a deceased persons tracing service, it will not be possible to identify all deceased service users, particularly those who have died most recently. It is very important that staff who take the calls are aware of this possibility and are prepared to respond sensitively to such calls.

• I would like to take part but English is not my first language.

The Co-ordination centre is offering interpreter services via a FREEPHONE line. In all mailings to service users, a multi-language sheet will be included offering service users the chance to complete the questionnaire by phone or have queries answered in one of twenty other languages. Alternatively, if a service user’s spoken English is better than their written English, they may be willing to have someone fill in a form on their behalf over the telephone.

• I do not wish to participate in this survey.

A few service users might call to say that they do not want to be involved in the survey, and fewer still may object to being sent the questionnaire in the first place. Staff should apologise to the service user and reiterate the statement in the covering letter - that the survey is voluntary, and that their care will not be affected in any way by their not participating. It might be helpful to point out the purpose of the survey, and to emphasise the potential value of their responses. If the service user is willing to tell the staff member the identification number written on their survey, it might also be possible to prevent any further reminders being sent to that person. It is also advisable to ask the service user to ignore any future reminders that they might receive.

Making a record of the cal ls

Where appropriate, ask the service users who call to tell you their Service User Record number, which should be on the address label of the envelope they received, and on the questionnaire itself. You can then use this number to identify people who do not want to receive any further reminders.

9.5 Covering letters

The standard covering letter for the first mailing can be downloaded from the new Mental Health survey website at:



You can add your own trust’s details and print it on the trust's letterhead paper. It has been given ethical approval and alterations are not permitted. Please send two paper copies of the letter you use to the Co-ordination Centre when you submit your data.

9.6 Mult i- language sheet

The Co-ordination centre are providing twenty FREEPHONE language lines for service users. Using these FREEPHONE lines, service users can complete the questionnaire by phone, ask questions about the survey and exclude themselves from the survey. A multi-language sheet that gives service users the phone numbers to access interpreters in the correct language, must be included with all three mailings. This multi-language sheet can also be downloaded from the new Mental Health surveys website.


9.7 Sending out quest ionnaires

Mai l ing labels

Three mailing labels are needed for each service user. One set of labels will be used for the first mailing, one for the first reminder and one for the second reminder.

We recommend using the mail merge feature in a word processing package to create the mailing labels from the database of service user names and addresses. The Service User Record Numbers should be printed on the address labels, as they have to be matched with the numbers on the front of the questionnaires.

Quest ionnaire packs

The envelope sent to each service user at the first mailing should include the following:

1. A questionnaire marked with the Service User Record Number. This must also match (or correspond to) the number on the address label and the number on your electronic list of service user details.

2. The standard covering letter.

3. A large envelope, labelled with the FREEPOST address on it.

4. A multi-language sheet

These items should be packed into an envelope that has a return address on the outside. This should be the Approved Contractor’s address (if one is used), or an address that does not identify Mental Health Services as the sender. (See Section 9.3 - IMPORTANT: Setting up a PO Box for mail returned undelivered) for further details.)

Postage

The postage is likely to exceed the standard letter rate. It is essential that the appropriate postage rate is paid.

Approved contractors – no honorary contract

In this case, the Approved Contractor sends pre-packed questionnaires to the trust for mailing out. The envelopes should be clearly marked with the Service User Record Numbers so that trust staff can match these with their service user list and put on appropriate service user address labels. The return address is the PO Box.


Approved contractors – honorary contract

In this case, the Approved Contractor sends out questionnaires directly to service users, and the return address is the Approved Contractor’s address.

9.8 Booking in quest ionnaires

When questionnaires are received, match up the Record Numbers against the list of service users, so that you can record (in the outcome column) which service users have returned questionnaires and will not therefore need to be sent reminders. You need to keep paper copies (or scanned pictures of all of the pages of the questionnaires, including the front page) of any questionnaires that are returned to you until 31st August 2006, but please do not send these to the Co-ordination Centre.

Approved contractors

If an Approved Contractor carries out the work, questionnaires will be returned directly to them, so they will be able to record these returns against the list of Service User Record Numbers. Trusts should inform the contractor of any questionnaires that are returned to them, whether or not they have been completed, and of any service user who dies. The contractor then records these details to ensure that reminders are not sent out to those service users.

9.9 Sending out reminders

It is essential to get a good response rate. The minimum response rate for this survey is 40%, but it is expected that many trusts will achieve a higher response rate. It is essential that you send out both reminders, even if you already have achieved the minimum response rate.

First reminders should be sent out about 1 week after the first mailing. The second reminder sent out after a further 2-3 weeks after that.

Depending on the time that has elapsed since you first checked your service user list for deaths, it might be necessary to send your list to the tracing service for a further check before you send out reminders. In any case, before sending out reminders you should check your own trust’s records to check that there is no record of the service user’s death in your own trust.

Approved contractors

When reminders are due to be sent out, survey contractors should send the pre-packed envelopes bearing the Service User Record Numbers of the non-responders. Again, the envelopes should be clearly marked with the Service User Record Number so that trust staff can match these with their service user list and put on appropriate address labels.


First reminders

The standard first reminder slip, which can be downloaded from the new Mental Health surveys website, should be sent to service users who have not responded after one to two weeks. Service users should also be sent a multi-language sheet.

You can add your own trust’s details. It has been given ethical approval and alterations are not permitted. Please send two paper copies of the reminder slip you use to the Co-ordination Centre when you submit your data.

Second reminders

The standard covering letter for the second reminder can be downloaded from the new Mental Health surveys website. It should be sent to service users who have not responded after a further two to three weeks.

You can add your trust’s details and print it on trust headed paper. It has been given ethical approval and alterations are not permitted. Please send two paper copies of the letter you use to the Co-ordination Centre when you submit your data.

The second reminder envelopes should include the following:

1. A questionnaire marked with the Service User Record Number. This must also match (or correspond to) the number on the address label and the number on your electronic list of service user details.

2. A covering letter.

3. A large return envelope, labelled with the FREEPOST address on it.

4. A multi-language sheet.

9.10 Comparing departments or teams within your t rust

If you want to go beyond the minimum requirements, you could use the national survey as an opportunity to gather data about different teams within your trust. You could extend the number of service users you target, and ensure that you target sufficient numbers from each of the units you want to compare so that you can get enough responses to make comparisons.


Important note

If you choose to increase your sample size, it is essential that you ensure that the sample of service users you draw according to the requirements for the national survey can be easily distinguished from any additional service users you include in the sample. You need to send the data to the Co-ordination Centre from only the 850 service users sampled according to these guidelines.


10 Entering and coding questionnaire data

The data should be entered into the pre-designed Excel file, which can be found on the new Mental Health surveys website.

At the bottom of the Excel screen, there are labelled tabs for each of the worksheets within the workbook. The first of these tabs is labelled "Data". Click on this tab to show the data entry window. Data should be entered thus:

• Each row records one service user’s responses to the survey

• For each question, the small number next to the box ticked by the service user should be entered as the response.

• If a response is missing in a returned useable questionnaire, it should be left blank, or coded as a full stop (.). 10

• If two boxes are ticked (where only one should be ticked), the response should be left blank or coded as a full stop (.).

• When saving this file to submit data to the Co-ordination Centre, please save only the first sheet as a worksheet, rather than saving the whole file as a workbook. (This saves disk space.)

10.1 Checking for errors

The data need to be checked carefully for errors. That is:

1. Have the data been entered accurately? You can check this by double-entering the survey responses, and comparing the lines of data for any discrepancies. (For example, you can do this by subtracting each cell in one data sheet in Excel from a comparison sheet in the same workbook and comparing the results. If there are no differences between the two sheets, each cell will be zero, showing that the two sheets match.)

2. Are all the data entries valid responses for that question? For example, if a question allows three response options: "1", "2" or "3", check that your data do not include any other numbers. If out-of-range numbers are found, go back to the original questionnaire and check those responses.

Scanned data are also likely to contain errors and must be checked.

10 If you want to use this data input file on the website to display frequencies on the other pages of the workbook, you will need to fill in the blank cells (on rows where service users have returned questionnaires) with a full stop (.).


10.2 Submitt ing data to the Co-ordination Centre

The data must be submitted to the Co-ordination Centre in the appropriate format by the deadline of 8th May 2006. If an Approved Survey Contractor is used, they will be responsible for all of the data entry and checking, and when the survey is completed they should submit the data to the Co-ordination Centre in the correct format and supply the trust with an anonymised data set.

Fi le format

• Microsoft Excel Worksheet (not Workbook). Any version of Excel is acceptable.

• File name must be in the format <NHSTrustName>_MentalHealth2006.xls

• One row of data for each service user in the sample

• One column of data for each item of service user information or response.

• Missing data should be left blank or coded as a full stop (.). 11

Table 2 shows the information that must be provided for each of the 850 service users in the original sample.

11 Data may be missing because the service user skipped a question or set of questions by following instructions. Alternatively, a service user may have not answered for some other reason. However, all missing data should be left blank or coded as a full stop (.), regardless of the reason for the omission.


Table 2 - Data fields to include in file submitted to Co-ordination Centre

Field Format Codes & formats Comments Service User Record Number

NNNN The unique serial number allocated to each service user by the trust or Approved Survey Contractor

Year of Birth NNNN Gender N 1 = male

2 = female If gender is not known or unspecified, this field should be left blank or coded as a full stop (.).

Ethnic Group N

1 = White 2 = Mixed 3 = Asian or Asian British 4 = Black or Black British 5 = Chinese 6 = Other ethnic group

Ethnic group should be included if the information is available. Do NOT automatically code unknown ethnic groups as 6 – this group is only for those service users who are known not to belong to any of the other 5 named groups.

Day of last contact or last review

N or NN Format as a number, not a date

For example, if the service user last had contact on 16th Nov 2005, this column should read 16.

Month of last contact or last review

N or NN Format as a number, not a date


Year of last contact or last review

NNNN Format as a number, not a date


Level of CPA N 1 = Standard 2 = Enhanced

Outcome N

1 = Returned useable questionnaire 2 = Returned undelivered by the mail service or service user moved house 3 = Service user died 4 = Service user too ill, opted out or returned blank questionnaire 5 = Service user not eligible 6 = Questionnaire not returned (reason not known)

Outcome of sending questionnaire Remember to fill in all the blank cells with 6s when the survey is complete.

Responses to each of the questions

Each column must be clearly headed with the question number. Data should be coded using the numbers next to the response boxes on the printed surveys. There is no need to send the comments to the Co-ordination Centre.

N.B. To comply with the Data Protection Act, details that allow individuals to be identified must not be sent to the Co-ordination Centre.


Table 3 shows the data that should be included in the file you submit. It should have 850 rows (one for each service user included in your sample). There will be several blank cells in the response section of the file because the file includes a row for every service user in the sample, but you will not have responses from all of them.

Table 3 – Example of data file to be submitted to Co-ordination Centre Service user sample information Service user response information

Serv

ice

User

Re

cord

Num

ber

Year

of b

irth

Gen

der

Ethn

ic G

roup

Day

of la

st

cont

act

Mon

th o

f las

t co

ntac

t

Year

of o

f las

t co

ntac

t

Leve

l of C

PA

Out

com

e

Q1

Q2

Q3

--

Q56

Q57

Q58

1001 1971 2 1 2 10 2005 1 3

1002 1958 1 3 14 11 2005 2 1 3 1 2 2 3 11

1003 1964 2 1 3 10 2005 1 6

1004 1980 2 1 3 11 2005 1 1 2 1 3 4 4 1

1005 1951 2 1 3 12 2005 1 1 3 2 3 4 1

1006 1979 1 4 14 11 2005 2 6

1849 1987 2 5 21 10 2005 1 6

1850 1946 2 1 30 12 2005 1 1 3 1 1 3 4 1

You do not need to send the service users’ written comments to the Co-ordination Centre.

Addit ional informat ion

The following information should also be submitted at the same time as the data file:

• Contact details (telephone numbers and e-mail addresses) of at least two personnel who will be available to answer any queries about the data.

• Two blank paper copies each of the questionnaires you used, the covering letters and the reminder letters.


• A completed copy of the checklist on the following page.

Del ivery

Paper documents should be posted to the address below. Data may be sent on floppy disc or by e-mail:

Postal address:

Mental Health Surveys Co-ordination Centre PatientDynamics Riverside House 5 Nutfield Lane High Wycombe HP11 2ND


Date

The data must be submitted by 5pm on 8th May 2006.


10.3 Checklist

Before sending your data to the Co-ordination Centre, carry out the checks listed below, and include this checklist when you submit paper copies of the questionnaire and covering letters.

Check Done? Please initial

1. Your file name should follow the naming convention: <NHSTrustName>_MentalHealth2006.xls>

2. Save the data sheet only as an Excel worksheet (rather than a workbook).

3. All data are correct. 4. All values are in range. 5. Send data only for the 850 service users sampled according to the

sampling procedures defined in Section 8.

6. Check that you have included data for all questions. 7. Check that all the data are in numeric format only. 8. To comply with Data Protection regulations, any service user name

and address details must be removed before the file is sent to the Co-ordination Centre.

9. Remove any passwords from the data file 10. Include two paper copies of the questionnaire you used. 11. Include two paper copies of all 3 letters you used for the first mailing,

the second mailing and the third mailing.

12. Check again that all data are correct, and that all values are in range! SEE NOTE BELOW.

13. Include the names of two staff members who can answer questions about the data (at least 1 must be a member of trust staff who was involved in the sampling). The other may be another member of staff from the trust, or from the Approved Contractor.

Name 1:_______________________________

Name 2:_______________________________

Note

It is essential that these checks are carried out thoroughly. The Co-ordination Centre is not obliged to make corrections to data supplied by trusts or Approved Contractors. If incorrect data are submitted, it is likely that the data will be considered unreliable and will not be used by the Healthcare Commission in your trust’s performance ratings and those indicators will be set to a minimum. We cannot accept re-submissions of data after the deadline.

Survey of users of services 2006 Community mental health services This survey aimed to find out about the experiences of people using mental health services in the community. These services provide care to people who have been referred to a psychiatric outpatient clinic or local community mental health team. Surveys are an important way of gathering information about the experiences of patients and listening to their views. The results are used by NHS trusts to help them set priorities to ensure they are delivering a better service for patients. They are also used as part of the Healthcare Commission’s annual health check to help measure the quality of care being provided to patients. Results are also used to assess performance against the Department of Health’s targets on patient experience. Measuring and reporting experiences in a structured way helps ensure that improving experiences remains a priority for NHS trusts. This survey involved 79 NHS trusts, which provide secondary mental health services, including combined mental health and social care trusts, and primary care trusts that provide mental health services. Each trust selected a random sample of 850 adults, (aged 16 years and over) who were registered on the care programme approach and had used community mental health services between September 1st 2005 and November 30th 2005. The care programme approach is the process used by providers of mental health services to coordinate the care for people who have mental health problems. All the trusts involved in the survey have received detailed reports containing the results of their own survey benchmarked against national results. These reports are available, along with tables containing the percentage results for each question, and the results from previous surveys on the Healthcare Commission’s website: http://www.healthcarecommission.org.uk/NationalPatientSurveyProgramme. Here we highlight some of the key findings from the 2006 survey and describe how the experiences of patients has changed over time. Where appropriate, comparisons are made with results from the 2004 and 2005 surveys. The 2004 and 2005 surveys covered adults of working age (16-65), whereas the 2006 survey extended the age range to include those aged over 65. In order to allow fair comparison with previous years, this briefing note is based on those aged up to 65 only. In 2006, 19,494 self-completion questionnaires were returned from respondents aged 16-65, an overall response rate of 39%. This compares with 41% in 2005 and 42% in 2004. Although the differences reported in this briefing note are small, as the size of the sample is so large, any changes from previous surveys are almost certain to be real (and not a result of chance or the way the survey was conducted).

http://www.healthcarecommission.org.uk/NationalPatientSurveyProgramme

The survey covered a range of issues identified by users of services as important to them. This note summarises findings on: the care programme approach, relationships with healthcare professionals, medication, counselling, crisis management and support in the community.

The care programme approach The care programme approach was adopted in 1991 and provides a structure for mental health care. All users of services should be given a written copy of their care plan, which identifies their needs and explains their care. The care programme approach is classified into ‘standard’ and ‘enhanced’. The latter is aimed at those with more acute mental health problems. The percentages of people allocated to each type varies widely from trust to trust. Since the needs, and experiences, of the two groups may differ, some of the subsequent results are presented separately for each. Just over half (53%) of all respondents said they had been given or offered a copy of their care plan. This is an increase from 49% in 2004 and 50% in 2005. Forty five per cent of those on the standard care programme approach had been given (or offered) a copy of their care plan (compared with 41% in 2005). This figure was much higher among those on the enhanced care programme approach (70% in 2005 and 71% in 2006). Respondents who had been given or offered a copy of their care plan were asked about their understanding of, and involvement in, their care plan. The percentage of people who said that they definitely understood what was in their plan has increased from 47% in 2004, to 58% in 2005 and 2006. Of those who wanted to be involved in drawing up their plan 40% said that they definitely had been (a decrease from 42% in 2005), while 35% said that they had been involved to some extent and 25% said that they had not been involved. A care review is a forum for users of services to meet with healthcare professionals to discuss their care plan. The percentage of people who had at least one care review in the last 12 months increased from 47% in 2005 to 51% in 2006. Respondents on the enhanced care programme approach were more likely to report that they had a care review (71%) than those on the standard care programme approach (41%). Almost three quarters (73%) said that they were given the chance to speak with their care coordinator before the care review meeting (compared with 74% in 2005). A similar percentage (72%) of those who wanted to invite a friend or relative to the meeting were told that they could do so (an increase from 70% in 2005). The percentage that felt that they were definitely given the chance to express their views at the meeting increased from 66% in 2005 to 68% in 2006. There was also an increase in the percentage of people who said that they definitely found the meeting helpful (up from 47% in 2005 to 49% in 2006).

Relationships with healthcare professionals Care coordinator A care coordinator is the person responsible for managing the care that people receive from different sources, and for liaising with the users of services and other healthcare professionals. The percentage of respondents who were told who their care coordinator was has increased each year from 67% in 2004 to 69% in 2005 and 70% in 2006. As in previous years, those on the enhanced care programme approach were more likely to know who their care coordinator was (86%) than those on the standard care programme approach (62%). Among those respondents who knew who their care coordinator was, 71% said that they could always contact them if they had a problem, and 25% said that they could sometimes contact them if they had a problem. Psychiatrists The majority of respondents (84%) had seen a psychiatrist in the last 12 months. Of these, 75% had seen the same psychiatrist on their previous two appointments (no change from 2005 but an improvement from 73% in 2004). Sixty one per cent of respondents said that they definitely had trust and confidence in their psychiatrist (compared with 59% in 2005). Those who had seen the same psychiatrist for their previous two appointments (66%) were more likely than those who had seen two different psychiatrists (47%) to say that they definitely had trust and confidence in the psychiatrist. This is a slight improvement on 2005 where 64% of respondents who had seen the same psychiatrist for their previous two appointments definitely had trust and confidence in the psychiatrist. The percentage of people who felt that they had definitely been treated with respect and dignity by their psychiatrist has increased slightly each year, from 79% in 2004, to 80% in 2005 and 81% in 2006. Similarly, the proportion of people who felt that the psychiatrist had definitely listened carefully to them increased from 68% in 2004, to 69% in 2005, and 70% in 2006. Sixty four per cent of respondents reported that they were definitely given enough time to discuss their condition or treatment, compared with 60% in 2004 and 63% in 2005. Community psychiatric nurses A community psychiatric nurse (or community mental health nurse) is a nurse with specialist training in mental health who works in the community. Their role includes providing support and supervising medication. Almost three fifths (57%) of respondents had seen a community psychiatric nurse in the last 12 months. Of these, 80% felt that the community psychiatric nurse had definitely listened carefully to them. There was a slight increase between 2005 and 2006 in the percentage that definitely had trust and confidence in their community psychiatric nurse (from 73% to 74%), and the percentage reporting that they were definitely treated with respect and dignity (from 85% to 86%). Other healthcare professionals Over half (56%) of respondents had seen a healthcare professional other than a psychiatrist or community psychiatric nurse. Other professionals were categorised as social workers (seen by 29%), occupational therapists (seen by 17%), psychologists (seen by 23%), and ‘someone else’ (30% of responses to this question). Of these, a

higher percentage reported that they had definitely been treated with respect and dignity in the 2006 survey (86%) compared with 84% in 2005 and 83% in 2004. The percentage reporting that they had definitely been carefully listened to was the same in 2006 and 2005 (79%), having increased from 77% in 2004.

Medication As in 2004 and 2005, the majority of respondents (93%) had taken medication for their mental health problems in the last 12 months. Of these, the percentage who felt that they had definitely been involved in decisions about this medication has increased, from 39% in 2005 to 42% in 2006. Forty per cent of respondents felt that they were only involved to some extent, and 17% said that they were not involved at all. Respondents who had been prescribed any new medications in the last 12 months (45% of total) were asked if they were told about the purposes and possible side effects. Sixty three per cent reported that they were definitely told about the purposes of their new medication, and 27% said that they had been told to some extent. There has been a gradual improvement in the percentage of respondents reporting that they were definitely told about the purposes of their new medication rising from 60% in 2004 and 62% in 2005. Thirty eight per cent said that they were definitely told about possible side effects of their new medication, 28% said that they had been told to some extent, with 34% reporting they were not told. Again, there has been a gradual increase in the percentage of respondents reporting that they were definitely told about the possible side effects of their new medication rising from 36% in 2004 and 37% in 2005.

Counselling There has been a slight decrease in the percentage of respondents reporting that they had received counselling sessions (such as talking therapy) from NHS mental health services in the last 12 months falling from 42% in 2004 to 40% in 2005 and 39% in 2006. The percentage saying they wanted talking therapy is the same as in 2005 (57%) though a slight decrease compared with 2004 (59%). Of those respondents who had not received counselling sessions, over a third (35%) said they would have liked them. More than half (52%) of respondents who had received counselling (such as talking therapy) said that they had definitely found it helpful (the same figure as in 2005).

Crisis management Just under half (49%) of respondents said that they had the number of someone from their local NHS mental health services whom they could contact out-of-hours compared with 48% in 2005. Of those who had a contact number, over a third (39%) had called it in the last 12 months. Sixty two per cent of these said that they had spoken to someone immediately (a decrease from 64% in 2005), while 21% got through in an hour or less (down slightly from 22% in 2005). Eight per cent said that they could not get through to anyone.

Respondents who had used this service were asked whether they had received the help they wanted. Under half (44%) said that they had definitely received the help they wanted – a decrease from the 2005 figure (48%).

Support in the community A fifth of respondents (20%) had visited a day centre in the last 2 months. Of these respondents, 58% reported going once or twice a week, 24% once or twice a month and 18% visited on most days. Fifty six per cent felt that the activities provided by the day centre were definitely helpful, an increase of three percentage points compared with 2004 (53%) Less than half (48%) of those who had wanted information about local support groups had received it. Responses to this question are not directly comparable with the 2005 survey findings because of a change in question wording. Around a fifth of respondents (21%) were in paid work at the time of filling in the questionnaire. Half (50%) of those who would have liked help in finding work (from mental health services) said they had not received any help. Sixty eight per cent of those who wanted help getting benefits had received it. Responses to this question are not directly comparable with the 2005 survey findings because of a change in question wording. Overall views of care Overall, a quarter (26%) of respondents described the care they had received from mental health services in the last 12 months as excellent. Twenty eight per cent rated it as very good, and 23% as good. In total 77% of respondents reported their care as at least good, the same figure as in 2005. Five per cent rated their care as poor, and a further 4% as very poor (again, the same figures as in 2005). The rest (14%) said their care was fair. Further Information For the detailed results of all 79 NHS trusts that took part in the survey, please visit http://www.healthcarecommission.org.uk/NationalPatientSurveyProgramme A comprehensive report using the detailed results of this survey will be published later. Information about the Department of Health’s public service agreement target on patient experience may be found on the Department of Health website: http://www.dh.gov.uk/PublicationsAndStatistics/PublishedSurvey/NationalSurveyOfNHSPatients/fs/en

http://www.dh.gov.uk/PublicationsAndStatistics/PublishedSurvey/NationalSurveyOfNHSPatients/fs/en

http://www.dh.gov.uk/PublicationsAndStatistics/PublishedSurvey/NationalSurveyOfNHSPatients/fs/en

http://www.healthcarecommission.org.uk/NationalPatientSurveyProgramme

06_mental_health_performance_indicators_methodology Last saved 13/06/2006 16:22:00 1

Healthcare Commission (CHAI)

Community Mental Health Service User Survey 2006 This document outlines the method used by the Healthcare Commission to group and score the performance indicator questions included within the Community Mental Health Service User Survey, carried out by mental health NHS trusts in spring 2006. It also details the methodology used to calculate the overall scores for each individual trust, in terms of the five domains of patient experience used by the Department of Health (see Figure 1.1). In 2006, the survey was extended to include service users over the age of 65 years. To enable comparability with previous surveys, the performance indicators were calculated using only respondents aged 65 years and under. Figure 1.1: Domains of patient experience 2. Domains: Selected indicator questions The Community Mental Health Service User Survey consisted of 58 pre-coded questions, and one open-ended question designed to collect written comments and feedback. Of these, 40 questions were classified as being potential evaluative questions (that is, able to be scored) and were allocated to one of the five Department of Health patient experience domains. The criteria listed in Figure 2.1 were used to assess the suitability of each individual question, in terms of its viability as an indicator of performance. Using these criteria, 14 of the questions were then selected to be used in the calculation of the performance indicators. See Appendix 1 for the questions included within each domain.

Domains:

• Access and waiting • Safe, high quality, coordinated care • Better information, more choice • Building relationships • Clean, comfortable, friendly place to be* *Note: The “Clean, comfortable, friendly place to be” domain is not applicable

to the Community Mental Health Service User Survey

Deleted: 56

Deleted: 7

Deleted: Mental Health Survey 2006_construction document vs 3.doc


Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Figure 2.1 Criteria for selecting performance indicator questions: 3. Scoring individual indicator questions The indicator questions were scored using a scale of 0 to 100. A listing of scores assigned to the responses to each individual question is provided in Appendix 2. The scores represented the extent to which the patient’s experience could be improved. A score of 0 was assigned to all responses that reflected considerable scope for improvement, whereas an answer option that was assigned a score of 100 referred to a positive patient experience. Where options were provided that did not have any bearing on the trust’s performance in terms of patient experience, the responses were classified as “not applicable” and a score was not given. For example, where

Patient priorities

Questions should cover issues that are known to be

important to patients. Wide range of issues within domains

The questions should cover a broad range of topics and

services within each domain. Overlap

Items should be selected so there is minimal overlap with other questions included in the performance indictors.

Numbers of questions in each domain

There should be between 2 and 6 questions in each

domain for each survey.

Ease of evaluating responses

Questions should have clear/uncontroversial positive and negative response categories, and it should be clear that the topic covered is under the responsibility and range of influence of the Trust. Non-response

Questions should have low numbers of missing responses



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


respondents stated they could not remember or did not know the answer to the question a score was not given. For example, question 38 (see Figure 3.1) in the Community Mental Health Service User Survey asked whether the respondent had received any information about local support services. The option of “No” was allocated a score of 0, as this suggested that improvement to the patient experience was required. A score of 100 was assigned to the option “Yes” as it reflected a positive patient experience. If the patient said that they “did not need any information” then this option this was classified as ”not applicable” as this option is not a direct measure of whether they received information. Figure 3.1 Scoring example: Question 38 (Community Mental Health Service User Survey) Q38. In the last 12 months have you received any information about local support groups for mental health service users (e.g. MIND, Alzheimer’s Society, Rethink? Yes 100 No, but I would have liked information 0 I did not need any information Not applicable Where a number of options lay between the negative and positive responses, they were placed in appropriate positions along the scale. For example, question 4 in the Community Mental Health Service User Survey asked service users whether the psychiatrist listened carefully to them (see Figure 3.2). The options included:

Yes, definitely Yes, to some extent No

A score of 100 was assigned to “yes” as this is best practice in terms of patient experience. A response of “no” was given a score of 0, and so the remaining answer was assigned a score that reflected its position in terms of best practice, spread evenly across the scale. Hence the option “yes, to some extent” was assigned a score of 50.



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Figure 3.2 Scoring example: Question 4 (Community Mental Health Service User Survey) Q4, Did the psychiatrist listen carefully to you? Yes, definitely 100 Yes, to some extent 50 No 0 4. Methodology: Overall domain scores The scores for each domain in each trust were calculated using the method described in sections 4.1 to 4.5 below. To summarise, age-by-sex weightings were calculated to adjust for any variation between trusts that resulted from differences in the age and sex of respondents. A weight was calculated for each respondent by dividing the national proportion of respondents in their age-by-sex group by the corresponding trust proportion. As shown in section 4.4, the final domain score was calculated by dividing the sum of weighted scores for all eligible responses by the weighted number of eligible respondents for each question that contributed to a domain. The mean of these ‘question means’ was then taken as the trust score. The reason for weighting the data was that younger people and women tended to be more critical in their responses than older people and men. If a trust had a large population of young people or women, their performance might be judged more harshly than if there was a more consistent age/sex distribution. The exact stages are described in the next four sections. 4.1 Weighted analysis The first stage of the analysis involved calculating national age-by-sex proportions. It must be noted that the term “national proportion” is used loosely here as it was obtained from pooling the survey data from all trusts, and was therefore based on the respondent population rather than the entire UK population. The questionnaire asked respondents to state their year of birth. The approximate age of each respondent was then calculated by subtracting the figure given from 2006. The respondents were then grouped according to the categories shown in Figure 4.1.1. If a respondent did not fill in their year of birth or sex within the questionnaire, this information was copied from the sample file. If information on a respondent’s age and/or sex was missing from both the questionnaire and the sample file, the respondent was excluded from the analysis. The national age-by-sex proportions related to the proportion of men and women within each age group. With the Community Mental Health Service User Survey, as

Deleted: .



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


shown in Figure 4.1.1, the proportion of men aged 51-65 years was 0.1559, and the proportion of women aged 36-50 years was 0.2378. Figure 4.1.1 National Proportions (Community Mental Health Service User Survey) Sex Age Group National Proportion

16-35 0.097 36-50 0.178

Men

51-65 0.156 16-35 0.135 36-50 0.238

Women

51-65 0.196 The trust age-by-sex proportions were also calculated individually for each set of trust data, using the same procedure. The next step was to calculate the weighting for each individual’s responses. Age-by-sex weightings were calculated for each respondent by dividing the national proportion of respondents in their age-by-sex group by the corresponding trust proportion. If, for example, a lower proportion of men aged between 51 and 65 years within Trust A responded to the survey, in comparison with the national proportion, then this group would be under-represented in terms of the final scores. Dividing the national proportion by the trust’s proportion would result in a greater weighting for members of this group (see Figure 4.1.2). This would increase the influence of responses made by respondents within that group over the final score, thus counteracting the low representation. Figure 4.1.2 Proportion and Weighting for Trust A Sex Age

Group National Proportion

Trust A Proportion

Trust A Weight (National/Trust A)

16-35 0.097 0.104 0.933 36-50 0.178 0.109 1.633

Men

51-65 0.156 0.109 1.431 16-35 0.135 0.203 0.665 36-50 0.238 0.232 1.026

Women

51-65 0.196 0.243 0.807 Note: All proportions are given to three decimals places for this example. The analysis included these figures to nine decimal places Likewise, if a considerably higher proportion of women aged between 36 and 50 from Trust B responded to the survey (see Figure 4.1.3), then this group would be over-represented within the sample, compared with national representation of this group. Subsequently this age group would have a greater influence over the final score. To counteract this, dividing the national proportion by the proportion for Trust B would result in a lower weighting for members of this group, and would in effect reduce the disproportionate influence held by this group.



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Figure 4.1.3 Proportion and Weighting for Trust B Sex Age

Group National Proportion

Trust B Proportion

Trust B Weight (National/Trust B)

16-35 0.097 0.073 1.329 36-50 0.178 0.099 1.798

Men

51-65 0.156 0.197 0.792 16-35 0.135 0.096 1.406 36-50 0.238 0.271 0.878

Women

51-65 0.196 0.264 0.742 Note: All proportions are given to three decimals places for this example. The analysis included these figures to nine decimal places 4.2 Obtaining the numerators for each domain score The responses given by each respondent were entered into a dataset in terms of the 0-100 scale described in section 3. Each row corresponded to an individual respondent, and each column related to a performance indicator question. For those questions that the respondent did not answer (or received a missing score for), the relevant cell remained empty. Alongside these were the weightings allocated to each respondent (see Figure 4.2.1). Figure 4.2.1 Scoring for “Building closer relationships” domain, Trust A (Community Mental Health Service Users)

Question: Respondent 4 6 11 Weight

1 100 50 50 0.933 2 50 100 100 1.431 3 . . 50 1.026

Respondents’ scores for each question were then multiplied individually by the relevant weighting, in order to obtain the numerators for the domain scores (see Figure 4.2.2). Figure 4.2.2 Numerators for “Building closer relationships” domain, Trust A (Community Mental Health Service Users)

Numerators: Respondent 4 6 11 Weight

1 93.300 46.650 46.650 0.933 2 71.550 143.100 143.100 1.431 3 . . 51.300 1.026

Deleted: Respondent



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


4.3 Obtaining the denominators for each domain score A second dataset was then created. This contained a column for each question, with each row corresponding to an individual respondent. A value of one was entered for the questions to which a response had been given by the respondent, and all questions that had been left unanswered or allocated a scoring of “not applicable” (see section 3) were set to missing (see Figure 4.3.1). Figure 4.3.1 Values for non-missing responses, for “Building closer relationships” domain, Trust A (Community Mental Health Service Users)

Questions: Respondent 4 6 11 Weight

1 1 1 1 0.933 2 1 1 1 1.431 3 . . 1 1.026

The denominators were calculated by multiplying each of the cells within the second dataset by the weighting allocated to each respondent. This resulted in a figure for each question that the respondent had answered (see Figure 4.3.2). Again, the cells relating to the questions that the respondent did not answer (or received a “not applicable” score for) remained set to missing. Figure 4.3.2 Denominators for “Building closer relationships” domain, Trust A (Community Mental Health Service Users)

Denominators: Respondent 4 6 11 Weight

1 0.933 0.933 0.933 0.933 2 1.431 1.431 1.431 1.431 3 . . 1.026 1.026

4.4 Final calculation The final score for each domain was calculated via a two-step process. Firstly, weighted mean scores were calculated for each of the questions that contribute to the domain, by dividing the sum of the weighted scores for a question (i.e. numerators), by the weighted sum of all eligible respondents to the question (i.e. denominators) at each trust. A simple mean of each of these question means was then taken, to give the final trust domain score. Deleted: Mental Health Survey

2006_construction document vs 3.doc


Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Using the example data for trust A, we first calculated weighted mean scores for each of the three questions that contributed to the ‘Building closer relationships domain: Q4: 93.300+71.550 = 69.734 0.933+1.431 Q6: 46.650+143.100 = 80.266 0.933+1.431 Q11: 46.650+143.100+51.300 = 71.106 0.933+1.431+1.026 The final domain score would be given as a mean of these 3 values:

69.734+80.266+71.106 = 73.702 3 This process was repeated for each of the domains defined within the survey. Standard errors were calculated for each domain score using a standard bootstrapping technique (Effron, B. & Tibshirani, J. (1998) An Introduction to the Bootstrap Chapman & Hall / CRC, London)

Deleted: The final score for each domain was calculated by dividing the sum of the weighted scores for all eligible responses within the domain (i.e. numerators) by the weighted sum of all eligible respondents to the questions within each domain (i.e. denominators).¶¶Using the example of Trust A, the domain score based on the data from the three respondents would be calculated as follows:



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


4.5 Statistical techniques used in the patient survey performance indicators Calculation of the ZD scores Patient survey indicators in the 2005/06 Annual Health Check were banded using a slightly modified version of the methodology as that used in 2003/04 and 2004/05 star ratings. This method was based on a process of standardisation where a ZD score was calculated for each trust which related to the difference between the trust score, and the national mean score of all trusts. For 2005/06, the ZD scores are calculated for a combined overall domain score, which combines the five distinct domains into one overall score, and uses the pooled variance of the five scores. More technical details on the calculation of the ZD score can be found in appendix 3. In summary, the ZD score for a trust was calculated as the trust score minus the national mean score, divided by the standard error of the trust score plus the variance of the scores between trusts. This method of calculating a ZD score differs from the standard method of calculating a Z score in that it recognises that there is likely to be natural variation between trusts which one should expect, and accept. Rather than comparing each trust to one point only (ie the national mean score), it compares each trust to a distribution of acceptable scores. This is achieved by adding some of the variance of the scores between trusts to the denominator.

Deleted: the same

Deleted: For the 2003/04 star ratings, a new method has been used to band many of the indicators.

Deleted: ¶



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Appendix 1: Performance indicator questions, grouped within each domain Service User Survey Access and Waiting

Q23. In the last 12 months have you had any counselling sessions (e.g. talking therapy) from NHS Mental Health Services? (note: scoring conditional on response to Q24)

Q27. Can you contact your care coordinator if you have a problem? Safe, high quality, coordinated care

Q5. Did you have trust and confidence in the psychiatrist you saw?

Q9. The last 2 times you had an appointment with a psychiatrist, was it with the same psychiatrist both times or with two different psychiatrists?

Q12. Did you have trust and confidence in the Community Psychiatric Nurse (CPN)

Q42. Do you have the number of someone from your local NHS Mental Health Service that you can phone out of office hours? Better information, more choice

Q21. Were the purposes of the medications explained to you?

Q22. Were you told about possible side-effects of the medications?

Q38. In the last 12 months have you received any information about local support groups for mental health service users (e.g. MIND, Alzheimer’s Society, Rethink)?

Q52. Do you have enough say in decisions about your care and treatment?

Q53. Has your diagnosis been discussed with you?



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Building closer relationships

Q4. Did the psychiatrist listen carefully to you?

Q6. Did the psychiatrist treat you with respect and dignity?

Q11. Did the Community Psychiatric Nurse (CPN) listen carefully to you?



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Appendix 2: Scoring of individual indicator questions Community Mental Health Service User Survey Q4. Did the psychiatrist listen carefully to you? Yes, definitely 100 Yes, to some extent 50 No 0 Q5. Did you have trust and confidence in the psychiatrist you saw? Yes, definitely 100 Yes, to some extent 50 No 0 Q6. Did the psychiatrist treat you with respect and dignity? Yes, definitely 100 Yes, to some extent 50 No 0 Q9. The last 2 times you had an appointment with a psychiatrist, was it….? With the same psychiatrist both times 100 With two different psychiatrists 0 Q11. Did the CPN listen carefully to you? Yes, definitely 100 Yes, to some extent 50 No 0

Q12. Did you have trust and confidence in the CPN? Yes, definitely 100 Yes, to some extent 50 No 0 Q21. Were the purposes of the medications explained to you? Yes, definitely 100 Yes, to some extent 50 No 0 Q22. Were you told about possible side-effects of the medications? Yes, definitely 100 Yes, to some extent 50 No 0



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Q23 and Q24 One score is calculated from the answers to Q23 and Q24. Q23. In the last 12 months, have you had any counselling sessions (e.g. talking therapy) from NHS Mental Health Services? Yes No Q24. In the last 12 months did you want talking therapy? If response to Q23 is option 2, Q24 scored: Yes No If Q23 is yes (received talking therapy) and Q24 is yes, (wanted therapy) 100 If Q23 is no (did not receive talking therapy) and Q24 is no, (did not want therapy) 100 If Q23 is no (did not receive talking therapy) and Q24 is yes, (wanted therapy) 0 If Q23 is yes (received talking therapy) and Q24 is no, (did not want therapy) 0 Q27. Can you contact your care co-ordinator if you have a problem? Yes, always 100 Yes, sometimes 50 No 0 Q38. In the last 12 months have you received any information about local support groups for mental health service users (e.g. MIND, Alzheimer’s Society, Rethink? Yes 100 No, but I would have liked information 0 I did not need any information Not applicable Q42. Do you have the number of someone from your local NHS Mental Health Service that you can phone out of office hours? Yes 100 No 0 Not sure/Don’t know Not applicable Q52. Do you have enough say in decisions about your care and treatment? Yes, definitely 100 Yes, to some extent 50 No 0 Q53. Has you diagnosis been discussed with you? Yes, definitely 100 Yes, to some extent 50 No 0

Formatted

Deleted: If response to Q24 is option 1, Q23 is scored:

Deleted: 100

Deleted: 0

Deleted: 0

Deleted: 100



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


Appendix 3: Calculation of Z-statistics Z statistics (or Z scores) are standardized scores derived from normally distributed data, where the value of the Z score translates directly to a p-value. This p-value then translates to what level of confidence you have in saying that a value is significantly different from the mean of your data (or your ‘target’ value). For many of the indicators in the 2005/06 Annual Health Check, the banding method has been based on the use of Z scores. A standard Z score is calculated as:

i

ii s

yz 0θ−= (1)

where si is the pooled standard error of the pooled (sum of the four domain scores) trust mean score, yi is the pooled trust domain score, and θ0 is the pooled national mean score (the target against which the trusts are being judged). From this point on, where we refer to ‘trust mean score’, ‘standard error’ or ‘national mean’, either in the text or symbolically in the equations, we are referring to the pooled scores for all four domains. For measures where there is a high level of precision (the survey indicators sample sizes average around 400 to 500 per trust) in the estimates, the standard Z score may give a disproportionately high number of trusts in the significantly above/ below average bands (because si is generally so small). This is compounded by the fact that you cannot control for all the factors that may affect a trust’s score. For example, if trust scores are closely related to economic deprivation then there may be significant variation between trusts due to this factor, not necessarily due to factors within the trusts’ control. In this situation, the data are said to be ‘over dispersed’. This problem can be partially overcome by the use of an ‘additive random effects model’ to calculate the Z score (we refer to this modified Z score as the ZD score). Under this model, we accept that there is natural variation between trust scores, and this variation is then taken into account by adding this to the trust’s local standard error in the denominator of (1). In effect, rather than comparing each trust simply to one national target value, we are comparing them to a national distribution. The steps taken to calculate ZD scores are outlined below, but for a more detailed explanation please refer to the ‘explanation of statistical methods’ document in the ‘more information’ section of the 2005/06 ratings website. Please note however that some of the formulae in this document differ from those in the methods document, because we are dealing with mean values rather than proportions or standardized rates. Winsorising Z-scores The first step when calculating ZD is to ‘winsorise’ the standard Z scores (from (1)). Winsorising consists of shrinking in the extreme Z-scores to some selected percentile, using the following method.

1. Rank cases according to their naive Z-scores. 2. Identify Zq and Z1-q, the 100q% most extreme top and bottom naive Z-scores,

where q might, for example, be 0.1.

Deleted: Page Break

Deleted: Under this scheme, a trust with a Z score of < -3.1 is placed in band 1 (

Deleted: significantly below average

Deleted: failed; p<0.001 that the trust score is below the national average), -3.1 < Z < -1.96 in band 2 (

Deleted: below average

Deleted: under achieved; p<0.025 that the trust score is below the national average), -1.96 < Z

Deleted: Z < 1.96

Deleted: in band 3 (

Deleted: average

Deleted: achieved

Deleted: ), 1.96 < Z < 3.1 in band 4 (above average; p<0.025 that the trust score is above the national average) and Z > 3.1 in band 5 (significantly above average; p<0.001 that the trust score is above the national average).

Deleted: ).

Deleted: However,

Deleted: because f



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43


3. Set the lowest 100q% of Z-scores to Zq, and the highest 100q% of Z-scores to Z1-q. These are the Winsorised statistics.

This retains the same number of Z-scores but discounts the influence of outliers. For 2005/06, a consistent winsorising factor of 10% was selected for use with any indicators banded using this method.

Estimation of over-dispersion An over dispersion factor φ̂ is estimated which allows us to say if the data are over dispersed or not:

∑=

=I

iizI 1

21φ̂ (2)

where I is the sample size (number of trusts) and zi is the Z score for the ith trust given by (1). The winsorised Z scores are used in estimating φ̂ . If φ̂ is less than 1+ (2 × √(2 / I)) then the data are not over-dispersed, and we can simply use (1) to calculate standard Z scores.

An additive random effects model If Iφ̂ is greater than (I - 1) then we need to estimate the expected variance between trusts. We take this as the standard deviation of the distribution of θi (trust means) for trusts which are on target, we give this value the symbol τ̂ .τ̂ may is estimated using the following formula:

∑ ∑∑−−−

=i i ik ii www

II2

2 )1(ˆˆ φτ (3)

where wi = 1 / si

2 and φ̂ is from (2). Once τ̂ has been estimated, the ZD score is calculated as:

22

0

τ̂

θ

+

−=

sy

zi

iDi (4)

For a worked example using this method, please refer to the ‘explanation of statistical methods’ document in the ‘more information’ section of the 2005/06 ratings website. Please note however, that where in that si

0 is used, we simply use si for the surveys data.

Deleted: (



Deleted: 12/06/2006 12:27

Deleted: 12/06/2006 12:24

Deleted: 08/06/2006 12:43

National Survey Programme. MH Questionnaire 65+. Dec 2005 v8 Page 1

Service User Questionnaire

What is the survey about?

This survey is about the health services you receive from the National Health Service.

Who should complete the questionnaire?

The questions should be answered by the person named on the front of the envelope. If that person needs help to complete the questionnaire, the answers should be given from his/her point of view – not the point of view of the person who is helping.

Completing the questionnaire

For each question please tick clearly inside one box using a black or blue pen. Sometimes you will find the box you have ticked has an instruction to go to another question. By following the instructions carefully you will miss out questions that do not apply to you. Don’t worry if you make a mistake; simply cross out the mistake and put a tick in the correct box. Please do not write your name or address anywhere on the questionnaire.

Questions or help?

If you have any queries about the questionnaire, please call the helpline number given in the letter enclosed with this questionnaire.

Your participation in this survey is voluntary.

If you choose to take part, your answers will be treated in confidence.


YOUR CARE AND TREATMENT

1. How long have you been in contact with NHS mental health services?

1 1 year or less Go to 2

2 1 to 5 years Go to 2


4 More than 10 years Go to 2

5 Don’t know/ Can’t remember Go to 2

6 I have never been in contact with mental health services

Go to Question 54 on Page 7

2. When was the last time you saw someone from

the NHS mental health services?

1 In the last week

2 More than 1 week but less than 1 month ago

3 1-3 months ago

4 4-6 months ago

5 More than 6 months ago

HEALTH PROFESSIONALS

Psychiatrists

3. Have you seen a psychiatrist in the last 12 months?

1 Yes Go to 4

2 No Go to 10

The LAST time you saw a psychiatrist…

4. Did the psychiatrist listen carefully to you?

1 Yes, definitely

2 Yes, to some extent

3 No

5. Did you have trust and confidence in the psychiatrist you saw?

1 Yes, definitely


3 No

Still thinking about the LAST time you saw a psychiatrist…

6. Did the psychiatrist treat you with respect and dignity?

1 Yes, definitely


3 No

7. Were you given enough time to discuss your condition and treatment?

1 Yes, definitely


3 No

8. In the last 12 months, have any of your appointments with a psychiatrist been cancelled or changed to a later date?

1 No

2 Yes, 1 appointment was cancelled or changed

3 Yes, 2 or 3 appointments have been cancelled or changed

4 Yes, 4 or more appointments have been cancelled or changed

9. The last 2 times you had an appointment with a psychiatrist, was it…?

1 With the same psychiatrist both times

2 With two different psychiatrists


Community Psychiatric Nurse (CPN)

10. Have you seen a CPN in the last 12 months?

1 Yes Go to 11

2 No Go to 14

The LAST time you saw a CPN…

11. Did the CPN listen carefully to you?

1 Yes, definitely


3 No

12. Did you have trust and confidence in the

CPN?

1 Yes, definitely


3 No

13. Did the CPN treat you with respect and dignity?

1 Yes, definitely


3 No

Other health professionals (e.g. a social worker, occupational therapist, or a psychologist)

14. Have you seen anyone else from mental health

services in the last 12 months?

1 Yes Go to 15

2 No Go to 18

15. The last time you saw someone, other than a psychiatrist or CPN, who did you see? (Tick ONE only)

1 A social worker

2 An occupational therapist

3 A psychologist

4 Someone else

The LAST time you saw this person…

16. Did the person listen carefully to you?

1 Yes, definitely


3 No

17. Did the person treat you with respect and

dignity?

1 Yes, definitely


3 No

MEDICATIONS

18. In the last 12 months have you taken any medications for your mental health problems?

1 Yes Go to 19

2 No Go to 23

19. Do you have a say in decisions about the medication you take?

1 Yes, definitely


3 No


20. In the last 12 months, have any new medications (e.g. tablets, injections, liquid medicines, etc.) been prescribed for you by a psychiatrist?

1 Yes Go to 21

2 No Go to 23

3 Can’t remember Go to 23

The LAST time you had a new medication prescribed for you…

21. Were the purposes of the medications explained to you?

1 Yes, definitely


3 No

22. Were you told about possible side effects of

the medications?

1 Yes, definitely


3 No

COUNSELLING

23. In the last 12 months have you had any counselling sessions (e.g. talking therapy) from NHS Mental Health Services?

1 Yes

2 No

24. In the last 12 months, did you want talking

therapy?

1 Yes

2 No

25. If you had any talking therapy from NHS Mental Health Services in the last 12 months, did you find it helpful?

1 Yes, definitely


3 No

4 I did not have any talking therapy

YOUR CARE CO-ORDINATOR

A Care Co-ordinator (or keyworker) is someone from Mental Health Services who keeps in regular contact with you. For example, this person could be a Community Psychiatric Nurse (CPN), a Psychiatrist or a Social Worker. 26. Have you been told who your Care Co-

ordinator is?

1 Yes Go to 27

2 No Go to 28

3 Not sure/ Don’t know Go to 28

27. Can you contact your Care Co-ordinator if you have a problem?

1 Yes, always

2 Yes, sometimes

3 No

YOUR CARE PLAN

A care plan shows your mental health needs and who will provide services for you. It might be a document given to you by one of the mental health team, or it might be a letter, explaining how your care has been planned. 28. Have you been given (or offered) a written or

printed copy of your care plan?

1 Yes

2 No

3 Don’t know/ Not sure


29. Do you understand what is in your care plan?

1 Yes, definitely Go to 30

2 Yes, to some extent Go to 30

3 No, I don’t understand it Go to 30

4 Not sure Go to 30

5 I do not have a care plan Go to 31

30. Were you involved in deciding what was in your care plan?

1 Yes, definitely


3 No

4 I did not want to be involved

YOUR CARE REVIEW

A care review is a meeting with you and the people involved in your care in which you discuss how your care plan is working. 31. In the last 12 months have you had a care

review?

1 Yes, I have had more than one Go to 32

2 Yes, I have had one Go to 32

3 No, I have not had a care review in the last 12 months Go to 36

4 Don’t know / Can’t remember Go to 36 32. Were you told that you could bring a friend or

relative to your care review meetings?

1 Yes

2 No

3 Don’t know / Can’t remember

4 I did not want to invite a friend or relative

33. Before the review meeting, were you given a chance to talk to your care co-ordinator about what would happen?

1 Yes

2 No

3 Don’t know / Can’t remember

The LAST time you had a care review meeting…

34. Were you given a chance to express your views at the meeting?

1 Yes, definitely


3 No

35. Did you find the care review helpful?

1 Yes, definitely


3 No

SUPPORT IN THE COMMUNITY

Day centres or day hospitals

Some mental health service users go to a day centre where staff are available to help with problems, and activities are arranged. 36. In the last 2 months, how often have you visited

a day centre?

1 Most days Go to 37

2 Once or twice a week Go to 37

3 Once or twice a month Go to 37

4 I have not visited a day centre in the last 2 months Go to 38

37. Were the activities provided by the centre helpful?

1 Yes, definitely


3 No

38. In the last 12 months have you received any information about local support groups for mental health service users (e.g. MIND, Alzheimer’s Society, Rethink?

1 Yes

2 No, but I would have liked information

3 I did not need any information


Other support in the community

39. Are you currently in paid work? (Tick ONE only)

1 Yes Go to 40

2 No Go to 40

3 No, I am retired Go to 41

4 No, but I work on a casual or voluntary basis Go to 40

5 No, but I am a full-time student

Go to 40

40. In the last 12 months have you received help with finding work?

1 Yes

2 No, but I would have liked help

3 I did not need any help

4 I am unable to work because of my mental health problems

41. In the last 12 months have you received help

with getting benefits (e.g. Housing Benefit, Attendance allowance)?

1 Yes

2 No, but I would have liked help

3 I did not need any help

CRISIS CARE

42. Do you have the number of someone from your local NHS Mental Health Service that you can phone out of office hours?

1 Yes Go to 43

2 No Go to 46

3 Not sure/ Don’t know Go to 46

43. In the last 12 months, have you called this

number?

1 Yes Go to 44

2 No Go to 46

44. The last time you called the number, how long

did it take you to get through to someone?

1 I got through immediately

2 I got through in one hour or less

3 A few hours

4 A day or more

5 I could not get through to anyone

45. The last time you called the number, did you

get the help you wanted?

1 Yes, definitely


3 No

STANDARDS

46. Have you been admitted to a hospital as a mental health patient in the last 12 months?

1 No

2 Yes, once

3 Yes, 2 or 3 times

4 Yes, more than 3 times

Mental Health Act 47. In the last 12 months, have you been detained

(sectioned) under the Mental Health Act?

1 Yes Go to 48

2 No Go to 49

48. When you were sectioned, were your rights explained to you?

1 Yes, completely


3 No

4 Not sure/ Don’t know


YOUR FAMILY OR CARER

49. Has a member of your family or someone else close to you been given enough information from health and social services about your mental health problems?

1 Yes, definitely


3 No, but they would have liked some information

4 No, but they got information from somewhere else

5 No information was needed

50. Has a member of your family or someone else close to you had enough support from health and social services?

1 Yes, definitely


3 No, they have not had any support from health and social services

4 No support was needed

OVERALL

51. Overall, how would you rate the care you have received from Mental Health Services in the last 12 months?

1 Excellent

2 Very good

3 Good

4 Fair

5 Poor

6 Very poor

52. Do you have enough say in decisions about your care and treatment?

1 Yes, definitely


3 No

53. Has your diagnosis been discussed with you?

1 Yes, definitely


3 No

54. Who was the main person or people that filled in this questionnaire?

1 The service user/client (named on the front of the envelope)

2 A friend or relative of the service user/client

3 Both service user/client and friend/relative together

4 The service user/client with the help of a health professional

ABOUT YOU

55. Are you male or female?

1 Male

2 Female

56. What was your year of birth?

(Please write in) e.g. 1 9 3 4

1 9

57. In general, how is your mental health right now?

1 Excellent

2 Very good

3 Good

4 Fair

5 Poor

6 Very poor


58. To which of these ethnic groups would you say you belong? (Tick ONE only)

a. WHITE

1 British

2 Irish 3 Any other White background

(Please write in box)

b. MIXED

4 White and Black Caribbean

5 White and Black African

6 White and Asian

7 Any other Mixed background (Please write in box)

c. ASIAN OR ASIAN BRITISH

8 Indian

9 Pakistani

10 Bangladeshi

11 Any other Asian background (Please write in box)

d. BLACK OR BLACK BRITISH

12 Caribbean

13 African

14 Any other Black background (Please write in box)

e. CHINESE OR OTHER ETHNIC GROUP

15 Chinese

16 Any other ethnic group (Please write in box)

OTHER COMMENTS

If there is anything else you would like to tell us about your experiences of mental health care in the last 12 months, please do so here.

THANK YOU VERY MUCH FOR YOUR HELP Please check that you answered all the

questions that apply to you. Please post this questionnaire back in the

FREEPOST envelope provided. No stamp is needed.

Is there anything particularly good about your care? Is there anything that could be improved? Any other comments?


Copyright of 2006 Community Mental Health Service Users Questionnaire The questionnaire may be reproduced free of charge in any format or medium provided that they are not for commercial resale. This consent is subject to the material being reproduced accurately and provided that it is not used in a derogatory manner or misleading context. The material should be acknowledged as © 2006 Commission for Healthcare Audit and Inspection and the title of the document specified. Applications for reproduction of the questionnaire should be made in writing to: The Surveys Team Commission for Healthcare Audit and Inspection 103-105 Bunhill Row London EC1Y 8TG e-mail: [email protected]


Service User Questionnaire

What is the survey about?

This survey is about the health services you receive from the National Health Service.

Who should complete the questionnaire?

The questions should be answered by the person named on the front of the envelope. If that person needs help to complete the questionnaire, the answers should be given from his/her point of view – not the point of view of the person who is helping.

Completing the questionnaire

For each question please tick clearly inside one box using a black or blue pen. Sometimes you will find the box you have ticked has an instruction to go to another question. By following the instructions carefully you will miss out questions that do not apply to you. Don’t worry if you make a mistake; simply cross out the mistake and put a tick in the correct box. Please do not write your name or address anywhere on the questionnaire.

Questions or help?

If you have any queries about the questionnaire, please call the helpline number given in the letter enclosed with this questionnaire.

Your participation in this survey is voluntary.

If you choose to take part, your answers will be treated in confidence.


YOUR CARE AND TREATMENT

1. How long have you been in contact with NHS mental health services?

1 1 year or less Go to 2



4 More than 10 years Go to 2

5 Don’t know/ Can’t remember Go to 2

6 I have never been in contact with mental health services

Go to Question 54 on Page 7

2. When was the last time you saw someone from

the NHS mental health services?

1 In the last week

2 More than 1 week but less than 1 month ago

3 1-3 months ago

4 4-6 months ago

5 More than 6 months ago

HEALTH PROFESSIONALS

Psychiatrists

3. Have you seen a psychiatrist in the last 12 months?

1 Yes Go to 4

2 No Go to 10

The LAST time you saw a psychiatrist…

4. Did the psychiatrist listen carefully to you?

100 1 Yes, definitely

50 2 Yes, to some extent

0 3 No

5. Did you have trust and confidence in the psychiatrist you saw?



0 3 No

Still thinking about the LAST time you saw a psychiatrist…

6. Did the psychiatrist treat you with respect and dignity?



0 3 No

7. Were you given enough time to discuss your condition and treatment?



0 3 No

8. In the last 12 months, have any of your appointments with a psychiatrist been cancelled or changed to a later date?

100 1 No

67 2 Yes, 1 appointment was cancelled or changed

33 3 Yes, 2 or 3 appointments have been cancelled or changed

0 4 Yes, 4 or more appointments have been cancelled or changed

9. The last 2 times you had an appointment with a psychiatrist, was it…?

100 1 With the same psychiatrist both times

0 2 With two different psychiatrists


Community Psychiatric Nurse (CPN)

10. Have you seen a CPN in the last 12 months?

1 Yes Go to 11

2 No Go to 14

The LAST time you saw a CPN…

11. Did the CPN listen carefully to you?



0 3 No

12. Did you have trust and confidence in the

CPN?



0 3 No

13. Did the CPN treat you with respect and dignity?



0 3 No

Other health professionals (e.g. a social worker, occupational therapist, or a psychologist)

14. Have you seen anyone else from mental health

services in the last 12 months?

1 Yes Go to 15

2 No Go to 18

15. The last time you saw someone, other than a psychiatrist or CPN, who did you see? (Tick ONE only)

1 A social worker

2 An occupational therapist

3 A psychologist

4 Someone else

The LAST time you saw this person…

16. Did the person listen carefully to you?



0 3 No

17. Did the person treat you with respect and

dignity?



0 3 No

MEDICATIONS

18. In the last 12 months have you taken any medications for your mental health problems?

1 Yes Go to 19

2 No Go to 23

19. Do you have a say in decisions about the medication you take?



0 3 No


20. In the last 12 months, have any new medications (e.g. tablets, injections, liquid medicines, etc.) been prescribed for you by a psychiatrist?

1 Yes Go to 21

2 No Go to 23

3 Can’t remember Go to 23

The LAST time you had a new medication prescribed for you…

21. Were the purposes of the medications explained to you?



0 3 No

22. Were you told about possible side effects of

the medications?



0 3 No

COUNSELLING

23. In the last 12 months have you had any counselling sessions (e.g. talking therapy) from NHS Mental Health Services?

1 Yes

2 No

24. In the last 12 months, did you want talking

therapy?

1 Yes

2 No

If Q23=1 and Q24=1 then score 100 If Q23=2 and Q24=2 then score 100 If Q23=2 and Q24=1 then score 0 If Q23=1 and Q24=2 then score 0

25. If you had any talking therapy from NHS Mental Health Services in the last 12 months, did you find it helpful?



0 3 No

- 4 I did not have any talking therapy

YOUR CARE CO-ORDINATOR

A Care Co-ordinator (or keyworker) is someone from Mental Health Services who keeps in regular contact with you. For example, this person could be a Community Psychiatric Nurse (CPN), a Psychiatrist or a Social Worker. 26. Have you been told who your Care Co-

ordinator is?

100 1 Yes Go to 27

0 2 No Go to 28

- 3 Not sure/ Don’t know Go to 28

27. Can you contact your Care Co-ordinator if you have a problem?

100 1 Yes, always

50 2 Yes, sometimes

0 3 No

YOUR CARE PLAN

A care plan shows your mental health needs and who will provide services for you. It might be a document given to you by one of the mental health team, or it might be a letter, explaining how your care has been planned. 28. Have you been given (or offered) a written or

printed copy of your care plan?

100 1 Yes

0 2 No

- 3 Don’t know/ Not sure


29. Do you understand what is in your care plan?

100 1 Yes, definitely Go to 30

50 2 Yes, to some extent Go to 30

0 3 No, I don’t understand it Go to 30

- 4 Not sure Go to 30

- 5 I do not have a care plan Go to 31

30. Were you involved in deciding what was in your care plan?



0 3 No

- 4 I did not want to be involved

YOUR CARE REVIEW

A care review is a meeting with you and the people involved in your care in which you discuss how your care plan is working. 31. In the last 12 months have you had a care

review?

100 1 Yes, I have had more than one Go to 32

100 2 Yes, I have had one Go to 32

0 3 No, I have not had a care review in the last 12 months Go to 36

- 4 Don’t know / Can’t remember Go to 36 32. Were you told that you could bring a friend or

relative to your care review meetings?

100 1 Yes

0 2 No

- 3 Don’t know / Can’t remember

- 4 I did not want to invite a friend or relative

33. Before the review meeting, were you given a chance to talk to your care co-ordinator about what would happen?

100 1 Yes

0 2 No

- 3 Don’t know / Can’t remember

The LAST time you had a care review meeting…

34. Were you given a chance to express your views at the meeting?



0 3 No

35. Did you find the care review helpful?



0 3 No

SUPPORT IN THE COMMUNITY

Day centres or day hospitals

Some mental health service users go to a day centre where staff are available to help with problems, and activities are arranged. 36. In the last 2 months, how often have you visited

a day centre?

1 Most days Go to 37

2 Once or twice a week Go to 37

3 Once or twice a month Go to 37

4 I have not visited a day centre in the last 2 months Go to 38

37. Were the activities provided by the centre helpful?



0 3 No

38. In the last 12 months have you received any information about local support groups for mental health service users (e.g. MIND, Alzheimer’s Society, Rethink?

100 1 Yes

0 2 No, but I would have liked information

- 3 I did not need any information


Other support in the community

39. Are you currently in paid work? (Tick ONE only)

1 Yes Go to 40

2 No Go to 40

3 No, I am retired Go to 41

4 No, but I work on a casual or voluntary basis Go to 40

5 No, but I am a full-time student

Go to 40

40. In the last 12 months have you received help with finding work?

100 1 Yes

0 2 No, but I would have liked help

- 3 I did not need any help

- 4 I am unable to work because of my mental health problems

41. In the last 12 months have you received help

with getting benefits (e.g. Housing Benefit, Attendance allowance)?

100 1 Yes

0 2 No, but I would have liked help

- 3 I did not need any help

CRISIS CARE

42. Do you have the number of someone from your local NHS Mental Health Service that you can phone out of office hours?

100 1 Yes Go to 43

0 2 No Go to 46

- 3 Not sure/ Don’t know Go to 46

43. In the last 12 months, have you called this

number?

1 Yes Go to 44

2 No Go to 46

44. The last time you called the number, how long

did it take you to get through to someone?

100 1 I got through immediately

67 2 I got through in one hour or less

33 3 A few hours

0 4 A day or more

0 5 I could not get through to anyone

45. The last time you called the number, did you

get the help you wanted?



0 3 No

STANDARDS

46. Have you been admitted to a hospital as a mental health patient in the last 12 months?

1 No

2 Yes, once

3 Yes, 2 or 3 times

4 Yes, more than 3 times

Mental Health Act 47. In the last 12 months, have you been detained

(sectioned) under the Mental Health Act?

1 Yes Go to 48

2 No Go to 49

48. When you were sectioned, were your rights explained to you?

100 1 Yes, completely


0 3 No

- 4 Not sure/ Don’t know


YOUR FAMILY OR CARER

49. Has a member of your family or someone else close to you been given enough information from health and social services about your mental health problems?



0 3 No, but they would have liked some information

- 4 No, but they got information from somewhere else

- 5 No information was needed

50. Has a member of your family or someone else close to you had enough support from health and social services?



0 3 No, they have not had any support from health and social services

- 4 No support was needed

OVERALL

51. Overall, how would you rate the care you have received from Mental Health Services in the last 12 months?

100 1 Excellent

80 2 Very good

60 3 Good

40 4 Fair

20 5 Poor

0 6 Very poor

52. Do you have enough say in decisions about your care and treatment?



0 3 No

53. Has your diagnosis been discussed with you?



0 3 No

54. Who was the main person or people that filled in this questionnaire?

1 The service user/client (named on the front of the envelope)

2 A friend or relative of the service user/client

3 Both service user/client and friend/relative together

4 The service user/client with the help of a health professional

ABOUT YOU

55. Are you male or female?

1 Male

2 Female

56. What was your year of birth?

(Please write in) e.g. 1 9 3 4

1 9

57. In general, how is your mental health right now?

1 Excellent

2 Very good

3 Good

4 Fair

5 Poor

6 Very poor


58. To which of these ethnic groups would you say you belong? (Tick ONE only)

a. WHITE

1 British

2 Irish 3 Any other White background

(Please write in box)

b. MIXED

4 White and Black Caribbean

5 White and Black African

6 White and Asian

7 Any other Mixed background (Please write in box)

c. ASIAN OR ASIAN BRITISH

8 Indian

9 Pakistani

10 Bangladeshi

11 Any other Asian background (Please write in box)

d. BLACK OR BLACK BRITISH

12 Caribbean

13 African

14 Any other Black background (Please write in box)

e. CHINESE OR OTHER ETHNIC GROUP

15 Chinese

16 Any other ethnic group (Please write in box)

OTHER COMMENTS

If there is anything else you would like to tell us about your experiences of mental health care in the last 12 months, please do so here.

THANK YOU VERY MUCH FOR YOUR HELP Please check that you answered all the

questions that apply to you. Please post this questionnaire back in the

FREEPOST envelope provided. No stamp is needed.

Is there anything particularly good about your care? Is there anything that could be improved? Any other comments?


Copyright of 2006 Community Mental Health Service Users Questionnaire The questionnaire may be reproduced free of charge in any format or medium provided that they are not for commercial resale. This consent is subject to the material being reproduced accurately and provided that it is not used in a derogatory manner or misleading context. The material should be acknowledged as © 2006 Commission for Healthcare Audit and Inspection and the title of the document specified. Applications for reproduction of the questionnaire should be made in writing to: The Surveys Team Commission for Healthcare Audit and Inspection 103-105 Bunhill Row London EC1Y 8TG e-mail: [email protected]

Trust name Trust number Trust codeIsle of Wight Healthcare NHS Trust 34 RR2Barnsley Primary Care Trust 409 5JECoventry Teaching Primary Care Trust 463 5MDCraven, Harrogate & Rural District Primary Care Trust 464 5KJDudley Beacon & Castle Primary Care Trust 476 5HVHerefordshire Primary Care Trust 523 5CNHillingdon Primary Care Trust 527 5ATMilton Keynes Primary Care Trust 560 5CQMorecambe Bay Primary Care Trust 561 5DDNorth Dorset Primary Care Trust 572 5CDNorth Warwickshire Primary Care Trust 590 5MPPlymouth Primary Care Trust 599 5F1Portsmouth City Teaching Primary Care Trust 601 5FESelby and York PCT 616 5E2Shropshire County Primary Care Trust 620 5M2South Warwickshire Primary Care Trust 642 5MQWalsall Teaching Primary Care Trust 678 5M3West Norfolk Primary Care Trust 691 5CYWolverhampton City Primary Care Trust 700 5MV5 Boroughs Partnership NHS Trust 801 RTVAvon & Wiltshire Mental Health Partnerships NHS Trust 802 RVNBarnet, Enfield and Haringey Mental Health NHS Trust 804 RRPBedfordshire & Luton Mental Health & Social Care Partnership NHS Trust 806 RV7Berkshire Healthcare NHS Trust 807 RWXSandwell Mental Health and Social Care Trust 808 TAJBolton, Salford & Trafford Mental Health NHS Trust 809 RXVBradford District Care Trust 810 TADBuckinghamshire Mental Health NHS Trust 811 RWTCambridgeshire & Peterborough Mental Health Partnership NHS Trust 812 RT1Camden and Islington Mental Health and Social Care Trust 813 TAFCentral and North West London Mental Health NHS Trust 814 RV3Cheshire & Wirral Partnership NHS Trust 815 RXACornwall Partnership NHS Trust 816 RJ8County Durham & Darlington Priority Services NHS Trust 817 RTCDerbyshire Mental Health Services NHS Trust 820 RXMDevon Partnership NHS Trust 821 RWVDoncaster and South Humber Healthcare NHS Trust 822 RXEDorset Healthcare NHS Trust 823 RDYEast Kent NHS and Social Care Partnership Trust 825 RTMEast London & The City Mental Health NHS Trust 826 RWKEast Sussex County Healthcare NHS Trust 827 RXDGloucestershire Partnership NHS Trust 828 RTQHertfordshire Partnership NHS Trust 831 RWRHumber Mental Health Teaching NHS Trust 832 RV9Lancashire Care NHS Trust 834 RW5Leeds Mental Health Teaching NHS Trust 835 RGDLeicestershire Partnership NHS Trust 836 RT5Lincolnshire Partnership NHS Trust 837 RP7Suffolk Mental Health Partnership NHS Trust 838 RT6Manchester Mental Health & Social Care Trust 839 TAEMersey Care NHS Trust 840 RW4Newcastle, North Tyneside and Northumberland Mental Health NHS Trust 843 RNPNorfolk & Waveney Mental Health Partnership NHS Trust 844 RMYNorth Cumbria Mental Health & Learning Disabilities NHS Trust 845 RNNNorth East London Mental Health NHS Trust 847 RAT

North Essex Mental Health Partnership NHS Trust 848 RRDNorth Staffordshire Combined Healthcare NHS Trust 849 RLYNorthamptonshire Healthcare NHS Trust 852 RP1Nottinghamshire Healthcare NHS Trust 854 RHAOxfordshire Mental Healthcare NHS Trust 855 RNUOxleas NHS Trust 856 RPGPennine Care NHS Trust 857 RT2Sheffield Care Trust 863 TAHSomerset Partnership NHS & Social Care Trust 865 RH5Birmingham and Solihull Mental Health NHS Trust 866 RXTSouth Downs Health NHS Trust 867 RDRSouth Essex Partnership NHS Trust 868 RWNSouth London & Maudsley NHS Trust 869 RV5South of Tyne & Wearside Mental Health NHS Trust 870 RW9South Staffordshire Healthcare NHS Trust 871 RRESouth West London & St George's Mental Health NHS Trust 873 RQYSouth West Yorkshire Mental Health NHS Trust 874 RXGSurrey & Borders Partnership NHS Trust 875 RXXTees & North East Yorkshire NHS Trust 878 RVXHampshire Partnerships NHS Trust 881 RW1West Kent NHS and Social Care Trust 882 RXJWest London Mental Health NHS Trust 883 RKLWest Sussex Health and Social Care NHS Trust 885 RW8Worcestershire Mental Health Partnership NHS Trust 887 RWQ

Non survey variable definitions: Community Mental Health Service Users Survey 2006 1. trust: Trust code (please see 06 mental health_trust list.xls for the name of

trusts) 2. trustnum: Trust number 3. cpa: CPA (Care Programme Approach) taken from trusts’ administrative

records 1 Standard=1 2 Enhanced=2

4. outcome: Outcome of sending questionnaire 1 Returned useable questionnaire=1 2 Returned undelivered=2 3 Service user dies=3 4 Service user died , opted out or returned blank questionnaire=4 5 Service user was not eligible to fill in questionnaire=5 6 Questionnaire not returned - reason not known=6

5. age_group age grouped (16-35, 36-50, 51-65, 66+)

Table 1. Name and number of trusts Trustnum Dhcode Trust name 801 RTV 5 Boroughs Partnership NHS Trust 802 RVN Avon And Wiltshire Mental Health Partnership NHS Trust 804 RRP Barnet, Enfield And Haringey Mental Health NHS Trust 409 5JE Barnsley PCT 806 RV7 Bedfordshire And Luton Community NHS Trust 807 RWX Berkshire Healthcare NHS Trust 866 RXT Birmingham And Solihull Mental Health NHS Trust 809 RXV Bolton, Salford And Trafford Mental Health NHS Trust 810 TAD Bradford District Care Trust 811 RWT Buckinghamshire Mental Health NHS Trust

812 RT1 Cambridgeshire And Peterborough Mental Health Partnership NHS

Trust 813 TAF Camden And Islington Mental Health And Social Care Trust 814 RV3 Central And North West London Mental Health NHS Trust 815 RXA Cheshire And Wirral Partnership NHS Trust 816 RJ8 Cornwall Partnership NHS Trust 817 RTC County Durham And Darlington Priority Services NHS Trust 463 5MD Coventry Teaching PCT 464 5KJ Craven, Harrogate And Rural District PCT 820 RXM Derbyshire Mental Health Services NHS Trust 821 RWV Devon Partnership NHS Trust 822 RXE Doncaster And South Humber Healthcare NHS Trust 823 RDY Dorset Health Care NHS Trust 476 5HV Dudley Beacon And Castle PCT 825 RTM East Kent NHS And Social Care Partnership Trust 826 RWK East London And The City Mental Health NHS Trust 827 RXD East Sussex County Healthcare NHS Trust 828 RTQ Gloucestershire Partnership NHS Trust 881 RW1 Hampshire Partnership NHS Trust 523 5CN Herefordshire PCT 831 RWR Hertfordshire Partnership NHS Trust 527 5AT Hillingdon PCT 832 RV9 Humber Mental Health Teaching NHS Trust 34 RR2 Isle Of Wight Healthcare NHS Trust 834 RW5 Lancashire Care NHS Trust 835 RGD Leeds Mental Health Teaching NHS Trust 836 RT5 Leicestershire Partnership NHS Trust 837 RP7 Lincolnshire Partnership NHS Trust 839 TAE Manchester Mental Health And Social Care Trust 840 RW4 Mersey Care NHS Trust 560 5CQ Milton Keynes PCT 561 5DD Morecambe Bay PCT

843 RNP Newcastle, North Tyneside And Northumberland Mental Health NHS

Trust 844 RMY Norfolk And Waveney Mental Health Partnership NHS Trust

845 RNN North Cumbria Mental Health And Learning Disabilities NHS Trust 572 5CD North Dorset PCT 847 RAT North East London Mental Health NHS Trust 848 RRD North Essex Mental Health Partnership NHS Trust 849 RLY North Staffordshire Combined Healthcare NHS Trust 590 5MP North Warwickshire PCT 851 RW7 North West Surrey Mental Health NHS Partnership Trust 852 RP1 Northamptonshire Healthcare NHS Trust 854 RHA Nottinghamshire Healthcare NHS Trust 855 RNU Oxfordshire Mental Healthcare NHS Trust 856 RPG Oxleas NHS Trust 857 RT2 Pennine Care NHS Trust 599 5F1 Plymouth Teaching PCT 601 5FE Portsmouth City Teaching PCT 808 TAJ Sandwell Mental Health NHS And Social Care Trust 616 5E2 Selby And York PCT 863 TAH Sheffield Care Trust 620 5M2 Shropshire County PCT 865 RH5 Somerset Partnership NHS And Social Care Trust 867 RDR South Downs Health NHS Trust 868 RWN South Essex Partnership NHS Trust 869 RV5 South London And Maudsley NHS Trust 870 RW9 South Of Tyne And Wearside Mental Health NHS Trust 871 RRE South Staffordshire Healthcare NHS Trust 642 5MQ South Warwickshire PCT 873 RQY South West London And St George'S Mental Health NHS Trust 874 RXG South West Yorkshire Mental Health NHS Trust 838 RT6 Suffolk Mental Health Partnership NHS Trust 875 RTJ Surrey Hampshire Borders NHS Trust 876 RTN Surrey Oaklands NHS Trust 878 RVX Tees And North East Yorkshire NHS Trust 678 5M3 Walsall Teaching PCT 882 RXJ West Kent NHS And Social Care Trust 883 RKL West London Mental Health NHS Trust 691 5CY West Norfolk PCT 885 RW8 West Sussex Health And Social Care NHS Trust 700 5MV Wolverhampton City PCT 887 RWQ Worcestershire Mental Health Partnership NHS Trust

NHS National Patient Survey Programme: data weighting issues 1. Introduction The following key outputs are produced on most of the surveys carried out on the NHS National Patient Survey Programme each year:

• A key findings report that summarises the key findings at national level. • Trust level tables presenting the percentage of responses for all questions on

the survey plus national response totals for England. • Benchmark reports that compare the results of each NHS trust with the results

for other trusts. • Performance indicators for use on the annual NHS performance rating.

Weighted data have been used to produce the key findings report and the national totals displayed in the trust level tables since 2003/4. The benchmark reports and performance indicators have always been derived from weighted data. This document describes the approach taken to weighting the data presented in the key findings report and the national totals displayed in the trust level tables on the surveys listed below.

• Acute trust inpatient survey, • Acute trust outpatient surveys, • Acute trust emergency department surveys, • Acute trust young patients survey, • Primary Care Trust (PCT) patient surveys, • Ambulance trust survey, • Mental health trust service user surveys.

The weighting method used to derive performance indicators is described in a separate document specific to each survey. Those documents description the derivation of performance indicators have been included in the survey documentation deposited with the UK Data Archive. 2. Samples In each of these surveys, the vast majority of trusts sampled 850 patients1. Different sampling methods were chosen for different surveys because of the particular constraints of the sampling frame to be used in each case: sampling methods used are summarised in Table 1.

1 In a few exceptional cases trusts were unable to sample 850 recent patients because of their low throughput of patients. Where this occurred, trusts were requested to contact the NHS Surveys Advice Centre and smaller sample sizes were agreed.

Table 1 Summary of sampling methods Survey Sampling method Inpatients 850 consecutively discharged patients aged 16+ Outpatients Systematic sample* of outpatient attendances during a reference

month by those aged 16+ Emergency Department

Systematic sample* of emergency department attendances during a reference month by those aged 16+

Young patients 850 consecutively discharged patients: overnight and day cases of those aged 0-17

PCT Systematic sample* of GP registered patients aged 16+ Ambulance trusts Multi-stage sample involving systematic and simple random

sampling of patients aged 16+ attended during a reference week. Mental health trusts

Simple random sample of service users aged 16-64 on CPA who were seen during a three-month reference period

Further details of survey populations and sampling methods can be found in the guidance notes for individual NHS patient surveys at www.nhssurveys.org. It is worth noting that the sampling method used determines the population about which generalisations can be made. Different approaches were taken in the different surveys, meaning that results generalise to correspondingly different types of population. For the surveys of inpatients and young inpatients, the survey populations comprised flows of patients attending over particular time periods (ie the population is one of people attending), whereas for the outpatients, mental health services users, and ambulance trusts and Emergency Department surveys the survey populations comprised attendances over particular time periods. The PCT survey population comprised the stock of all GP registered patients. Below we point out some of the implications of these differences. Patients v. attendances: the difference between attendances and patients as used here may be understood by comparing two hypothetical equal sized groups of patients: group 1 patients attended once during the reference period and group 2 patients attended twice. In such a situation, a sample based on patients will represent the two groups equally, whereas a sample based on attendances will deliver twice as many from group 2 as from group 12. In other words, frequently attending patients will have a greater impact on results where samples are based on attendances than where they are based on unique patients. Stock v. flow: for a stock sample attendance frequency will have no bearing on the results. For a flow sample the make-up of the survey population will depend upon the length of the reference period used, such that relatively infrequent attendees will make up larger proportions of the sample (and hence survey population) with longer reference periods. In other words, if a survey uses a flow sample with a short * This involves sorting the sample frame based on some critical dimension(s) – eg age – and selecting units at fixed intervals from each other starting from a random point. For more detailed information, see the survey guidance documents for individual surveys. 2 This is a slight simplification as it assumes a with-replacement sampling method. This does not, however, affect the essential point.

reference period, its results will be less influenced by the experiences of infrequent attendees than they would have been had a longer reference period been used3. 3. Weighting the results Weighting to trust and patient populations In the key findings report and the national totals displayed in the trust level tables of surveys on the 2003/4 and 2004/5 NHS National Patient Survey Programmes, patient data were weighted to ensure that results related to the national population of trusts. The aim of this was to give all trusts exactly the same degree of influence when calculating means, proportions and other survey estimates. National estimates produced after weighting in this way can be usefully regarded as being estimates for the average trust: this was felt to be the most appropriate way to present results at a national level. However, it is worth noting that an alternative approach could have been taken, namely to weight to the national population of patients. This would be the appropriate approach to take if the primary interest had been to analyse characteristics of patients rather than characteristics of trusts. Weighting to the population of trusts ensures that each trust has the same influence as every other trust over the value of national estimates. If unweighted data were used to produce national estimates, then trusts with higher response rates to the survey would have a greater degree of influence than those who received fewer responses. Had we weighted to the national population of patients, a trust’s influence on the value of a national estimate would have been in proportion to the size of its eligible patient population4. 4. Illustrative example To illustrate the difference between the two approaches, we have devised a simple fictitious example concerning the prevalence of smoking in three universities, A, B and C, situated in a single region. This is shown in table 2. Table 2 Students and smoking

University No. students

Proportion smoking

A 10000 0.2B 8000 0.3C 1000 0.6Regional total 19,000

3 It is worth noting that, conceptually, a stock sample can be regarded as a flow sample with an infinite reference period, so long as all registered patients have a non-zero probability of attending. 4 For example, for the ambulance survey this would be the number of attendances of eligible patients aged 16+ during the reference week.

If we were interested in knowing the smoking prevalence of the average university, we would take the simple mean of the three proportions: 1... prevalence in average university = (0.2 + 0.3 +0.6)/3 = 0.3667. If, on the other hand, we were interested in knowing what proportion of students smoked in the region we would have to multiply each university’s proportion of smokers by its student population to give an estimate of total smokers in the university, sum these totals across universities and divide by regional student total: 2... regional prevalence = ((0.2*10000) + (0.3*8000) + (0.6*1000))/19000

= 0.2632. 5. Weighting for national level patient survey estimates As stated above, for estimates from the NHS National Patient Survey Programme, we were interested in taking the equivalent to approach 1 rather than 2. This could have been done in one of two ways:

a. analyse a dataset of trusts and apply no weight – this would entail calculating estimates for each trust and then taking means of these estimates.

b. analyse a dataset of patients after weighting each case – weights must be calculated to ensure that each trust has the same (weighted) number of responses for each item.

These two approaches produce identical estimates, but the latter method is the one used on the 2004/5 national patient surveys (the former approach was used on the 2003/04 surveys). In order to use weights to eliminate the influence of variable response rates, it is necessary to base them on the inverse of the number of responses for each trust, such that the weight for each trust is equal to k/niq where:

k is a constant niq is the number of responses to question q within trust i).

Although k may take any value, in practice it is set to the mean number of respondents answering the relevant question in all trusts because this equalises weighted and unweighted sample sizes for the national level results. Thus, the formula used to calculate weights can be expressed as:

Example of weighting to the trust population By way of example, in table 3 we have three trusts, X, Y and Z in a particular area: in each trust a different number of patients responded and in each a different estimate of proportion of patients who didn’t like the food they were given was obtained.

iq

qiq n

nw =

Note first, that if these data were held in a trust level dataset (ie with one record per trust) we would have calculated the simple unweighted trust-based mean as:

trust mean = (0.2 +0.23 +0.3) / 3 = 0.2433 Table 3 Weighting to trust population Trust 1

No. responders to food question in trust (niq)

2 Proportion of respondents disliking the food

3 Weight

1 * 2 * 3

1 * 3

X 600 0.2 0.7778 93.33333 466.6667Y 500 0.23 0.9333 107.3333 466.6667Z 300 0.3 1.5556 140 466.6667All 1400Mean 466.6667 However, in practice we often apply a weight in a patient level dataset instead. In the table 3 above, we have calculated the weight as: trust weight = (mean value of niq)/ niq. For example the weight for trust X is calculated as 466.6667/600 = 0.7778. By applying these weights (eg by using the SPSS “weight by” command) when running tables showing proportion of patients disliking the food, we obtain the simple trust based means. The way this works when calculating the proportion can be seen below: numerator for proportion = (600 * 0.2*0.7778 ) + (500 * 0.23 * 0.9333)

+ (300 * 0.3 * 1.5556) = 340.6667 denominator for proportion = (600 *0.7778 ) + (500 * 0.9333)

+ (300 * 1.5556) = 1400 Estimate = 340.6667 / 1400 = 0.2433 As can be seen, this is same as the simple mean calculated using a trust-level dataset shown above. If we did not weight, our estimate would be 325 / 1400 = 0.2321. In other words, the overall estimate would be dragged towards the estimates for those trusts with larger numbers of respondents.

Dealing with missing data and filtered questions The weighting method outlined above involves the calculation of weights for each combination of trust and question. An alternative might have been to simply calculate a single weight per trust where trust weight = (mean value of nicases)/ nicases (where nicases is the of total number of responding cases in trust i). This would be a simpler approach to implement, as it would involve substantially fewer calculations and different weights would not have to be applied for each question. In spite of this, it was considered inappropriate to use this simpler method because the number of responses varies betweens questions. Numbers of responses for different questions vary because not every respondent will answer every question. The largest source of variance is filtering – the surveys frequently include ‘filter’ questions that direct patients to answer only the parts of the questionnaire which are relevant to them. For example, a patient may be prompted to skip questions on medicines if they have not used any in the past year. Patients may also fail to answer a particular question either in error, because they refused, or because they were unsure how to answer. Similarly, responses may be missing because a patient has given multiple responses for a question. For these reasons we often find that, in practice, the number of respondents answering a particular question in trust i (niq) is less than nicases. If the proportion of respondents answering a particular question varies across trusts, then applying the trust weight as defined in the last paragraph will not give each trust exactly the same level of influence on the survey estimate. Generally, this variation should be trivial for well constructed and well laid out unfiltered questions, because the great majority of respondents will answer them in all trusts. However, the variation may in some cases become too great to ignore, particularly where questions are filtered. This is a particular issue where the numbers of people within a trust responding in certain ways to a ‘filter’ question are likely to be related to the type of trust – for instance, some specialist acute hospitals might have a very high proportion of patients responding to questions about elective admissions, but few or none responding to questions about emergency admission. Clearly, in such cases, using a single set of weights for all questions would be insufficient. For other applications users may be content to calculate a weight based upon nicases. If there is no substantial variation in the proportion of respondents answering questions of interest across trusts, this approach will deliver very similar results to those obtained using niq. Likewise, if the number of people being filtered past or skipping questions is of interest, it is possible to include these outcomes as ‘dummy’ responses for each question and therefore analyse data from different questions whilst retaining a constant base and thus ensuring all trusts have an equal degree of input. What weight should be used? Weighting to the trust population provides the most appropriate national estimates for trust comparisons. It is however, not the most appropriate approach for many other purposes. If the main area of interest relates to patients rather than trusts, it will be necessary to weight data to the national population of patients. This will require the calculation of new weights. Examples of what we mean by areas of interest are shown below: Patients Trusts

• What proportion of patients nationally felt that the toilets and bathrooms were not very or not at all clean?

• What proportion of patients in the average trust felt that the toilets and bathrooms were not very or not at all clean?

• Were males or females more likely to say that toilets and bathrooms were not very or not at all clean?

• Were small acute trusts more or less likely than medium / large acute trusts to have patients who said that toilets and bathrooms were not very or not at all clean?

Calculating patient population weights Although patient population weights have not been calculated, users may well need to use these for some of their analyses. These should be calculated as:

patient population weight = (k * Ni)/ nicases,

where: nicases is the number of respondents in trust i5, Ni is the number eligible patients in the survey population in trust i, k is a constant, which is usually set so as to equalise the overall weighted and unweighted sample sizes.

Probably the main difficulty in calculating this weight will be obtaining a reliable figure for Ni. Ni is the population to which each trust’s results are to be generalised. Ideally this should be the size of the population from which the sample was actually selected. For example, for ambulance trusts, Ni would ideally be the total number of attendances during the exact reference week (ie the number of cases from which the sample of 850 was actually drawn). However, we acknowledge that this information is unlikely to be available, and it will therefore be necessary to substitute an estimate instead. In doing this it should be borne in mind that the definition of the population from which the estimate of Ni will be derived should be as close as possible to the definition of the population from which the sample was actually selected. For example, the trust population figures used to calculate weight Ni for the PCT surveys should relate to the stock of patients and not the flow of patients or attendances; a flow sample should, ideally, be weighted to a population using the same reference period (eg the Emergency Department data should be weighted to monthly throughput). Furthermore the population figures used for weighting should, of course, relate to the same year (at least!) as that in which the survey was conducted. Of course, if there is a dearth of available population information, non-ideal population data have to be used. If this is the case, it is worthwhile spelling out the additional assumptions that will, by implication, have to be being made. For example, if inpatient data are weighted to inpatient attendance figures instead of patient flows, 5 In principle it would be possible to use niq in this formula for unfiltered questions (it could not be done for filtered questions because this would require us to substitute number in the population eligible for the filter question – an unknown value - for Ni). To our knowledge, in practice this approach is never taken.

an implicit assumption is being made that the proportion of patients making n attendances over the reference period is constant across trusts6. Use of unweighted data If a user decides simply to analyse unweighted data, the implications of so doing need to be understood. Given the sampling methods used, an unweighted sample would deliver approximately equal numbers of responses if response rate did not vary widely between trusts. In effect this would mean that the sample would be approximately equivalent to one weighted by: trust weight = (mean value of nicases)/ nicases As such, it could be regarded as crudely representing the population of trusts (crudely, because in practice response rates did vary, and as a result trusts with good response rates would have greater influence on the results that would trusts with poor response rates). It would, however, be wholly inappropriate for analyses of patients. This is because, unweighted, the data will substantially under-represent patients in trusts with large numbers of patients, and substantially over-represent patients in trusts with small numbers of patients. To the extent that that large and small trusts differ systematically from one another on survey variables, the use of unweighted data will introduce systematic bias into the results. Patten Smith 4 November 2005

6 An added (but, in practice, trivial) complication is that for the inpatient and young patient surveys there is no “perfect” definition for a population data reference period. This is because the sampling method itself used a variable reference period: trusts with large patient throughputs used shorter reference periods than trusts with smaller throughputs.

The Healthcare Commission The Healthcare Commission was created under the Health and Social Care (Community Health and Standards) Act 2003. The Healthcare Commission exists to promote improvements in the quality of healthcare and public health in England. We are committed to making a real difference to the delivery of healthcare and to promoting continuous improvement for the benefit of patients and the public. The Healthcare Commission’s full name is the Commission for Healthcare Audit and Inspection. We have a statutory duty to assess the performance of healthcare organisations in the NHS and award annual ratings of performance, to coordinate inspections and reviews of healthcare organisations carried out by others, and register organisations providing healthcare in the independent sector. We have created an entirely new approach to assessing and reporting on the performance of healthcare organisations - our annual health check - which will examine a broad range of factors enabling us to focus on what really matters to patients and the public. The Patient Survey Programme Understanding what patients think about the care and treatment they receive is crucial to improving the quality of care being delivered by the NHS and to ensuring that local health services meet the needs of patients and the public. One way of doing this is by asking patients who have recently used their local health services to tell us about their experiences. In 2004, the Healthcare Commission carried out seven national surveys of patients covering primary care, community mental health, hospital inpatient care for adults, accident and emergency care for adults, services for children and young people, ambulance services and outpatient services. In 2005, we revisited three areas already covered by the survey programme – community mental health, primary care and adult inpatient services - again asking patients to give us their views about the care and treatment they have received. The programme for 2006 consists of a survey on community mental health and a survey of adult inpatient services. In 2006, the community mental health survey involved 79 NHS trusts (including combined mental health and social care trusts, and primary care trusts) that are responsible for providing secondary mental health services. We received responses from more than 19,000 service users. Four trusts offering mental health services were not included in the survey. Two PCT’s began offering mental health services in June 2005. As the survey asks patients about the services that they received between March 2005 and February 2006, it was not appropriate to include them in the survey. Two mental health trusts are not

included since they have insufficient numbers of patients receiving secondary mental health services on the Care Programme Approach (CPA). In 2006, the community mental health survey was extended to include service users over the age of 65 years. However, to enable comparability with previous surveys, the benchmark reports were calculated using only respondents aged 65 years and under. Benchmark results for the whole sample are available on request to the Surveys Team at the Healthcare Commission. Please email: [email protected] Introduction to the report Three types of results on the 2006 Community Mental Health Survey are available from the Healthcare Commission website:

• Benchmark reports (including this one) present scores for each question for each trust compared with national benchmark results. This should be used to understand the trust’s performance.

• Spreadsheets containing the percentage of respondents giving a particular response to a survey question for each trust or combined PCT and mental health trust. These spreadsheets describe the percentage of service users who have had a particular experience, for example, the percentage of people saying that the psychiatrist definitely listened carefully to them. The spreadsheets should be used to understand the detailed results for the survey on individual trusts.

• A briefing note that provides a commentary of the key national findings and changes from previous surveys.

The documents described above are produced by the Mental Health Survey Co-ordination Centre at the National Centre for Social Research on behalf of the Healthcare Commission. Interpreting the report The benchmark reports are calculated by converting responses to particular questions into scores. For each question in the survey, the individual responses were scored on a scale of 0 to 100. A score of 100 represents the best possible response1. Therefore, the higher the score for each question, the better the trust is performing. A ‘scored’ questionnaire showing the scores assigned to each question may be downloaded from the Healthcare Commission website (see further information section). Please note that the scores are not percentages, so a score of 80 does not mean that 80% of service users in the trust have had a particular experience but that the trust has scored 80 out of 100. (The results using percentages to describe patients’ experiences are

1 Different trusts will have a different profile of service users, for example, one trust may have more male service users than another trust. This can potentially affect the results since it is known that some people tend to answer questions in a different way than others. Therefore, the results have been weighted by the age and sex of respondents to ensure that no trust will appear better or worse than another because of its sample profile. The trust level results are standardised, so that their age-sex profile reflects the national age-sex distribution (based on all of the survey respondents). This enables results from trusts with different patient profiles to be compared.

presented in the national spreadsheets which are also available from the Healthcare Commission website). The following graphs display this trusts scores, compared with national benchmarks. Each bar represents the range of results across all trusts that took part in the survey for each question. In the graphs, the bar is divided into three coloured segments: • the left hand end (coloured red) shows the scores for the 20% of trusts with the

lowest scores • the right hand end (coloured green) shows the scores for the 20% of trusts with the

highest scores • the middle section (coloured orange) represents the range of scores for the

remaining 60% of trusts The score for this trust is shown on each bar by a white diamond. So, for example, if the diamond is in the green section of the bar, the trust is in the best 20% of trusts in England. The line either side of the diamond shows the amount of uncertainty surrounding the trust value, as a result of random fluctuation2. The data used for the charts and background information about the service users that were surveyed may be found at the end of the report. Notes on specific questions Q23 and Q24 Please note that the information collected by question 23 “In the last 12 months have you had any counselling sessions (e.g. talking therapy) from NHS Mental Health Services?” and question 24 “In the last 12 months, did you want talking therapy?” is presented together to show whether the provision of talking therapy met the service user’s requirements. The combined question is numbered in this report as question 23 and has been reworded to read “In the last 12 months, did the provision of talking therapies meet your requirements?” For further details, please see the ‘scored’ questionnaire which shows the scores assigned to each question. This may be downloaded from the Healthcare Commission website (see further information section). Q48 Please note that though reported in 2004 and 2005, question 48 “When you were sectioned were your rights explained to you” is not shown in this report. This is because this year, no trust had more than 30 respondents to this question and results were therefore not reliable enough to report (see footnote 2).

2 Since the score is based on a sample of service users in a trust rather than everyone, the score may not be exactly the same as if everyone had been surveyed and had responded. Therefore a confidence interval is calculated as a measure of how accurate the score is. We can be 95% sure that if everyone in the trust had been interviewed, the ‘true’ score would fall within this interval. The diamond is not shown at questions answered by fewer than 30 people because the uncertainty around the result would be too great. Note also that when identifying trusts with the highest and lowest scores and thresholds, trusts with fewer than 30 respondents have not been included.

Further information Full details of the survey methodology can be found at: www.healthcarecommission.org.uk/mentalhealthsurvey2006 More information on the NHS Patient Survey Programme is available on patient survey website: http://www.healthcarecommission.org.uk/NationalPatientSurveyProgramme The 2006 Community Mental Health Survey results, questionnaire and scoring can be found at: http://www.healthcarecommission.org.uk/PatientSurveysMentalHealth2006 The 2005 Community Mental Health Survey results can be found at: http://www.healthcarecommission.org.uk/PatientSurveyMentalHealth2005 The 2004 Community Mental Health Survey results can be found at: http://www.healthcarecommission.org.uk/PatientSurveyMentalHealth2004 More information on the 2005/2006 NHS performance ratings is available on the Healthcare Commission website: www.healthcarecommission.org.uk/ratings

National patient survey programme - UK Data Service€¦ · The national patient survey programme...

Documents

Transcript of National patient survey programme - UK Data Service€¦ · The national patient survey programme...