Museums Galleries Scotland · Web viewAdditionally, If the ‘museums should’...

Museums Galleries Scotland

Quality Improvement System – Pilot Report

The Sport and Leisure Consultancy May 2010

REPORT ON THE MUSEUMS GALLERIES SCOTLAND - QUALITY IMPROVEMENT SYSTEM PILOT

Contents1.0 Introduction...............................................................................................................................2

2.0 Methodology.............................................................................................................................3

3.0 Pilot Findings and recommendations........................................................................................4

3.1 Briefing the pilotees...............................................................................................................4

3.2 Self-evaluation guidance........................................................................................................7

3.3 QIS toolkit..............................................................................................................................8

3.4 Ongoing co-ordination and project management..................................................................9

3.5 Framework format, content and language..........................................................................11

3.6 Submissions.........................................................................................................................13

3.7 Evidence gathering..............................................................................................................15

3.8 Validation.............................................................................................................................17

3.9 Validation Review Teams.....................................................................................................20

3.10 Post validation.....................................................................................................................23

4.0 QIS and the Independent Museum Sector...............................................................................24

5.0 QIS and university museums....................................................................................................28

6.0 QIS and the National Collections.............................................................................................30

7.0 Alignment with HGIOCS?.........................................................................................................32

8.0 Alignment with Accreditation..................................................................................................34

9.0 Resource issues........................................................................................................................37

10.0 Next steps – future roll out......................................................................................................39

11.0 QIS application process............................................................................................................40

12.0 Conclusions..............................................................................................................................42

Appendix A – Step-by-step approach for QIS applicants.....................................................................43

1


1.0 Introduction 1.1 The Sport and Leisure Consultancy were commissioned by Museums Galleries Scotland

(MGS) to coordinate and manage a pilot project to test the Quality Improvement System (QIS).

1.2 The MGS QIS is a self-assessment tool for Accredited museums and galleries to help them to continuously monitor the quality of their services. QIS is designed to provide a framework for continuous improvement once museums have achieved Accreditation.

1.3 The intention is that the framework can be used by all museums in Scotland with the aim of helping them evaluate their quality of provision, aid with planning, and help implement improvements in services.

1.4 QIS has been designed to collect data that can feed into any existing regional and national frameworks while avoiding any duplication of work. It has been developed in parallel with How Good is our Culture and Sport? (HGIOCS?), the new quality framework for local authority culture and sport services.

1.5 QIS works by asking museum organisations to measure their effectiveness against one or more of the following seven ‘impact measures’:

Service planning and development

Access and inclusion

Meeting users needs

Users’ experiences and learning

Management of resources and space

Leadership, ethos and values

Management of collections.

1.6 The pilot involved three Local Authorities (Falkirk, Fife and Orkney), one Trust (Culture and Sport Glasgow), one independent museum (Auchindrain Museum), and one university Museum (the Hunterian Museum). The main aims of the pilot were to test the processes involved and the relevance and ease of use of the framework and to gather feedback which will be incorporated into final development of QIS before it is rolled out more widely.

2


2.0 Methodology 2.1 The pilot commenced on 23 September 2009 (date of first briefing) and was completed on

25 March 2010 (date of post-pilot workshop). All pilotees were asked to complete Impact Measure 1: Service Planning and Development which would act as a control section. In addition pilotees could choose up to two other Impact Measures. Between them all seven Impact Measures were completed by at least one pilotee.

2.2 To assist pilotees through this process, they attended an initial briefing session, had access to self-evaluation guidance notes which explained how to tackle self-evaluation and had access to on-line toolkit resources. They were also provided with on-going consultant support in the form of weekly email updates, social network site updates/discussion forums, telephone/email support and site visits upon request.

2.3 Feedback on the pilot was collected at various stages and from a range of sources including:

MLA/MGS Glasgow Accreditation Consultation event (16 September 2009)

QIS pilot briefing event

BOP Consulting, Evaluation framework study for MGS

Comments received from pilotees on the social network site

Comments received from pilotees in email correspondence during the pilot

Meetings of the Quality Improvement Framework (QIF) Implementation Group (responsible for developing How Good is our Culture and Sport?)

Pilotees during the pilot

MGS staff involved in the pilot

The Highland Council independent museum consultation session (5 March 2009)

Peer reviewers (pre and post on-site review visits)

Feedback From The Royal Commission on the Ancient and Historical Monuments of Scotland (who are part of the National Collections)

Pilotees’ feedback workshop session held at the end of the pilot (25 March 2010).

3


“It told us we were right in our judgements about our service and it has helped us take issues forward”

3.0 Pilot Findings and recommendations

3.1 Briefing the pilotees

What we did3.1.1 Two briefing sessions were held with 22 representatives of the pilotees. The first session

comprised 16 attendees during a half day. The second session lasted six hours and comprised 6 attendees. The briefing sessions by their very nature required the imparting of pertinent information about the QIS framework and outline process to be followed; however, there was some time for discussion and group exercises to foster understanding and cooperation among the peer group. The session objectives were:

To provide the pilotees with the confidence to get started

Build trust and empathy with the pilotees so they felt the consultants and MGS representatives were ‘on their side’

Provide pilotees with comfort that there is an effective and responsive support network in place

Deal with concerns and questions about the process

Confirm QIS Impact Measure sections choices ensuring all of the quality indicators were covered.

Session 13.1.2 The first session included 16 attendees. Although the facilitators were able to get across

most of the essential information, four hours was insufficient to complete the group exercises effectively and some of the content towards the end of the session was rushed. Feedback regarding the content was largely positive although some mentioned the size of the room which was not ideal for the numbers attending and the lack of time available for discussion. Despite this it was felt that the session objectives were broadly achieved.

Session 23.1.3 In contrast to the first session, the second was held with fewer people, for longer. In

addition, more time was scheduled for discussion and group exercises, which was a more beneficial approach. All attendees’ feedback was very positive.

4


Findings3.1.4 For many the briefing session will be their first contact with the QIS and it is natural there

will some anxiety about what lies ahead and their ability to deliver. Pilotees liked the fact the sessions were upbeat and positive, provided comfort about the process and demonstrated how QIS should be tackled step-by-step. All bar one of the pilotees said the briefing session equipped them well enough to tackle QIS.

3.1.5 Briefing sessions work best with lower numbers in venues that are spacious and conducive to an informal approach. Session times need to be carefully considered to ensure sufficient time has been allocated to cover the material. These sessions can be labour intensive and one of the challenges for QIS when it is rolled out is to identify a way of briefing large numbers in a productive but cost and time effective manner

3.1.6 Sessions work best when there are a minimum of two facilitators; in the region of six hours scheduled including lunch and coffee breaks; a maximum of 12 attendees and lunch provided on-site

3.1.7 Pilotees commented that MGS should strongly recommend that as many of the self-evaluation team as possible should attend briefing sessions. As well as helping with team building, this means that team leaders can save valuable time on their return to the workplace because they do not have to re-present and interpret this information for colleagues.

3.1.8 Feedback suggested that participants would like more detail on what constitutes relevant evidence and how much evidence they are expected to list in support of their strengths.

3.1.9 At submission stage some pilotees struggled to score themselves against the prompt question. Some viewed the six-point scale as a relative scale (i.e. “we are doing an excellent job given our limited resources” rather than an absolute judgement against a fixed standard).

Recommendations1. Be clear about who should attend the briefing sessions.

2. Independent museums should be encouraged to bring a trustee to the session.

3. As a minimum the briefing session content should cover:- The principles of self-evaluation - The key stages involved and milestones- What a completed submission looks like- How to arrive at an appropriate award level score- The validation process- Support and guidance available.

5


4. As well as imparting information the session leaders should allow sufficient time for participants to discuss areas of uncertainty and practical difficulties they may have and how these can be overcome.

5. Ensure sessions are facilitated by a minimum of two members of staff and that sufficient time is scheduled.

6. Consider use of appropriate venues in Scotland for QIS briefing sessions. These could include those used for the Accreditation Workshop programme.

7. Involve any newly recruited peer reviewers at the briefing sessions to introduce and acquaint them with QIS and potentially those organisations they may review later in the process.

6


3.2 Self-evaluation guidance

What we did3.2.1 This guidance was based on the principles of the European Foundation for Quality

Management, Assessing for Excellence model and builds on elements from other quality frameworks. The guidance produced by the consultant was also adapted and subsequently adopted for HGIOCS? which should ensure the same methodology is used for both frameworks which will cement links between them.

Findings3.2.2 Overall the feedback from pilotees was very positive. The self-evaluation guidance was

regarded as a valuable tool. They remarked that the guidance set out a simple process template they could follow and they liked the fact it was practical and concise; however, there was some confusion about whether the self-evaluation guidance or the QIS framework was the ‘lead’ document which informed what a QIS submission should comprise of

3.2.3 Initial feedback on the usefulness of the guidance from the HGIOCS? pilot is also very positive.

3.2.4 Some pilotees thought there was some flexibility about how they put together their submission. For instance two pilotees did not include an improvement plan in their submission and others did not include narrative with their evidence list (which explained what the evidence demonstrated and why it had been included). The guidance could have been clearer on these points.

Recommendations1. As per the briefing sessions, provide more support on evidence gathering and scoring.

2. Ensure there is a section in the self-evaluation guidance which clearly sets out what a QIS submission should comprise of. This will ensure greater consistency and reduce the likelihood of incomplete and/or incorrect submissions.

7


3.3 QIS toolkit

What we did3.3.1 One of the first tasks for the consultants was to decide on a portfolio of on-line support

materials that could be uploaded onto the MGS website. Initial documents that formed part of the toolkit were influenced by the Public Libraries Quality Improvement Matrix (PLQIM) on-line toolkit. Further documents were added up until the pilot launch. In total 10 documents including pro-formas, submission templates, evidence examples, staff presentations, customer questionnaires among others, were available to download.

Findings3.3.2 The volume of toolkit resources was considered by some to be overwhelming and therefore

pilotees tended not to use them. It was suggested this could be resolved by ‘thinning out’ and deleting some less important items or by grouping resources by heading.

3.3.3 Having to enter a name against each downloaded document was time consuming and off-putting for some.

3.3.4 Only one strengths/improvement template should be posted as having two caused confusion.

3.3.5 Adjust strengths and improvement template – no need for ‘evidence to support’ or ‘evidence gaps’ or ‘improvement plan priorities’ as these can be recorded elsewhere.

3.3.6 An additional support resource was pilotees working together on their submissions i.e. one pilotee who was struggling with their submission invited another to provide guidance.

Recommendations1. Ensure all essential pro-formas needed for the submission are grouped together and

given prominence.

2. Determine if all resources are actually needed.

3. Group the remaining items by Impact Measure or other group headings.

4. Make downloading the documents easier.

5. Encourage peer collaboration and support among QIS participants.

8


3.4 Ongoing co-ordination and project management

What we did3.4.1 After the briefing sessions we considered letting the pilotees tackle QIS on their own with

minimal contact until all pilots had been completed; however, it soon became apparent that further co-ordination support and guidance would be necessary if the pilots were to be completed on time. Consequently, the consultants supported consultees by:

Keeping in regular contact with the project leaders

Sending out weekly email updates to all pilot self-evaluation team members to update them on progress and issues that had arisen during the week

Regular reminders of milestones that had to be accomplished

Setting up a social network site which encouraged debate on discussion forums and blogs

Visiting three out of the six pilotees to review progress to date, check their understanding of the process, discuss issues that had arisen and to provide support, encouragement and confidence they were on the right lines.

Findings3.4.2 One of the key findings is that ‘momentum is everything’. If the QIS process stalls pilotees

said it could be difficult to kick start. Co-ordination and regular contact with pilotees was essential to ensure that pilotees maintained progress and the QIS did not get put on the ‘back-burner’ or get lost among the myriad of other day-to-day issues that had to be dealt with. While the content was not regarded as important, weekly email updates helped remind pilotees of the need to progress.

3.4.3 The other significant benefit of supporting the pilot was to foster ‘team spirit’ i.e. everyone is in this together, whereby pilotees could learn from each other’s experiences. For example, one pilotee requested another to visit and assist with their submission.

3.4.4 Pilotees really appreciated having a ‘one-stop’ contact for all their QIS queries and support issues.

3.4.5 While the social network site was regarded as innovative and the majority of team members joined, the level of interaction was disappointing and therefore, we conclude it is not worth the time and effort to maintain in future.

3.4.6 Providing on-going support to pilotees is high maintenance and can be time consuming. Around 1-2 hours per week should be allocated to this task and significantly more when approaching critical milestones such as submission deadlines or on-site visits.

9


Recommendations1. Ensure there is one person nominated to provide pro-active, on-going support for those

going through QIS. This should include ad-hoc telephone support and availability for site meetings to help with progress if required.

2. Focus on keeping in regular contact with participants to ensure agreed timescales are being adhered to. An example of this could be fortnightly e mails which simply ask participants to update on their progress or inform participants of any new QIS related developments.

10


3.5 Framework format, content and language

What we did3.5.1 The framework was developed in 2008 and further amended and improved by MGS. It was

designed to be applicable for local authority museum services, trusts, university and independent museums. The QIS framework aims to provide information and guidance on key areas where quality matters. For each of the seven Impact Measure sections there are:

Main themes

An Impact Measure description which describes what an effective service looks like

Key evidence requirements which the museum and gallery should consider

Prompt questions to help focus discussion and support the self evaluation process

A level 5 illustration which sets out what we would expect to find in an organisation that considers itself to be ‘very good’.

Findings3.5.2 All pilotees commented that there appeared to be some duplication throughout the

framework in the form of prompt questions which were worded slightly differently but seemed to ask for similar information. There is scope to combine prompt questions or maybe even delete some to streamline the framework. This highlights the need to refine QIS before it is rolled out.

3.5.3 Other feedback included:

The framework document stretches to 49 pages which can make it an unwieldy working document. Thinning out would help resolve this

Independent museums argued that some of the prompt questions were not relevant to their type of operation and were too local authority oriented (this is explored more in section 4)

It was questioned whether it is important to have a level 5 Impact Measure description, a ‘Museums should’ section and a ‘why is this important’ prompt question section. To have so many prongs was regarded as unnecessary by some

The ‘Museums should’ description, prompt questions and level 5 illustration were not always consistent which suggests further refinement is required. Additionally, If the ‘museums should’ description is retained, consider changing use of the word ‘should’ which might be seen as presumptive, to ‘Museums should work towards’ or ‘Museums should consider’, which some regarded as more user-friendly

The prompt questions are very useful but should not be called ‘prompts’ as they are in fact what the museum is measuring itself against. If the prompts are to be used in this

11


way in future, alternative headings might include: ‘questions to ask’, ’criteria’ or ‘quality measures’

Need to strengthen links to other strategic plans and documents in the introduction

Some pilotees answered the ‘how?’ part of the prompt question but did not actually form a judgement about how well they did

All questions should ask ‘how well...’ or “how effectively...” rather than ‘Do you have a....’

Need to include award level scoring narrative and a ‘how to’ guide in the introduction

The language throughout the framework could be crisper and use fewer words

The level 5 illustration is very helpful but it would be useful to have at least one other illustration, perhaps at level 2. This would provide another benchmark to measure the organisation against to assist with the scoring

There is likely to be scope to edit the framework to make it more relevant to independent museums.

Recommendations1. Use the findings to refine and revise the framework.

2. Ensure the refined framework is appropriate for local authorities, trusts, university museums and independent museum sector. Refining the master document will make it more relevant, easier to work through and less overwhelming, whilst ensuring it retains its rigor.

12


3.6 Submissions

What we did3.6.1 The original intention was not to be prescriptive about what a submission should look like;

however, we found that all organisations wanted clear guidance about what they should provide. In response we asked for a portfolio of information to be submitted in advance of the review visit to inform the validation. This included the following documents:

A brief summary (2/3 pages A4) describing the organisation’s structure, aims, future developments and key issues

A strengths and areas for improvement matrix

Pilotees evidence in support of their strengths which was indexed by Impact Measure with brief narrative about why this is important (see next section for more on evidence gathering)

An improvement plan.

3.6.2 In addition to these elements we reviewed key documents already held by MGS including Single Outcome Agreements (SOA’s), Forward Plans and other relevant corporate plans or strategies.

Findings3.6.3 Pilotees welcomed the submission templates we developed and which were available on-

line although there was some confusion about whether these had to be used or were just recommended.

3.6.4 Most pilotees concluded that developing the submission “was hard work but manageable”.

3.6.5 The quality of the submissions was inconsistent. This could be improved by issuing clearer guidance, providing examples of good practice on-line and setting out mandatory elements e.g. “A submission WILL consist of...” Examples of inconsistency included:

Some used sentences to describe their strengths and weaknesses, while others used single word answers or a short phrase

Some provided many items of evidence against each prompt question with a good explanation about why these had been included, while others cited only two or three examples with no narrative

Some decided not to submit an improvement plan, arguing that they already had a pre-existing planning cycle which they did not want to replicate

Some decided not to prioritise improvements identified

13


During the pilot we became aware of a software programme (Covalent) which could help manage all aspects of a participant’s submission on-line. This could reduce paperwork further, be a repository for evidence, lead to greater consistency of submissions and support improvement planning. Another advantage of this type of programme is it would help participants cross-reference evidence between different Impact Measures. Further work is needed to develop this system but potentially it could be very beneficial to the process.

Recommendations1. Reduce the potential for inconsistent or incomplete submissions by providing clear

and unambiguous information about what a submission should consist of.

2. Explore the scope for using Covalent or an equivalent software programme as a tool to make the QIS process less burdensome for participants and Review Teams.

14


3.7 Evidence gathering

What we did3.7.1 A sub-issue of the submission process is to determine how pilotees should submit evidence.

The over-riding purpose of identifying evidence is to support judgements that are made as part of a QIS submission. The self-evaluation guidance clearly states that judgements should not be made that can’t be evidenced.

3.7.2 Pre-pilot we considered a number of formats, including:

• Collection of headline statements only

• Collection of full copies of all evidence for submission (like current Accreditation and PLQIM evidence submissions)

• Assemble evidence for the assessor to review in the organisation (like HMIe inspection)

• Submission of a narrative sheet, which briefly describes the relevance of the evidence and a sample is then requested to view electronically before the site visit to review the service.

3.7.3 In deciding upon the most appropriate format we were guided by the following considerations:

QIS should encourage an innovative, creative and flexible approach to what constitutes

evidence. This could include photographs, digital media, blogs, observation and websites, etc. This can be in addition to more conventional evidence such as strategic plans, forward plans and performance indicators

Gathering evidence for QIS should not be bureauocratic or burdensome for participants or the Review Team

QIS should consider ‘manageability’ and sustainability issues such as minimising human resource input, reducing print and keeping storage to a minimum.

3.7.4 The process we arrived at asked pilotees to list all relevant evidence for each impact measure on a pro-forma which included brief narrative on ‘what the evidence demonstrates’. Then prior to the on-site review visit we requested a sample of the evidence (typically 20-25% of that cited) to be provided on a USB stick beforehand.

3.7.5 To guide pilotees, examples of evidence for each quality indicator were compiled by the MGS QIS working group and are available on the MGS website in the QIS pilot toolkit section.

15


Findings3.7.6 Pilotees found the examples of evidence on the website useful.

3.7.7 Pilotees were not confident about how much evidence was enough and would welcome more guidance on this at the outset.

3.7.8 Pilotees appreciated only having to provide a sample of the total evidence cited and found that being able to provide evidence electronically (via USB stick)was more favourable than copying documents. Some commented that this was a better system than most other quality frameworks which required hard copy of documents available on-site which could be time consuming and wasteful.

Recommendations1. Provide clearer guidance about how much evidence should be listed and what type

of evidence we are looking for.

2. Overall the evidence element of the process worked well and should be replicated.

3. Ensure evidence is submitted at least a fortnight before the on-site validation visit.

16


3.8 Validation

What we did3.8.1 The purpose of validation is to bring rigour and consistency to the self-evaluation process. It

involved recruiting Review Teams, a desk exercise to review and validate judgements made in relation to the Impact Measure prompt questions and an on-site Review Team visit. The approach adopted aimed to be challenging but supportive and was chosen as it:

Is light touch

Is focused primarily on judgements rather than on measurement

Supports improvement.

3.8.2 In practical terms the main steps of the process were:

1. Recruit and brief the Review Team.2. Issue guidance on what the on-site visit will consist of and ask the participant to draw

up an itinerary.3. Review Team pre-meet to review the QIS submission along with key documents.4. Identify evidence to be reviewed on-site and the lines of enquiry to follow.5. Prepare the on-site review and notify the museum of evidence that should be sent prior

to the on-site review.6. Review evidence provided and identify additional follow up questions7. For organisations completing Impact Measure 3 – ‘Meeting users needs’ we carried out

mystery visits at two sites and considered lines of enquiry that fell out of the report findings.

8. Carry out the on-site review using the sample site visit programme set out in the ‘Validation Process guidance’ which includes:- Meet with the self-evaluation team- Meet a cross section of staff- Meet with members of the Senior Management team e.g. Head of service- Meet the elected member with responsibility for culture (where appropriate)1

- Meet a trustee (where appropriate)- Observe the service and meet other front-line staff.

9. Review the visit findings.10. Write a summary report of the findings (including award level) based on the style of the

level 5 illustration.

1 Meeting with an elected member with portfolio or committee responsibility for museums is valuable where a local authority has completed Impact Measure 1 – Service Planning and Development. Where participants are not completing this Impact Measure this meeting may not be necessary.

17


Findings3.8.3 Pilotees felt all of the components of the review were appropriate. They felt we spoke to the

right people and thought that meeting with elected members was particularly useful in terms of raising the profile of the service.

3.8.4 Typically two and half to three hours were allocated to meet the self-evaluation team. This proved sufficient where pilotees had undertaken just one Impact Measure but for most, this amount of time proved insufficient.

3.8.5 The consensus was that one day is insufficient for an on-site visit unless only one Impact Measure is being validated.

3.8.6 Due to its emphasis on strategic planning, Impact Measure 1 is a significant section as it also requires the Review Team to meet the Head of Service or trustee, an elected member and a cross-section of staff. Not all Impact Measures require this degree of thoroughness therefore once this element has been completed for the first time, future Impact Measures tackled by the organisation are likely to require a much shorter time and resource input to complete.

3.8.7 In the majority of cases the day felt rushed. The schedule allowed little time for over-runs with the consequence that sessions were cut short or planned lines of enquiry were abandoned. A day and a half (or even two days for larger local authorities) should be scheduled in future for any participant tackling two or more Impact Measures which include Impact Measure 1.

3.8.8 Reviewers felt there was insufficient time to observe the service. Observation was built into the day but typically this involved a visit to just one or two sites selected by the pilotee. This meant it was difficult to accurately gauge standards of provision.

3.8.9 Some meetings involving a cross-section of staff were also attended by a member of the self-evaluation team (typically a senior manager). Reviewers felt this constrained the discussion which meant this session was less open and honest than it may have been otherwise.

3.8.10 It was suggested that QIS reviews should consider asking users about the service. The libraries quality framework which does this, was felt to be a useful model.

3.8.11 With regard to the use of mystery visits, feedback from the pilotee indicated this extra level of scrutiny was very positive and was welcomed. The benefits of providing mystery visits in future needs to be set against the cost of providing this.

3.8.12 Pilotees felt that the on-site reviews were pitched at the right level and were rigorous enough without being over-bearing or too intense.

18


Recommendations1. The different elements of the validation programme are broadly recognised as providing

the correct level of scrutiny and should be replicated.

2. The amount of time allocated for each site visit needs to be carefully considered taking into account the scale and complexity of the organisation and the Impact Measures completed. Large and complex organisations that have completed Impact Measure 1 among others are likely to require a minimum of 1.5 to 2 days for the on-site visit element. Most on-site visits for organisation going through QIS for the first time are likely to require a minimum of 1.5 days.

3. The Review Team should request which museums they visit to observe the service rather than leave it to the participant.

4. The guidance notes should make it clear that the meeting with the cross –section staff will not include any members of the self-evaluation team.

19


3.9 Validation Review Teams

What we did3.9.1 We used six QIS reviewers. Different combinations of Review Team members were piloted.

Some Review Teams had two and others three members. The Review Teams consisted of:

Consultants

Museums Galleries Scotland Managers

Peer reviewers.

3.9.2 The definition of a peer reviewer is an experienced museum and gallery professional who has managed or is managing frontline services and/or is involved in shaping the strategic direction of their organisation. During the pilot we particularly wanted to test whether peers would be able to provide meaningful challenge to their colleagues or whether they would feel constrained in this role.

3.9.3 The process for the Review Team included the following steps:1. Recruit team member and send briefing notes.2. Distribute the submission of the participant organisation to team members.3. Review Team members review the submission and key documents.4. Review Team members come together to discuss the submission and decide upon lines

of enquiry, decide upon evidence to be requested and agree roles for the on-site visit.5. Request sample of evidence to be sent to the Review Team leader.6. The Review Team leader reviews evidence to determine its relevance and identifies

additional questions line of enquiry for the on-site visit.7. Execute on-site visit.8. Team leader writes report and distributes to other team members for comment.9. Report agreed and sent to museum organisation as a draft for comment on factual

errors or misinterpretation.10. Final report agreed and signed off.

3.9.4 The final report produced by the Review Team included an introduction to QIS, a summary about the organisation being reviewed, narrative using the same sub-headings as shown in the level 5 illustration, the Review Teams award levels against each of prompt questions and headline strengths and areas for improvement. The report typically took between a day and a day and a half to write. We aimed to complete a draft within 2 weeks of the on-site visit.

20


Findings3.9.5 All pilotees felt that where possible peer reviewers should always be involved in the

composition of Review Teams. An implication of adopting this approach is how best to recruit and resource a pool of peer reviewers. Scottish Library Information Council (SLIC) recruit peer reviewers from organisations which have already been through their quality framework. If adopted, this would ensure recruitment of peer reviewers who already have a working knowledge of QIS, which means their involvement can be fast tracked.

3.9.6 There was no evidence to suggest that peer reviewers were any less rigorous than consultants or MGS staff involved in the review process.

3.9.7 Feedback from pilotees and peer reviewers highlighted a number of benefits from using peer reviewers:

They understand the pressures and challenges of running museums and therefore can bring valuable insight to the table

Have credibility in the eyes of those being reviewed

It demonstrates that the profession can drive improvement from within

They will see other museums and services in operation which will help improve their own organisations and their professional development.

3.9.8 The quality of the review was not affected whether there were two or three reviewers; however, where there were three team members it was possible to cover more ground. The third member also provided an extra check and additional input which gave greater confidence that the final report was an accurate reflection of the service.

3.9.9 The best mix of Review Team members was for a peer reviewer to provide operational insight, a MGS officer to provide national context and professional credibility and a consultant to provide ‘fresh eyes’, culture sector overview and independence. For a two member Review Team, a peer reviewer plus either an MGS manager or consultant was felt to be most effective.

3.9.10 Pre-meetings (when Review Team members come together in advance of the on-site visit to discuss the submission) were felt to be extremely valuable, but the pilot demonstrated that these weren’t always necessary where geographical barriers prevented them from happening. If a pre-meeting is not possible, it is clearly essential that reviewers prepare for the on-site visit thoroughly and that the pre-meeting is replaced with a scheduled phone call(s) to cover the same ground as a pre-meeting.

3.9.11 Typically the validation process for each site including the planning, preparation, execution and writing of the final report required three and half days to four days input for the Review Team leader and two days input of each of the other team members.

21


3.9.12 Pilotees felt that Review Team members were knowledgeable, credible and had prepared well for their visits; although, the independent museum pilotee felt the Review Team should include someone with experience of this sector.

3.9.13 One of the local authority pilotees felt that on-site visits could be more robust and that the principal of triangulation (that is forming a judgement using more than one piece of evidence) could be applied more rigorously.

3.9.14 Of the six reports which summarised the Review Team findings, four were signed off with only minor factual amendments while the other two were finalised after amendments were made relating to the interpretation of evidence presented in the report.

Recommendations1. Overall the process for Review Teams was proven to be sound; however, there would

appear to be some scope for fine tuning before being replicated. Before finalising the process we would recommend taking into account the views of the local authority pilotee which felt the on-site visit element could be improved further.

2. Peer reviewers should be used wherever possible as members of the Review Team and supplemented by MGS staff or consultants. Recruitment and development of peer reviewers should be the responsibility of the QIS coordinator. We recommend a rolling programme of peer reviewer recruitment and that peer reviewers serve a two-year term. Peer reviewers should be able to complete two reviews in a one year period. The number of peer reviewers required will be determined by the number of QIS participants in any one year period

3. In the first instance peer reviewers should be recruited from organisations that have already been through QIS.

4. The guidance notes issued to Review Team members proved sufficient to explain their role and input into the process. Training and development of Review Team members (including peer reviewers) can be supplemented by their attendance at QIS participant briefing sessions.

5. Where possible Review Teams should consist of three members for large and complex organisations. Two members is sufficient for smaller local authorities, independent and university museums.

6. The style and content of the final report should be replicated taking into account the recommendations in the next section (post-validation) overleaf.

7. The report writing can be onerous if left for too long after the site visit. The aim should be to complete summary reports within seven days of the on site visit so a draft can be circulated to the Review Team for comment and a final draft sent to the QIS participant within a fortnight of the visit.

22


3.10 Post validation 3.10.1 The underlying principle of continuous improvement is that it is a cyclical process. In

practical terms this means organisations should implement the improvements they have identified, self-evaluate other aspects of their business and re-evaluate Impact Measures at regular intervals. The pilot was unable to test ‘what happens next’; however, prior to a national roll out there should be consideration of what (if any) monitoring and review of QIS organisations takes place.

Recommendations1. The final report should include a concluding section on the organisations capacity to

implement improvement and summarise what happens next. This should reference any obstacles (if any) the organisation may face in implementing its action plan in the short to medium term. It could also include the priorities for action.

2. MGS should consider how it will monitor organisations improvement on an on-going basis. This will help ensure improvement actions are implemented. Without this check there is a risk organisations will consider their completed submission as the end of the process rather than the start of it.

23


“QIS is a very good idea and very relevant to small independents as an acid test of excellence”

4.0 QIS and the Independent Museum Sector4.1 The independent museum sector is characterised by many small museums. Many of them

manage on a shoestring budget, might have just one paid full time member of staff and are heavily reliant upon volunteers. A large number operate as stand-alone trusts reporting to a board of trustees. Many receive some funding from their local authority which is an important part of their total revenue stream. In some cases this will be underpinned by a service level agreement which stipulates what the museum will provide in return. Museums going through Accreditation or who are Accredited will benefit from the appointment of curatorial advisers. Some local authorities provide officer support to assist the independent sector e.g. (Highland Council); however, the majority of independent museums tend to rely on an informal support network.

4.2 This lack of resources means that independents often have to manage in creative and innovative ways. Relationships with partner organisations and community groups can be very effective but are often informal which can make it difficult to evidence partnership working.

4.3 If QIS is to be relevant to the whole museum sector it needs to be mindful of the different set of challenges and environments that characterise this sector compared to a local authority museum service which may have a large number of sites, employ large numbers of paid staff, engage in outreach work and be knowledgeable about local and national outcomes and other public sector agendas.

4.4 To help determine whether the QIS framework is relevant and appropriate for the independent sector, we involved a small independent museum in the QIS pilot and organised a consultation event with representatives from 20 independent museums in the Highland Council area. Both exercises produced rich qualitative information which should be taken into account when QIS is refined.

The pilotee experience – Auchindrain Museum4.5 Auchindrain Museum is a small and remote site which attracts in the region of 4-5,000

visitors per year. The museum has a curator who reports to a Board of Trustees. As well as managing the museum, a service level agreement with the council requires him to provide curatorial support to the council owned museum. The museum employs a further full-time and one part-time member of staff and is reliant upon several volunteers.

4.6 The museum decided to complete three sections of QIS. It managed to complete all three sections within the timescale, although it helped that the museum was closed to the public during the time QIS was carried out.

4.7 Much of the feedback about the experience of going through QIS and the relevance of QIS to a small independent museum was positive. Comments included:

24


“It is not onerous – unlike we feared”

“A very revealing process – it has made us feel good about ourselves in a number of areas”

4.8 Less favourable comments and observations related to the language and evidence requirements:

“It has a local authority look and feel – it may need pared down for independents”

“To make QIS user-friendly to independents it needs to be understood that our mindset is different to that of a local authority”

“In very few cases is there a standard paper-trail... I think this is partly a product of the organisation's small size and informal nature, but it is also a consequence of trying to achieve a great deal in a very short space of time - we do the job and don't bother with the paper-trail because there isn't time”

4.9 Other areas where QIS could be made more relevant for the independent sector include:

Some areas being evaluated did not seem relevant or are outside the control of independents e.g. contribution to community planning. The applicability of these areas for independent museums should be reconsidered

Some of the language was hard to grapple with to the extent the museum found it hard to understand what some questions were asking

There seemed to be some duplication. Some prompt questions appeared to ask the same question in a slightly different way

Some of the Impact Measures seemed to overlap and request information which had already been provided elsewhere

A less formal approach to on-site reviews would be beneficial

Don’t view the purpose of an independent museum through a local authority perspective

Need to be explicit in terms of what we ask for as part of the submission documents

Evidence less likely to be ‘hard’ e.g. documents, minutes, formal processes. QIS needs to take a wide approach to what it regards as evidence e.g. speaking to people to corroborate evidence, e mail correspondence and observation.

25


Independent museums consultation event4.10 The consultation event involved 20 small independent museums. Around half of the

participants had read through the framework in advance while the remainder were hearing about QIS for the first time. Given this was the first contact many had had with QIS there was a great deal of engagement with the presentation and interest in the framework. Much of the discussion related to how QIS might be used in future e.g. to help with funding applications, how it aligned to Accreditation and the practicalities for small, not very well resourced museums implementing it.

4.11 Headline feedback included:

Approximately 70% of participants thought QIS was a useful framework although some remain to be convinced that this is what they should be spending their time on right now

Many viewed QIS through the prism of Accreditation and found it hard to differentiate between having a minimum standard (Accreditation) and an improvement framework (QIS). Some of the participants struggled to see how QIS was substantially different from Accreditation and why we needed both. There was also a concern that QIS and the new version of Accreditation might be similar and require a duplication of effort

Some said that if they went through QIS they would like some form of recognition for doing so and wondered if QIS could be tied into funding applications

On a practical level there was agreement that in its current form the framework could look overwhelming to some and this would need to be addressed

Some felt there were too many prompt questions and several looked similar so there was scope to rationalise these

There was very low awareness and understanding of national outcomes and Single Outcome Agreements let alone how museums contribute to these. General low awareness of the need for the sector demonstrate their contribution – “we’ve been around for years so we must be doing something right”

The purpose of ‘doing QIS’ was not clear. We can use Accreditation to support funding applications which we understand, but using QIS to ‘get better’ seems a bit woolly

It would be helpful to map the framework to national outcomes

Small independents have informal networks, so it is not always possible for them to provide evidence of effectiveness so the framework needs to be flexible

How might QIS be used for benchmarking in future?

26


Implications for this study4.12 To be relevant and appropriate to the independent museums sector the framework should:

Recognise the scale of operation and sphere of influence of independent museums. The language should resonate with the everyday experiences of running a small site

It is essential that there is on-going support for independents going through the process – also it is very unlikely that an independent would go through the process if there is a cost attached

Explain how QIS is not the same as Accreditation. MGS needs to demonstrate the two frameworks are aligned but also needs to clarify what the ‘clear blue water’ is

Convince doubters of the compelling advantages of undertaking QIS.

Recommendations1. Edit and refine the existing QIS framework to acknowledge the differences highlighted

between a small independent museum and a local authority service. This should be a shorter and punchier document which is more accessible and less overwhelming.

2. Use clearer language - less “local authority speak”.

3. Consider how those going through QIS might receive recognition for doing so. One option could be to develop a QIS logo which features on a plaque or certificate which can be displayed publicly. This should identify that the service or site has been MGS-QIS validated and is committed to improving performance.

27


“It reminded everybody of the need to match the day-to-day with the vision”

5.0 QIS and university museums5.1 University museums are unique within the museum sector. Much of their reason for

existence is to serve their student and academic audience; however, most also have an important remit to make their collections as accessible as possible to the wider public. This means they attempt to contribute to the aims of the University as well as the wider cultural offer and aspirations within the town or city where they are located. Governance arrangements means the museum is likely to be funded by and report to the University management team. The Hunterian Museum which is part of the University of Glasgow completed three Impact Measures as part of the QIS pilot.

The pilotee experience – The Hunterian Museum5.2 The museum managed to complete QIS within the timescale but found this a very

demanding process. There were a couple of factors that contributed to this including the retirement of the museum’s Director at the start of the process which meant that staff resources were extremely stretched. A major renovation project which ran concurrently with the pilot was also a distraction. With the benefit of hindsight, choosing to undertake three Impact Measures was unrealistic and created undue pressure.

5.3 These pressures meant that Impact Measures tended to be completed by a designated senior member of staff rather than a team or whole staff approach being adopted.

5.4 Despite these pressures QIS was viewed favourably as a process which helps identify priorities, encourages better planning and a less reactive approach to ‘what gets done around here’.

5.5 Feedback confirmed that there is some duplication and a feeling that some of the prompt questions did not appear directly relevant to the museum or central to its core business e.g. community planning.

5.6 Some of the staff involved had previous hands-on experience of quality assurance models which helped decode the QIS framework.

5.7 There seemed to be some uncertainty about how much evidence was required and what constituted relevant evidence. The submission was ‘evidence heavy’ which placed a greater burden on the pilotee and was unhelpful for the Review Team.

5.8 There seemed to be some uncertainty about the over-riding purpose of QIS. The museum appeared to regard their submission as the end of the process and need convincing of the need to continue evaluating using the QIS framework. Perhaps because of the staff resource constraints there was a focus to ‘just get through it’.

Implications for this study As noted elsewhere there is a need to provide clear guidance about how much evidence

is enough and what this should consist of

28


The guidance should also highlight the necessity and value of tackling QIS as a team rather than individuals being responsible for separate sections and working on them independently

A need to reinforce at the start of the process the over-riding purpose of embarking on QIS and the long-term commitment required

Perhaps the biggest implication, which is relevant to all QIS participants, is that organisations who are in the midst of significant change, departmental restructures or implementing other major projects should think long and hard about their ability to commit to QIS as well.

Recommendations1. Most of the framework appears relevant to the university sector; however, there are

some prompt questions which are not relevant or central to what they do. Given the relatively low number of university museums it is difficult to justify a separate QIS framework, therefore the refined version of QIS should consider the needs of this sector.

29


“I like the fact it is about performance management and a framework for genuine discussion”

6.0 QIS and the National Collections6.1 The four National Collections of Scotland are overseen and funded by the Scottish

Government. They are responsible for collecting and publicly exhibiting items and archives of national and international importance. They include:

The National Library of Scotland

National Museums Scotland

National Galleries Scotland

The Royal Commission on the Ancient and Historical Monuments of Scotland (RCAHMS).

6.2 A meeting with RCAHMS was held to introduce the framework and obtain feedback on its content, structure, usefulness to the National Collections and its applicability. RCAHMS is a large and complex organisation working at national and local level and therefore is a good test of the frameworks relevance and ‘useability’. Feedback from RCAHMS included:

QIS is relevant and applicable to National Collections organisations

RCAHMS liked QIS because :

- It is flexible i.e. not about meeting pre-ordained standards

- It is well laid out and covers essential areas

- It focuses on an organisations goals and outcomes

- The validation process and evidence submission does not appear to be burdensome

- It looks more manageable than Accreditation

- It is “not scary”.

RCAHMS currently uses a range of national standards and has to demonstrate how it performs against the National Performance Framework. Aims and work plans are articulated in their Corporate Plan. QIS represents an opportunity to review improvement across a range of operational areas and would be particularly helpful as a planning tool which integrates with the Corporate Plan

Some elements of QIS were neither central to nor relevant to their work e.g. references to community planning. A version of QIS which edited out these elements would be far more attractive to the National Collections

30


Some areas of work which are core to RCAHMS work are omitted, but these could be easily incorporated if some of the Impact Measures are less specific

MLA not MGS has always been responsible for assessing and validating Nationals for Accreditation; therefore, peer review input at the review stage is essential for any of the Nationals embarking on QIS

RCAHMS said that all The National Collections would have to approve and agree to carry out QIS before any individual organisation could embark on this. It is recognised that this could be a substantial barrier to overcome.

Recommendations1. Consult and obtain feedback from the other National Collections.

2. Using feedback, refine and revise the QIS framework so it is more relevant to the National Collections and consider producing a National Collections version of QIS.

31


“I like it’s breadth – it covers all parts of the service including staff and stakeholders”

7.0 Alignment with HGIOCS?7.1 How Good is our Culture and Sport? (HGIOCS?), is the quality framework for culture and

sport services provided by local authorities. It is intended that all culture and sport sectors will develop ‘Taking a closer look...’ frameworks which are integral to HGIOCS? but sit beneath it. The HGIOCS? framework exists but has not yet been piloted.

7.2 When the QIS framework was developed, HGIOCS?, did not exist; therefore, it is not surprising that there is scope to improve the alignment and integrate QIS better with this framework.

7.3 Despite this, a mapping exercise conducted by MGS prior to the pilot commencing, demonstrated how well the QIS impact measures relate to HGIOCS? Quality Indicators. While the language and content in QIS could be developed further to make these relationships even clearer it is our view that there is no need for a significant overhaul of QIS.

7.4 The QIS pilot was delayed to enable QIS to align with HGIOCS?. It was hoped that these pilots would develop in tandem, to allow any significant issues to be considered and incorporated into the frameworks. Unfortunately the HGIOCS? pilot did not start until the QIS pilot finished, so it has not been possible to incorporate any lessons learned from this exercise which is ongoing. Representatives from MGS and the consultant have attended HGIOCS? Implementation Group meetings. These meetings improved understanding of how the QIS framework could be better aligned to HGIOCS?, while the findings from the QIS pilot have been used to shape the HGIOCS? pilot process.

7.5 Some of the other key points to have emerged, which have implications for the refinement of QIS, are:

QIS Impact Measures need to better signpost the HGIOCS? Quality Indicators they relate to

The QIS framework could strengthen its relationship with HGIOCS? by clearly setting out that it is an integral part of the overarching framework. One way QIS could do this is to adopt the ‘core script’ used on the Scottish Government website in the self-evaluation guidance introduction

The wording and terminology in QIS should mirror those in HGIOCS?

A level two illustration in addition to the level 5 illustration in QIS would mirror the HGIOCS? framework and help pilotees assess award levels.

32


Recommendations1. Refine QIS so that it uses similar language and terminology to the HGIOCS? framework.

2. Link each Impact Measure in the QIS framework to the appropriate HGIOCS? Quality indicator.

3. Write a level 2 illustration for each Impact Measure which is equivalent to the level 2 illustration in HGIOCS?.

33


8.0 Alignment with Accreditation

State of play8.1 QIS is designed to provide a framework for continuous improvement once museums have

achieved Accreditation. Perhaps the most significant difference between the current Accreditation scheme and QIS is that Accreditation requires organisations to meet a minimum standard, while QIS is interested in encouraging organisations to develop an improvement action plan

8.2 There are other areas of ‘clear blue water’ which taken together make a compelling case for QIS running alongside Accreditation, these include:

Accreditation QISPrimarily concerned with inputs and outputs Concerned with outcomes and impacts

States “you must have...” Asks “how well do you...” using existing evidence

States evidence required Asks you to demonstrate how well you are doing

Is a UK scheme Recognises the Scottish policy context

Multi-site organisations must complete applications for each site

Multi-site organisations complete one submission for the whole organisation

Organisations achieve the Accreditation standard. Successful organisations are monitored through biennial returns

QIS provides an accurate reflection of a level on a spectrum of continuous development. While achieving an ‘award level’ you do not ‘get’ QIS – continuous improvement is a cyclical process which means organisations should be continually evaluating how good they are and implementing improvement actions

The primary motivation for doing this is to be able to access funding

The primary motivation for those doing this is to continually improve services and maximise their impact

8.3 The MLA is currently leading the Accreditation Development Project, aimed at ensuring that the current Accreditation scheme (launched in 2004) remains helpful, relevant and appropriate. To this end MGS, other home countries museums membership organisations and the MLA met in 2009 to discuss the devolved issues around the development of the Accreditation scheme and agreed the following:

Commitment to maintaining and developing Accreditation as a nationally agreed standard for museums, for public benefit

Appropriate branding to reflect UK coverage, ensuring representation from all devolved countries

34


Using Accreditation as a foundation, working in partnership across the UK to ensure continuous improvement while respecting national initiatives (this point relates to the development of QIS as a continuous improvement model in Scotland).

8.4 In addition MGS are members of the Accreditation Advisory Panel and have worked closely with MLA on Accreditation and regularly report on the progress of QIS. The joint Museums Standards event held in Glasgow on 16 September 2009 is evidence of close and effective partnership working.

8.5 Following a comprehensive consultation process, a new Accreditation scheme is likely to be launched in autumn 2010.

QIS and Accreditation – a twin track approach8.6 Information gathered as part of the QIS pilot has already fed into the development of the

Accreditation Scheme to ensure that both QIS and the new Accreditation models are streamlined as much as possible and meet the requirements of the sector.

8.7 A potential difficulty with the development of a new Accreditation framework is that if it looks similar to QIS and incorporates some of the principles of a continuous improvement model it could blur the distinction between it and the need for QIS in the eyes of those who are responsible for implementing the schemes. This could result in organisations choosing to do one or the other.

8.8 The emerging Accreditation model appears to borrow elements from the QIS framework. The most recent proposed structure for Accreditation includes:

Questions which ask “How do you...” rather than “Do you...”

Greater emphasis on outcomes

Inclusion of criteria which relate to community engagement.

8.9 A worst case scenario would be the development of a new Accreditation model which is neither a minimum standard nor a systematic continuous improvement model. This could devalue Accreditation and the reputation it has built up over a number of years, while at the same time weakening the need for QIS but leaving many issues unresolved, not least the lack of recognition of Scottish policy and non-alignment with HGIOCS?.

8.10 A best case scenario is for QIS to be acknowledged as an integral part of the Accreditation options being considered.

Recommendations1. MGS will need to continue to work closely with MLA to ensure it keeps abreast of latest

developments to ensure the two frameworks dovetail rather than compete with each other to avoid the worst case scenario.

35


2. Feedback from those introduced to the QIS concept for the first time at consultation events, suggests that more work is required to explain the differences between the frameworks and why it is in the interests of organisations to embark on QIS and Accreditation. MGS and MLA need to be clear about the future purpose of both frameworks and communicate this to their audience.

3. The QIS guidance notes should set out more clearly the purpose of both frameworks and how they relate to each other.

4. Explore the scope for an organisation which achieves Accreditation being able to use this as evidence for elements of QIS.

36


“It’s a practical tool – it can focus on areas you want people to notice”

9.0 Resource issues9.1 If QIS is to become a fixture of the museum quality landscape, consideration needs to be

given to its sustainability. The biggest resource issue for QIS is the level of dedicated human resources to maintain this process. Key tasks are:

Manage the number of participants going through QIS at any one moment in time

Brief participants

Provide ongoing support

Recruit, brief and manage peer-reviewers

Co-ordinate and carry out submission reviews including on-site visits

Write summary reports

Carry out follow-up meetings to monitor improvement progress

Promote the benefits of QIS to ensure future take up

Organise and disseminate benchmark information.

9.2 A useful model to consider in these respects is PLQIM, for whom two senior staff from SLIC have responsibility for carrying out these tasks. The applicability of this model for MGS will clearly depend on staff capacity.

9.4 Other quality models such as Investors in People and Quest will also shed light on capacity and resource issues.

9.3 An alternative to this approach is to commission consultants or another third party to manage the scheme on behalf of MGS. The ongoing costs of this model and other advantages/disadvantages need to be considered and compared to QIS being managed in-house.

37


Recommendations 1. In the first instance identify in-house staff that have the skills and experience to manage

QIS and others who may be able to support this function.

2. If there is limited in house capacity, weigh up the pros and cons of recruiting staff to fill this role or using third parties to manage QIS.

3. Consider the use of consultants to provide additional support, for instance validation visits under a framework agreement.

4. Research other quality models to understand how they approach capacity and resource issues.

5. Develop links with other organisations that may be able to play an active role in the QIS process, for example involving VisitScotland in the mystery visit component.

38


“It enables you to focus on key areas with senior managers and elected reps”

10.0 Next steps – future roll out

10.1 The recommended next steps for developing QIS are set out below:

Step Task

1. Revise and refine master QIS framework document and toolkit resources based on the pilot findings.

2. Ensure that the revised framework is relevant and appropriate for independent museums and the National Collections.

3. Gain support in principle from the National Collections and a commitment to implement QIS.

4. Set target date for launch and develop a four year programme for QIS which identifies staff resources required, budget and a target number of museum organisations who will implement QIS.

5. Identify staff resources within existing establishment or recruit a member(s) of staff who will have responsibility for managing all aspects of QIS for the next four years. If no member of staff can be identified or recruitment is not possible, consider the use of third parties to manage this process on behalf of MGS.

6. Recruit a team of peer reviewers to serve a two year term.

7. QIS launch – recruit first tranche of participants.

8. Annual monitoring and review of the programme.

9. Align QIS with HGIOCS?

10. Work with the Accreditation Development Project to align systems.

39


11.0 QIS application process

11.1 The number of participants MGS can support through QIS is informed by the experience of SLIC. PLQIM was launched in 2007 and in the three and a half years to date, SLIC have carried out 28 on-site visits with another two pending. PLQIM is supported by two full-time members of staff who also fulfil other duties. With a dedicated member of staff supported by others, MGS could aim to have all 32 local authorities carrying out QIS within four years. Unlike the libraries sector, MGS also serves a large independent sector which it wishes to see adopt QIS. Rather than set targets for the independent sector it may be more prudent to set limits on the number of independent museums it can support at any one time in addition to the local authority participants.

11.2 Clearly QIS will need to be managed in a way which ensures those going through QIS can be adequately supported. We propose that the application process initially allows for a maximum of four local authority or trusts and two university or independent museums going through QIS in any six month period. This number can be increased whenever the project manager feels there is slack in the system or resources allow. For instance if participants only choose to complete one Impact Measure or organisations complete their QIS commitments ahead of schedule. Although six months has been allowed for each tranche of participants to complete QIS, they should be encouraged to complete submissions within four months of starting.

11.3 The proposed application process is set out in Diagram 11.1:

11.4 In addition, a step-by-step approach for those going through the QIS process in future is set out in Appendix A.

40


Diagram 11.1 – QIS Application Process

41


12.0 Conclusions

12.1 The pilot demonstrated that QIS is a quality system which works, produces tangible benefits and is well regarded by those who took part. All pilotees said that they would go through QIS again and all would recommend it to other museum organisations.

12.2 The framework as it stands, works better for local authority museum services and large trusts than small independent museums and to a lesser extent university museums who found it hard to relate to some of the criteria. Refinements to the framework need to reflect this.

12.3 A refined version of QIS which acknowledges and reflects the needs of the independent museums sector and National Collections will help ensure broader support for the framework.

12.4 While the experiences of pilotees are mostly positive there are a number of areas where the framework can be improved and made even more relevant and practical. Most of the suggested refinements relate to the structure, language and content of the framework rather than the process.

12.5 QIS aligns well with HGIOCS? as evidenced by the MGS mapping exercise. There is scope to fine tune the QIS content and language to ensure they are more consistent which will reinforce the relationship between the two frameworks.

12.6 Accreditation could change significantly over the coming months. MGS needs to influence developments to ensure the new version is compatible with QIS.

12.7 There is room for both QIS and Accreditation which serve different purposes; however, the museum sector in Scotland will remain to be convinced of the need to do both. QIS pilotees could be valuable advocates for the roll out of QIS in future.

12.8 Before timescales for a roll out of QIS can be considered MGS will need to continue working to align QIS with HGIOCS? and Accreditation.

42


Appendix A – Step-by-step approach for QIS applicantsTask Who By when2

Preparation

Weeks 1- 4

Before embarking on QIS, secure the agreement and commitment of key people within the organisation (e.g. trustees, heads of service, Corporate Management Team, elected members, etc.). Explain rationale and benefits, expected time commitment, timescales. Also secure agreement on approach, scope and process

Decide and agree the submission deadline date with MGS

Set up a self –evaluation team comprising of a cross-section of staff to oversee/manage the self-evaluation process.

Attend the QIS briefing meeting (include as many of your self-evaluation team as possible)

Make a final decision on how many and which QIS sections to tackle.

Draw up a meetings schedule with key milestones include the number, timing and content of self –evaluation sessions. Ideally the team should meet weekly.

Brief the rest of the organisation, partners and stakeholders.

Agree roles and responsibilities of self-evaluation team members and establish self-evaluation methods – what, how and by whom will evidence be gathered and strengths and areas for improvement evaluated.

Conducting self-evaluation

Weeks 5-15

Form an initial broad view of the organisations strengths and areas for improvement; consider existing evidence which supports your strengths and any evidence gaps which can be easily plugged.

Consider scheduling of user focus groups, self-evaluation workshops and/or interviews and meetings with partners, paid staff, volunteers and other stakeholders as appropriate. This will add to your evidence base and help you form a judgement on your level of performance - test the robustness of the evidence.

Form a consensus on your performance level for each section, develop your report and start to develop an improvement plan as you go.

Validation: before finalising your submission consider asking someone within your organisation or someone who can act as a ‘critical friend’ to challenge your evidence base and the judgements you have reached.

Complete written submission ensuring compliance with submission criteria.

2 This assumes that no more than three sections are attempted at once43


Validation and reporting findings

Weeks 16-20

Prepare for the on-site visit by the Review Team. Arrange for members of your self-evaluation team to be available on the day as well as other key personnel as requested. Be prepared to substantiate the judgements made in your submission.

Consider the summary of findings report provided by the Review Team (this will normally be sent to you within two to three weeks of the on-site visit) and feedback any comments on factual errors or perceived misinterpretation.

Once the report has been signed off, report the findings within the organisation (e.g. trustees, university board, local authority committee, etc.) and consider how you may be able to use the report as an advocacy tool to secure support for identified areas for improvement. Consider communicating the report headlines to partners and stakeholders

Agree a prioritised improvement plan with SMART actions and start to implement these.

Monitor, review and amend the improvement plan Ongoing

Consider which QIS sections to tackle next

44


Contact detailsFor further information on this report contact:

Tim DentDirectorThe Sport and Leisure Consultancy26 Nile GroveEdinburgh EH10 4RFTel 07747 024191e mail: [email protected]

45

Museums Galleries Scotland · Web viewAdditionally, If the ‘museums should’...

Documents

Transcript of Museums Galleries Scotland · Web viewAdditionally, If the ‘museums should’...