West Midlands Museum Development Officer Network

Family Friendly Mystery Shopper 2010-11

Evaluation Report

Executive Summary

The 2010/2011 Mystery Shopping project built on a previous project which took place in 2009.

Its aim was to encourage museum and gallery staff to think more deeply about the overall quality of experience offered to family visitors.

It was designed both to help staff develop their own skills and confidence, and to sharing skills, knowledge and experiences within the sector.

The project looked at four key areas: the pre-visit experience (‘Virtual visits’), arrival and welcome, the displays (‘the sold experience’) and facilities such as shop, café and toilets.

Shoppers were generally complimentary about both the pre-visit experience and the sense of welcome they received. They were also positive about the overall experience of exhibitions and interpretation. Facilities were generally rated as good or adequate.

The museums that were most highly rated by the Mystery Shoppers had well laid out websites, attractive promotional material and easy to read events calendars. They realised the importance of first impressions, with friendly and enthusiastic staff. The core offer included specific provision for children and opportunities for children and adults to learn together. In short, they all paid great attention to detail.

Feedback to the participating venues was provided mostly in the form of written reports, although a small number of museums were able to have face-to-face meetings with their mystery shoppers.

Feedback from the participating venues has been consistently positive and shows that the project successfully achieved its aims. Both shoppers and venues stated how much they had valued the experience.

Page 1 of 16

Contents

Page 3 Introduction and background

Page 4 The process

Page 6 Key findings from mystery shopping visits

Page 9 The evaluation process

Page 10 Evaluation Feedback

Page 2 of 16

Introduction and background

The Mystery Shopper project aims to provide an independent review of the quality of family friendly facilities and activities in museums. Participating museums are visited by a mystery shopper who assesses the venue against a standard set of criteria. Following each visit, the results are fed back to the museum. The mystery shopper’s report is intended to help the museum to assess its family friendly offer from a new perspective. Concentrating on simple but effective improvements, the project is designed to help museums improve their services for families.

The Marches network had the overall management of the project, and the West Midlands Museum Development Officers co-ordinated the project and collated evaluation materials in their respective areas. One region (Conurbation) provided a small grant to implement recommendations made by mystery shoppers.

The project was funded through the MLA Renaissance programme via the West Midlands Museum Hub.

Mystery Shopper Project Partners:

Conurbation Museums Officer: Wolverhampton Arts + Heritage Service

Herefordshire Museum Development Officer: Herefordshire Museums Service

Shropshire Museum Development Officer: Shropshire Museums Service

Staffordshire Museum Development Officer: Staffordshire Arts + Museums Service

Community Museums Officer for Warwickshire, Coventry & Solihull: Warwickshire Museum Service

Page 3 of 16

The Process

Background

The 2010/2011 Mystery Shopping project built on a previous project which took place in 2009. Its aim was to encourage front-line and support staff in museums and heritage attractions to think more deeply about the quality of experience offered to families visiting their museums and galleries. It was designed both to help staff develop their own skills and confidence, and to foster a family friendly approach to visitors in their museums.

The Mystery Shoppers

Shoppers were recruited mostly from participating museums and attractions. They were selected to include a cross section of both front-of-house and other staff with a variety of skills and interests. In order that they did not have to travel long distances to visit their venues shoppers were allocated sites within a 20-30 mile radius of their own museum.

Training

Three training days were organised for both new and existing shoppers. The training sessions were designed to provide a ‘refresher’ for previous participants and to introduce the new ‘self-organising’ toolkit that had been generated by the earlier project. Guidance was also given on how to write and give constructive criticism. Shoppers were also reminded that family friendliness does not depend simply on the amount of money spent by a venue on its facilities or events programme.

Mystery Shopper visits

Visits were made to twenty-two museums and heritage sites in the region. These included seven local authority run venues, a National Trust property and fourteen independent museums, including both professionally staffed sites such as Avoncroft and smaller, volunteer run sites. Visits took place between late October 2010 and mid-January 2011. One of the museums was closed for the winter during this period.

In general the visits followed the original plan and the exercise went much as most participants had anticipated. In one instance, a shopper asked members of a visiting school group some questions about their experiences. Two shoppers took along an additional, first-time visitor, as they thought this might offer a fresh perspective on venues with which they were already familiar. One venue expressed surprise that the mystery shoppers did not bring children with them or come at a weekend or during a school holiday.

One shopper did the ‘visit’ retrospectively basing the report on the experience of a previous trip as the museum was closed during the mystery shopping period.

The questionnaireDuring the visit the shopper used a questionnaire in order to assess the family friendliness of the site. The questionnaire and guidance notes were a revised version of those used in the previous project.

The questionnaire covered four key areas.

Page 4 of 16

The pre-visit experience – virtual visits and information gatheringThis looked at the attraction’s web-site, asking whether it was easy to find, navigate, and use, printed promotional material, and visitor enquiry services. Shoppers made e-mail and telephone enquiries to each venue in advance of their visits. The visit experience – the welcome environmentThis section looked at the quality and utility of the attraction’s approach signage, the external welcome environment, internal visitor welcome and the quality of service offered by the reception desk. Particular attention was given to issues such as environment, branding, visitor orientation, presentation, ambience, lighting and cleanliness The quality of the displays (the ‘sold attraction’)This section looked at the museum’s core business (the displays rather than ancillary services), in particular the quality of presentation and interpretation, and the level of opportunity for family interaction and learning. The quality of facilities and servicesThe final section looked at the range of facilities and services offered to families, including toilets and baby changing provision, shop stock, layout, and presentation, and the quality of refreshments available in the café.

All questions were scored by using the following scale: very good good ok poor

The overall results were very encouraging, with 79% of facilities and services within the participating museums being rated as ‘very good’ or ‘good’.

very good good ok poor143 104 43 2146% 33% 14% 7%

Feedback

Following the visit the contents of the completed questionnaires were fed back to the participating venues.

A small number of museums had face-to-face follow up meetings with their shoppers. This enabled them to question their findings, seek clarification to their questions and discuss ideas for improvement.

Page 5 of 16

Key findings from Mystery Shopping visits

Virtual visits

The pre-visit experience was generally of a high standard. Twelve websites were rated as ‘very good’ and only one as ‘poor’. The calendar on one website had not been updated, while the contact phone number given on another was out of date. Promotional material was mostly rated as ‘good’ or ‘very good’ with only one museum’s material considered inadequate. The response to telephone and email enquiries was generally quick and accurate, although the shopper was unable to test the quality of response to a telephone enquiry at one museum as no-one answered. This may, however, have been the result of poor weather. In two instances, there was no response at all to the e-mail enquiry.

Arrival and welcome

Shoppers were generally complimentary about the sense of welcome they received. None of the museums were rated as giving a poor first impression. One museum was poorly sign-posted from outside. Another was closed despite having advertised an open day and seemed to offer no notice of apology or explanation to disappointed visitors.

The sold experience

Shoppers were positive about the overall experience of exhibitions and interpretation. None of the museum displays were rated as offering poor value as part of the whole visit. At only one venue did they find limited opportunities for children and families to engage with the site, while the quality of presentation in one venue was seen as poor.

Facilities

Toilet/changing and other facilities for families were generally rated as good or adequate. The shopper was unable to use the facilities at the museum that was closed and consequently rated these as ‘poor’.

Shops were mostly well laid out with appropriate and affordable merchandise. The layout of one museum shop was described as ‘a little haphazard and can feel cramped with more than just a couple of visitors.’

The food in one museum was given a poor score because it only served hot and cold drinks, cakes and crisps. It did not have any healthy cold snack options available, such as fruit, or cold meals such as a ploughman’s lunch or sandwiches nor were there any 'child friendly' options.

Key characteristics

Four of the museums were rated as ‘good’ or ‘very good’ in all categories. More than half had no ‘poor’ scores at all. Only one museum failed to achieve at least one ‘very good’ rating. The museums that were most highly rated by the Mystery Shoppers had a number of common characteristics. Their websites were well laid out, easy to use and informative. The best included family sections or a children’s zone. Their promotional material was attractive, informative and well-illustrated, but concise. The best included easy to read events calendars, with clear details of children’s activities. They realised the importance of first

Page 6 of 16

impressions, with friendly and enthusiastic receptionists keen to offer help and advice to visitors. Internal signposting was clear and they avoided clutter. The core offer included not only specific provision for children such as inter-actives, trails and activity areas, but also opportunities for children and adults to learn together. Their shops were attractively laid out with a good range of pocket-money goods relevant to their collections. Baby-changing and other facilities were easy to find and well-maintained. In short, they all paid attention to detail.

Mystery shopper questionnaire

Very good

good ok poor

1. How good is the attraction's web site? How easy is it to find and is the information presented accessible, useful and well presented?

13 5 2 1

2. How good is the attractions promotional material, Does it make you want to visit the attraction?

9 8 1 1

3. How good is the response to your telephone enquiry? Is it handled well and was accurate information given?

11 5 2 2

4. How good is the response to your email enquiry? Is it handled quickly and was useful information given?

13 1 2 4

5. How good was the attraction's approach signage? Did it help make the attraction easy to find?

6 9 5 1

6. How good is the sense of welcome, considering environment, branding, signing & lighting?

11 6 3 1

7. How welcoming is the attraction inside, considering ambience, environment, lighting, branding & signing. How good is your first impression?

12 6 2 0

8. What is your impression of the counter/enquiry desk? 12 6 2 09. How good was the presentation? Was it clean, tidy, well lit and inviting?

11 6 3 1

10. How good was the sold experience? What was the overall value of the exhibitions or interpretation as a part of the whole visit?

9 9 2 0

11. How good was the sold experience in terms of opportunities for engaging with children & families through fun and interactive activities & media?

7 9 3 2

12. How good were the toilet/changing and other facilities for families?

6 4 8 2

13. How good was the shop's layout and presentation? 10 7 0 114. How good was the shop's merchandise and pricing? 3 12 3 015. How good was the café? Was it clean, well presented, inviting and friendly?

7 5 2 2

16. How good was the café's food and drink? Was there a good range of good quality food suitable for families at affordable prices?

3 6 3 3

Totals 143 104 43 21

Page 7 of 16

The Evaluation process

Both shoppers and venues were asked to complete an evaluation form. This was designed to find out what they felt about the mystery shopping process, what worked and what didn’t, and how it could be developed or broadened.

Participants were also asked how the experience challenged their views or expectations and what improvements they now proposed to make in their museums.

The response was overwhelmingly positive, and is covered in greater depth later in this report (Evaluation feedback). The key findings are summarised below.

Key findings from evaluation

Both shoppers and venues were very positive about the mystery shopping process, describing it as well thought out, comprehensive and easy to understand.

Both shoppers and venues viewed the experience as a useful professional development opportunity that helped to give them a wider perspective.

Shoppers felt that they had gained practical insights into what actually works in other museums and identified elements of good practice that they intended to adopt in their own venues.

By looking at the site from a different perspective, shoppers were able to highlight issues that had previously been overlooked.

Venues appreciated the shoppers’ honesty and generally welcomed constructive criticism and suggestions for improvement.

Venues have a better understanding of what is important to visitors and of the need to adopt a holistic approach to visitor provision.

Limited space and resources do not prevent a museum from becoming more family-friendly and small improvements can make a big difference to the overall experience of a visit.

Family friendliness needs to be integrated within all aspects of a museum’s work, including training and forward planning.

There was a general consensus that mystery shopping should not take place in isolation and that follow up sessions would be helpful.

The scope of mystery shopping could be widened to incorporate other visitor groups.

Recruiting people from a non-museum background as shoppers might address the problems associated with facilitating mystery shopping in a small sector where many people are familiar with each other

Museums offered small grants to help implement improvements saw this as a real incentive.

Page 8 of 16

Evaluation Feedback

Shoppers

The Mystery Shopping process

Mystery shoppers were very positive about the process, describing it as well thought out, easy to understand, and comprehensive in its coverage of all aspects of family friendly visiting.

Many felt that they had gained practical insights into what actually works in other museums. Several reported that this had led them to initiate changes in their own venues. One commented that ‘It made me think about how visitors may view my venue and where our strengths and weaknesses might be. It also allowed me to think of interpretation and visitor expectations as an overall experience whichever venue you are. ‘

One shopper, with very little personal experience of children and families, commented that mystery shopping had given them a new perspective by making them think about family needs and expectations.

While most shoppers did not depart from the original plan, one shopper took advantage of a school visit to ask members of the group some questions about their experiences. Two shoppers were accompanied by an additional, first-time visitor, in the hope that this would provide a fresh perspective on venues that were already familiar to them.

One shopper did the ‘visit’ retrospectively from an earlier trip as the museum was closed during the mystery shopping period.

What worked best?

Many shoppers enjoyed the opportunity to take time out from the day job to visit and evaluate another museum. They commented positively on the structured approach of the visits, the focus on family friendly facilities, and the opportunities provided for shared learning.

For one shopper the questions on interpretation and presentation worked best, as they provided a framework for evaluating how well ideas about engagement work in practice. Another commented on the usefulness of being able to observe visitors ‘interacting with staff, activities or just the space’.

Several found it instructive to compare promotional materials and on-line information with the actual experience of visiting.

One of the shoppers who went with a companion felt that this was particularly beneficial, as they felt less conspicuous and were able to discuss the strengths and weaknesses of the venue onsite.

Page 9 of 16

Problems and weaknesses

Some shoppers felt uneasy using the same assessment criteria for both small independent museums with few resources and large organisations with significant provision for families. One shopper admitted that the score given to one museum was probably inflated by the fact that it had tried to make some provision for children despite very limited resources.

Another felt uncomfortable about being too harsh in expressing ‘constructive criticism’.

Some shoppers felt that a number of questions overlapped, which could lead to repetitive answers. One solution might be to incorporate the guidance notes within the questionnaire, making it easier for shoppers to focus their answers.

The time of year that the visits took place posed problems for some shoppers, with seasonal opening times and adverse weather conditions creating complications.

The difficulty of facilitating ‘true’ mystery shopping in a small sector where many people are familiar with each other was also mentioned.

Developing and broadening the process

Shoppers were asked how they thought the mystery shopping process might be developed or broadened.

Several felt that follow up sessions focusing on strategies for visitor retention would be helpful. One suggestion made was to set up cross-venue family focus groups to share ideas and develop resources. For example, venues close to each other might work together to create a family friendly project that links the sites through a shared trail, or a passport with stamps for children to collect.

One shopper thought that there should be further survey a year later to see what (if anything) had changed. Another favoured on-going surveys but acknowledged that this would be expensive.

It was also suggested that more museums might be involved, and that the scope of mystery shopping might be widened to incorporate other visitor groups as well as families. These might include younger couples, disabled visitors, school parties, and overseas visitors. At the same time, recruiting people from a non-museum background as mystery shoppers might give a better indication of what families really expect from a museum. This might also ‘reduce the chances of the game being given away’.

One shopper commented that ‘more time to mystery shop during our training day would have been useful’. Another felt that feedback from the venues could provide a helpful indication as to how the shoppers might improve their technique.

It was also suggested that the scope of the questionnaire might be extended to cover the ways in which venues handle difficult customers, sensitive issues and complaints.

Page 10 of 16

Challenging expectations

Shoppers were asked how the experience of mystery shopping challenged their family friendly views or expectations. Several commented that their visits to other venues had reminded them how simple things can make a big difference and how much can be done with limited resources. The experience had enhanced their awareness that museums can achieve significant improvement even within a context of funding constraints if they think creatively.

Several shoppers reported that the visits had broadened their perceptions of how family engagement is possible in a museum setting. The experience had also reinforced the importance of staff attitudes. If families do not receive a positive welcome from staff they are unlikely to return.

The importance of integrating a family friendly approach within the work of the museum as a whole was brought home to one shopper, who wrote, ‘family friendly issues are seen as separate in the museum world but this made me feel more strongly that they are not’.

One of the shoppers was concerned that there was an uneasy balance between making museums more family friendly and maintaining the underlying quality of experience. Another admitted that ‘the whole thing was a challenge’ as they found it very difficult to think of a way to include more family friendly elements in an adult-themed museum without changing its character.

A non-parent found the training quite challenging as it offered ‘a sense of what parents are looking for and how they think’.

Several people remarked that the experience had given them a fresh perspective. This could have been even more interesting, suggested one shopper, if they had taken a child along to test their assumptions about how children would react to the museum.

What was learned?

Shoppers were asked whether they had learnt anything new about what makes a museum family friendly as a result of their visits. Several mentioned the quality of information offered to visitors to enable them to make an informed decision about visiting. Others stressed the importance of a friendly welcome from staff, as this is the first impression families get of any venue. Even if the subject matter doesn’t have an obvious appeal to children, enthusiastic staff and things to touch will help them to find something of interest.

Above all, shoppers highlighted the need to address the whole experience and not just one aspect. If a site is intrinsically unfriendly to families, an add-on programme of workshops and events will not work. The museum must make sure that the whole site is family friendly, not simply one specific area. This means embedding a consideration of the family visitor in all that the museum does, above all in its forward planning.

Adopting ideas from the ‘shopped’ museums

One of the shoppers commented that ‘there is always something new or innovative to be learned from other venues’. Most identified elements of good practice that they intended to adopt in their own venues. Often, these were simple and inexpensive ideas, such as

Page 11 of 16

children’s games, more things to touch, or improvements to the baby changing area. Several people stated that they now intended to improve the information they offered to family groups, both printed and online.

The key idea, identified by many of the shoppers, was that of adopting a holistic approach to visitor provision, taking in signposting, linked pathways, landscaping, seating, litter bins, toilets, café menu, as well as the contents and interpretation of displays. This would be underpinned by better staff training and increased sharing of good practice.

Face-to-face follow up meetings

Shoppers had a follow up meeting with two of the venues. This enabled them to meet and talk to the venue’s staff in order to explain their findings, answer questions and develop ideas.

Venues

The Mystery Shopping process

The venues were generally positive about the programme, felt that it was well organised and that they were clearly informed about the process.

They gained significant reassurance from shoppers’ feedback that they are generally on track but were also appreciative of their suggestions for improvement.

The opportunity to gain a fresh perspective through an independent assessment and analysis of their visitor facilities was particularly welcomed. Several venues reported that they had shared this feedback with their teams as a whole, prompting open and constructive discussion.

One venue mentioned that they had fed information from the shopper’s report into their Forward Plan and would also be using it for advocacy purposes.

In general the exercise went much as most participants had anticipated although one venue expressed surprise that the mystery shoppers did not bring children with them or come at a weekend or during a school holiday.

One participant commented that ‘the only surprise was just how useful the project was. It highlighted parts of the visitor experience (both positive and negative) that staff had not previously considered being important’.

What worked best?

Page 12 of 16

Venues welcomed the fact that shoppers were well-informed and they appreciated their honesty. As one venue put it, ‘It was incredibly valuable to have someone who we felt was honest with us giving feedback’. While venues were pleased to receive positive feedback and share this with their staff, criticism was generally seen as constructive.

By paying attention to detail, shoppers were often able to identify small improvements that make a significant difference to the overall visitor experience. Venues also felt that the shoppers had a sympathetic understanding of the constraints and limitations faced by many sites.

In the Conurbation, small grants were made available for follow-up work. This was seen as a great strength by the two museums involved. As one commented, ‘The most positive impact for a small museum like ourselves is that there is funding associated with the project so that we can actually implement the suggested improvements’.

Problems and weaknesses

Generally, the venues felt that the mystery shopping process had been made clear from the outset and that it worked smoothly.

However, several thought that the reliance on a single individual to undertake the shop was a potential weakness as it only offered one view. There is also a danger that staff at a particular venue will recognise the mystery shopper.

The time of year that during which the visits took place was seen by some as problematic. One visit took place when there was a heavy fall of snow and few visitors. Others took place on days when there were no children or families visiting a particular site.

It was suggested that the range of marks within the scoring framework should be wider. One museum in particular felt that it should be amended to include ‘Not Applicable’ as this would be fairer to the smaller or voluntary museums that might not have certain facilities such as toilets or refreshments.

Developing and broadening the process

The views expressed by venues are very similar to those of the shoppers.

Suggestions included follow up visits, a further survey after twelve months, more in-depth testing of customer care, and widening the scope of mystery shopping to incorporate other visitor groups and age ranges.

As one respondent put it, ‘It would be great to get an even broader range of opinions. For a family friendly theme an actual family (or group of families) visiting would be excellent. Different age ranges would undoubtedly recognise different factors as being important. If that sort of honest feedback was available it would be great because I simply do not think we get that from the paper based feedback we currently employ’.

Differing perspectives

Venues were asked how the shopper’s analysis of the venue compared with their own.

Page 13 of 16

Most felt that the shoppers’ views were broadly similar to their own. One venue had just completed a large audience research programme and the shopper’s comments complemented the report’s findings. Several venues admitted that their own awareness of their site’s shortcomings tended to be more broad brush than the more detailed analysis provided by the shoppers. As one observed, it was also useful to have a written statement from an independent assessor to present to the staff and to board members and staff. The views of service users have a greater impact than self-assessment.

In several cases, by looking at the site from a very different perspective, shoppers were able to highlight issues that had not previously been considered by a venue.

In one instance a venue felt that the shopper ‘appeared to come with pre-conceived ideas’ which did not match the reality of the visit. It seemed ‘almost as if he had pre-written the report’. Despite this, the venue has reacted positively to criticisms within the report by implementing improvements.

What was learnt?

Venues were asked how the shoppers’ comments added to their own perceptions of the site.

It was general agreed that the mystery shopping process had given the venues a fresh perspective. As one commented, ‘it is always useful to have a fresh pair of eyes to see things that we may not notice’.

Above all, the exercise helped the venues to build a better picture of what is important to visitors and what they need to do to become more visitor focused. Several mentioned that they had been persuaded of the need to adopt a more holistic approach. One commented that the shopper’s report had ‘made us look at the whole site and the experience of the visitor from arrival right through to departure with a fresh pair of eyes’. Equally important in rethinking their approach to promotion was an understanding that impressions are formed before visitors have actually arrived at a museum.

One entirely volunteer run site was encouraged to discover that, seen from the perspective of a museums professional, it was offering a high standard of service to its visitors.

Implementing the shoppers’ recommendations

Most of the venues were implementing recommendations made by the shoppers. Specific improvements mentioned included the installation of baby changing facilities, improved internal and external signage, enhancements to exhibition areas, a family trail, the redesign and restocking of a shop and improvements to marketing and website activity.

Face to face follow up meetings with the shopper

The two museums that had face to face contact with their shoppers felt that this was particularly useful as it enabled them to discuss points raised in the report in greater depth. Indeed, one venue felt that this session was ‘the most helpful part of the process’.

Page 14 of 16

Page 15 of 16

Page 16 of 16