How do you measure experience and impact in the culture sector?

Experience has long been at the centre of cultural and leisure offerings. In the 1990s, it was firmly put on the agenda by Pine and Gilmore. Recently, there has been increasing talk of the transition from the "Experience Economy” to a "Transformation Economy”. That’s quite significant, from experience to impact, because as we will argue later, transformation is seen in the literature as the strongest form of impact. The concept of impact is high on the research agenda in the cultural sector. In this article, we describe some theoretical principles and explain how to measure experience and impact. We do this on the basis of a study we are currently conducting for the Netherlands Open Air Museum in Arnhem.

Pieter de Rooij is a lecturer Experience Marketing and researcher of consumer behaviour at the Academy for Leisure & Events.
Adriaan van Liempt is a lecturer of Sociology, Statistics and Data Science and researcher of consumer behaviour at the Academy for Leisure & Events.

Why impact?

Cultural organisations often have an important role within the playing field of social relevance. They are well aware that this relevance goes beyond numbers of visitors, numbers of performances, or good satisfaction ratings. Nowadays, it is also mainly about the effect of cultural visits on individual visitors and thus on society. In the past, organisations carried out many satisfaction surveys, and more recently experience studies. Our idea is that in the near future it will increasingly be about impact research. It is all about "to prove” and "to improve” as stated by Vermeulen and Maas (2021). Impact research not only provides opportunities to demonstrate impact to grant providers and sponsors, but also to improve strategy and make effective investments to increase impact.

What is meant by experience and impact?

We define experience as an immediate, personal, subjective outcome, as a result of an interaction between a visitor and environmental stimuli designed by an organisation and other visitors which is affective, cognitive, and sensory in nature. We emphasise "immediate”, as opposed to impact, which involves a "growing” and "delayed” effect. 

Three levels of experience impact: memorable, meaningful and transformational.

Some people focus on the instrumental value of impact, such as economic or social significance. Here, we focus on individual impact. So what is the impact on or what is the effect for the individual? Duerden et al. (2018) distinguishes three levels of "experiential impact” that we also adhere to. The first level of impact is memorable. The visitor's attention has been caught and held and (strong) emotions are involved. The second impact level is meaningful. The experience is not only memorable, but also leads to significant and personally relevant, new insights. The third and highest impact level is transformational. The experience is not only memorable or meaningful, but also leads to a personal change in values, beliefs, intentions or behaviour.

So the big difference between experience and impact is that impact is about what the experience actually triggers. Impact, in other words, is the effect of the experience. Visitors can characterise their experience of a cultural visit immediately afterwards. But creating impact requires (some) reflection and thus has a delayed character.

How do you measure experience and impact?

Experience and impact are difficult to measure. They are both personal and subjective. Moreover, all kinds of environmental factors play a role, making it difficult to generalise outcomes. Consider, for example, the context of the provider (mission, image, physical environment) and the composition of the audience. It is therefore important for any experience and impact research to properly describe the situation and circumstances in which the service is delivered. It is particularly important to properly describe the purpose of the organisation (the "intended experience” and the "intended impact”). After all, this will enable you to evaluate the experience and impact in a targeted way and make improvements.

Opinions are divided on whether you should use qualitative or quantitative visitor research. Those in favour of a qualitative approach (using interviews, for example) say you should give the visitor a real voice. This way, you embrace the subjective nature. The focus is then on understanding experience and impact. Those in favour of a quantitative approach point out that there is a strong need to make experience and impact measurable. Moreover, you can better explain experience and impact because of the larger numbers. In our study for the Arnhem Open Air Museum, we measured experience and impact in a quantitative way because we wanted to explore the relationships between service ratings, experience, impact, satisfaction and the NPS (Net Promoter Score).

Experience and impact are both personal and subjective and therefore more difficult to measure.

Experience dimensions in museum research

In our research for the Open Air Museum, we used experience dimensions previously described by researchers in the museum sector (Packer et al., 2018). In the first phase of our research, we made a selection from the 17 dimensions identified. We used a short survey to ask visitors how important they thought each experience dimension was (n = 120). Then, taking into account the mission of the Open Air Museum, we had a discussion with marketing manager Carola van der Woude about which dimensions were relevant to select: what would fit the "intended experience”? In the end, we included eight experience dimensions in the follow-up study. We measured each experience dimension by means of four questions (taken from Packer et al., 2018).

Below is a list of all the experience dimensions and, in brackets, two examples of the questions. 

• Engagement (I felt very engaged, I was very interested)
• Appreciation of beauty (What I saw was attractive to me, I enjoyed the many beautiful things I saw)
• Relaxation (I felt relaxed, I was able to unwind well)
• Reflection (It made me think, a mirror was held up to me)
• Sociability (I felt involved with others, I had a sense of togetherness)
• Marvel (I had a sense of wonder, I loved it)
• Joy (I enjoyed it, I was enthusiastic)
• Choice / Autonomy (I felt free to do the things I wanted to do, I was able to make my own choices).

Impact measurement

We measured the "experience impact” by means of the three-way classification already described by Duerden et al. (2018) (examples of questions are in brackets):

• Memorable (I was touched emotionally, the visit will stay with me for a long time)
• Meaningful (The visit led to new insights about myself, others, or the world around me)
• Transformational (It led to a change in my attitude, belief, identity or behaviour; I felt inspired to do something new or different in my life).

We conducted the impact measurement at two points in time: immediately after the visit to the museum and two months after the visit. As we are currently collecting the data, we cannot yet describe any results.

Impact research provides opportunities to demonstrate effects to grant providers and sponsors, improve strategy and make effective investments.

In conclusion
In this article, we have described the theoretical principles underlying the choices we have made in experience and impact research in the cultural sector. We hope that the results we will soon provide can contribute to the further development of this kind of research, and that these results will support the Open Air Museum in the choices they make in the field of experience and impact design.

Sources

• Duerden, M. D., Lundberg, N. R., Ward, P., Taniguchi, S. T., Hill, B., Widmer, M. A., & Zabriskie, R. (2018). From ordinary to extraordinary: A framework of experience types. Journal of Leisure Research, 49(3-5), 196-216. https://doi.org/10.1080/00222216.2018.1528779
• Packer, J., Ballantyne, R., & Bond, N. (2018). Developing an Instrument to Capture Multifaceted Visitor Experiences: The DoVE Adjective Checklist. Visitor Studies, 21(2), 211-231. https://doi.org/10.1080/10645578.2018.1553925
• Vermeulen, M., & Maas, K. (2021). Building Legitimacy and Learning Lessons: A Framework for Cultural Organizations to Manage and Measure the Social Impact of Their Activities. The Journal of Arts Management, Law, and Society, 51(2), 97-112. https://doi.org/10.1080/10632921.2020.1851839