Evaluation

Professor Jean McNiff

Research Consultant, ccConsultancy Associate

The experience of working on the OTLA (English) programme has been exhilarating, as has the subsequent process of writing the evaluation report.
The evaluation process itself has meant being caught up in working and talking with as many people as possible involved in the OTLA programme, including members of the ETF, teachers, managers, support workers, librarians and others, and the ccConsultancy team, all working collaboratively, with enthusiasm and enjoyment and with a compassionate understanding of the other’s needs.

It has been uplifting to work with, talk with and listen to people developing ideas together about the programme, negotiating meanings and coming to conclusions: this has included members of the ETF as well as practising teachers and the ccConsultancy team itself.

A striking feature of the programme is the way in which all have developed innovative ways of working together to improve the quality of teaching to help learners learn more effectively and enjoyably.
It has been especially enjoyable to listen to teachers’ shared presentations of good pedagogical practices, grounded in their commitment to developing a strong research base for their enquiries, and reading their reports in this and other publications. The reports themselves are inspiring: they contain accounts of energetic, highly committed people developing collaborative practices, with close support from the ccConsultancy team, mentors, team leaders and other colleagues, all with the intention of improving the quality of their own and their learners’ education.

A key value-added feature has been the opportunity of working with the ccConsultancy team, to explore ideas together regarding the development of a strong research base in which to share and test the validity of ideas, and in turn to pass these on to teachers, who will, it is hoped, pass them on to learners. This ‘passing it on’ process forms the basis of the OTLA programme; it could, indeed, be seen as the process of all current and future educational development, with implications for the improved wellbeing of the planet and its rich multitude of participants.

Purpose of Evaluation

The specific purpose of the evaluation of the OTLA programme was to give an objective assessment of its value for all involved in its planning, funding, delivery and experience. This process may also be understood as judging the extent to which the aims of those involved have been achieved in practice: that is, whether people’s involvement in the programme met their expectations of what they wished to get out of it, and the extent to which the provision and experience of the OTLA programme met their wider needs of practice and professional learning. The programme has been overwhelmingly successful in this regard: no negative experiences have been reported and all participants appear to be fully satisfied with the overall experience.

The Evaluation Process

The evaluation process has throughout been collaborative. It has involved accessing relevant documents, including government reports and scholarly literatures; gathering amounts of data, including data from conversations with colleagues, surveys and focus groups; checking which of those data may stand as evidence, and as evidence of what; coming to considered conclusions; and testing the truthfulness and relevance of those conclusions against the feedback of appropriately-informed others.
This process has reinforced the understanding that evaluation should be seen as a rigorous form of research, where aims are considered and practices judged in terms of the extent to which those aims have been achieved. It has also been conducted in a rigorously ethical way, meeting the high standards of agencies such as the British Educational Research Association and the National Evaluation Society.
The evaluation report may stand as a bone fide account of rigorously-conducted practice-based research. It has followed a standard action research methodology of:

  • Identify an area of interest or concern;
  • Explain why the issue is important and relevant to the field;
  • Produce data to show what is happening in the field;
  • Imagine ways of improving the situation;
  • Try those ways in action;
  • Monitor practices to see whether the action is having any effect;
  • Take stock and consider whether the situation is better in light of the action;
  • If yes, continue with the action and new practices; if no, then re-think and start again;
  • Continue this process until the situation is satisfactory (while remembering that new ideas evolve through working with other people, so it is unlikely that an entirely satisfactory situation would ever be reached).

The evaluation process has involved two sets of enquiry: an inner set and an outer set, each taking the form of an action enquiry. The inner set constitutes an account of the teachers’ own enquiries, while the outer set constitutes an account of the evaluation process.
In terms of a standard action research approach these two enquiries appear as follows:

Inner set: Teacher’s Enquiries

Outer Framework: The Evaluation Process

Findings

The main findings of the evaluation were as follows:

Finding 1 Teachers can produce high quality research reports when appropriate support is provided, especially in terms of expert knowledge of:

  • the nature of professional education practices
  • research approaches relevant to enhancing collaborative and person-centred forms of professional education
  • the production of high-quality practice-embedded research reports;
  • appropriate attitudes on the part of participants in terms of readiness to learn and openness to innovative forms; of providers, in arranging the most beneficial conditions for learning and in delivering appropriate forms of support; of managers, in providing opportunities for the delivery of research-based professional development programmes; of policy makers in openness to the development of new participative forms of supporting professional learning.

Finding 2 Practitioners can learn best and benefit from professional development provision most when the provision is presented in terms of their lifeworld experience. This includes an appropriate form of relevant personal and professional content presented in a practitioner-friendly and practitioner-relevant form of language.

Finding 3 Practitioners’ learning is best supported from an understanding that research may be carried out and its findings may be put into immediate effect.

Finding 4 Sufficient amounts of time should be allocated to professional education programmes that involve ongoing learning and reflection: a concern was voiced consistently by teachers that more time would have been helpful in developing learning and skills acquired from participation on the OTLA programme.

Finding 5 The best ideas and suggestions for research programmes usually come from practitioners themselves, when they see the relevance of the research topic for their own practices, with possible application for colleagues and for the profession.

Finding 6 Learning from previous life episodes may be seen as informing new practices, as shown in the incremental learning by both the team and participants across the years 2018–2020, and as communicated through teachers’ reports and through the nature of the technology-rich content of professional development days; this incremental form of learning often takes the form of and is embedded in a rolling-out of ideas and insights.

Finding 7 Professional development should include the provision of new ideas and new technologies.

Finding 8 Sustainable professional development often best takes the form of the development of a practitioner’s own knowledge, not simply the acquisition and application of others’ knowledge.

Finding 9 Professional education programmes should be grounded in a commitment to dialogue, participation and negotiation. In the OTLA programme it was a case of the team negotiating content and form with teachers, and of teachers negotiating content and form with learners. This was altogether a demonstration of the values of participation, inclusion and democratic discourses as communicated in the aims and values of both the ETF and ccConsultancy.

Recommendations

The recommendations of the evaluation were:

  • For the provision of similar programmes, more extended time is needed for engagement with and the application of learnings acquired from participation;
  • The current OTLA programme for teachers should be continued, with built in progression, to show the potential uses and benefits of the professional
    learning acquired from the programme; the issue now is to see whether the abundant evidence that shows the benefits of participation in terms of participants’ professional learning may realistically be brought to real-life situations; this implies –
  1. An extension of the evaluation process in terms of investigating the development of learners’ learning in relation to teachers’ professional learning: specifically, to see whether the quality of learners’ learning has been improved through teachers’ participation on the current OTLA programme;
  2. Production of further evaluation reports to show the ongoing effects of teachers’ professional learning in relation to the quality of learners’ learning; and in relation to issues of its sustainability and ongoing application to everyday life. This would involve accessing learners, together with their teachers, possibly in their work and study places;
  3. Production of appropriate texts that show these processes in action, produced in both practice-friendly forms of language for appeal to a practitioner audience, to be made available to the wider public, appropriately dated for ease of reference in the literatures, and also in scholarly forms for publication as in refereed journals. The production of such texts would show that practitioners can work with and communicate in multiple forms of discourse, and as full participants in what Geras (1995) calls ‘the conversation of humankind’.

End Word

In summary, may I say that it has been a joy and a privilege to work with outstanding teachers and colleagues in this most worthwhile endeavour; I consider myself fortunate indeed to have had the opportunity to enjoy such excellent companionship.