So ... Did It Work?!

This is a period of transition for us here at Akoranga o Naenae, as we finish our evaluations of Round 1 and begin anew with 48 eager students for Round 2.

As we said at the beginning, Akoranga o Naenae is based on a hunch, with the main principles well researched.  Now we wanted to determine whether or not AoN has proven itself effective.  Our chief supporters, our network leaders, are realistic and experienced people.  They know that change takes time, and have not put unfair expectations on us.  However, we always knew that we would have to prove ourselves pretty quickly to ensure that we could continue AoN.  As such, the evaluations from Round 1 have been much anticipated (particularly from data geeks like me!) Keep in mind we had (only) 14 days with our learners. 

As with most things at Akoranga o Naenae, when considering methods for collecting evaluations, we tried to think outside the box.  As there was not one tool that was going to give us all the information we needed, we used a variety of tools, and we asked everyone who had been involved for their perspective.  These included self-assessment, questionnaires, video recordings, and reflective discussions.

We started with the most important players - the students.  Way back at the

Start of Round 1

beginning of AoN, we had asked the students to do a self-assessment, indicating their knowledge of the active learner traits, and their understanding of when they show these traits themselves.  We then asked them to complete the same self-assessment at the end of AoN.  By turning their answers into numerical ratings, based on a pre-determined criteria, we could compare the beginning and end data.  This showed that:

• the students were more able to articulate what active learning was after working with us at AoN for 14 weeks (total score at the end of 81 points, compared with 49 points at the beginning) 

= 65% improvement

• the students showed an improvement in their ability to describe when they are active learners in real life situations (79 points at the end; 47 at the beginning)

End of Round 1

= 68% improvement

• the students were able to explain why active learning is important and relevant to their lives (especially now, not just in the future).  This was the area that we knew the students found the most difficult, and we had worked hard on, so we were pleased to see a total of 51 points at the end of AoN, compared with 33 points at the beginning

= 54% improvement

These three key points showed us that the learning had been effective for many of our 48 learners.  Bearing in mind that many, if not most, of these kids are priority learners, whatever impact we had (positive or negative) was always going to be important.  We were pretty excited by this results, and proud of what we all had achieved over 14 days.  It gives us more confidence to hope (as we always have) that AoN might end up being one of the things that leads to improved academic success for some of these students.  We have anecdotal evidence from several kids, families, and leaders to say that they have seen just that.  Kids telling us about how they used not to be at the National Standard, and now they are; parents describing their child as 'sky rocketing' at school; and leaders collating data on their eight AoN students, showing significant improvements in reading, writing, and maths.

We used Infinity Learning Maps as an assessment tool throughout our time at AoN, and these provided us with very interesting data, too:

Learning maps #1, #2 and #3

• as students became more active in their learning, the number of connections with people outside of their normal 'bubble' became much larger.  This then decreased as the students became more purposeful/specific in their questioning
• students came to better understand how technology can help them connect and learn
• the students became more aware that learning happens everywhere, not just in a classroom

Students' survey

Our third method of collecting student evaluations was through a survey.  The main themes were:

• the students loved getting to know others from the network and working with them
• they appreciated what the community space could offer
• they really enjoyed the food we had on our Learning Maps days!!

We also asked our schools' leaders for their evaluations of AoN, with a questionnaire targeting specific areas of interest to us, as well as giving them room to add other thoughts.  This is a small sample size, with only six schools in our network, but their comments were thoughtful, considered, and helpful to us.  These included:

• a concern that the 'off-site' nature of AoN appears to imply that schools are not able to cater for their priority learners (which we know is not the case, but those making such an inference may not realise that)
• the difficulties with transferring the active learning ideas from AoN into school classrooms, despite a variety of efforts
• the workload on the teachers at AoN, particularly at the beginning and in the set-up phase
• the effectiveness of connections with whanau
• the increased "student agency" that was seen in some AoN students back at their schools (for example, at ine school some students starting a coding club. At a different school, students coaching younger sports teams)
• the benefit of having another 'shared purpose' amongst the network schools

Some the Leaders Questions of

When asking whanau for their evaluations, we mostly used video recordings to capture their thoughts about AoN, and its impact for their child.  These resulted in some very heart-warming moments, and some honest truths.  The highlights were:

• hearing that students had developed their own 'drive' to learn
• confirming the fact that the students were motivated by the power of choice
• believing that AoN had contributed to an improvement in academic success at school

Some of the teacher questions

The next group we wanted to get evaluations from was the teachers of these students back in their own schools.  We have been really interested in how the skills they had been developing at AoN could transfer back to other areas of the students' lives.  We asked teachers to fill in a quick online survey.  The teachers who did so gave us some useful feedback, including:

that some students clearly understood the link between their learning at AoN and school, while others did not

• the enjoyment that most students got by learning about their passion
• that we have not made clear enough to some teachers the basis for AoN 

Lastly, Natasha, Kelly, and I did our own evaluation.  We knew there were a lot of positives that would come from this, as we had been there each day, seeing the change in the students.  These positives were tempered with some points for future improvement.  We are active learners who reflect!  The main themes were:

AoN teacher evaluation

• the difficulty in striking the right balance between too much and not enough communication with teachers and leaders
• the tensions caused by some people in the network being more engaged than others
• the difficulty with wifi!
• the high workload
• the incredible professional development that AoN was for each of us
• the benefits of AoN being held in a community space
• the significant observable changes in the learners from Day 1 to Day 14

So, overall, did AoN 'work'?  Did the 14 days we spent together make a difference to these students?  Did it help them to build skills and understanding about how they can enhance the way they learn in all areas of their lives?  It sure did!  In the words of one of our students:  "Akoranga o Naenae changed my life".

As active learners ourselves, we're not stopping there!

We have been cogitating over all of this evaluation data, considering our next steps, and contemplating about how what we've learnt so far can help us in Round 2 and beyond.  The main points we will be working on are:

• the possibility of co-constructing success criteria for our Active Learning trait assessments with students,
• the purpose of each planning, assessment, and evaluation tool, and how to best use these,
• how we can streamline the whole time-frame (and we have to, as Round 2 is only 12 weeks!), and ...
• continuing to try to find ways to make transference between AoN and schools understandable and practicable for leaders, teachers, and students

How do you feel about AoN finishing?

This is just a snippet of what we wanted to say.  If you have any questions or want further information, please contact us.  We'd be really excited to talk about these evaluations in more depth.