The team at RM Results are proud to have been a part of the Programme for International Student Assessment (PISA) 2015. We spoke to Project Manager Sam Smith and Educational Consultant Andy Bruce to find out more about RM Results’ involvement in the delivery of this study.
What is PISA and what was RM Results’ involvement?
Sam: PISA is led by the Organisation for Economic Co-Operation and Development (OECD) and provides an insight into the abilities and school experiences of 15-year-olds in different countries. The research aims to truly understand the strengths and weaknesses of various education systems around the world, and provide robust comparative results to participating countries such that they can pitch their aspirations at a level that is both world-class, and achievable.
During the PISA 2015 cycle, the RM Results team held the role of the National Centre and managed the Field Trial exercise (2014), the recruitment of schools for the Main Study (2015), the test administration of the Main Study (2015) and the collation of responses and production of both the National and School reports for England, Northern Ireland and Wales (2016).
Why RM Results?
Sam: I think RM Results was a natural fit for the task on a number of levels. Being a technology company in the educational assessment space, we have an understanding of schools from an educational perspective, and an understanding of what PISA is trying to achieve. This was also the first time that the PISA tests were computer-based, making RM Results an ideal partner. The majority of our Educational Consultants have previously worked as teachers or head teachers, so they know the benefits and challenges that participating in PISA presents to schools, and the barriers that teachers and senior leaders face when taking part in extracurricular programmes.
What were the first steps of the project?
Sam: The first steps of the project were to identify the objectives of the study and the key success criteria. These steps were achieved through meetings with the Department for Education (DfE) along with the Welsh and Northern Ireland Governments.
Andy: In 2014, RM Results carried out PISA Field Trials, testing pupils from between 30 and 40 schools throughout England, Wales and Northern Ireland. Following the Field Trials and lessons learnt from this activity, RM Results were responsible for recruiting schools to participate in the Main Study. As an Educational Consultant for RM Results, it was part of my job to encourage schools to take part in PISA. We began the process with a target list of 500 schools that we aimed to recruit, this list was provided to us by OECD and each school had a specific first and second replacement school should the sample school decline to take part. We commenced recruitment by sending out invitations to the sample schools. Once we’d reached out to the schools, we arranged an initial follow up and further discussion into the programme and what it involves. If, following discussions, the school declined to participate we accepted their concerns and then commenced contacting replacement schools.
How did the assessment work?
Andy: The 2015 PISA assessment was the first to be conducted via computer rather than a paper based test. Ahead of the tests, participating schools were asked to upload assessment data for their school and candidates onto a purpose built PISA website, created by RM Results. In order to ensure that schools had the necessary specification on their computers to run the PISA test, a system diagnostic test was sent to all participating schools and the findings from these tests provided to RM Results. If schools encountered any issues at this time or had any questions or concerns they were able to contact the National Centre, where one of the Educational Consultants would assist in overcoming these technical difficulties prior to test day.
The PISA 2015 Main Study was loaded onto individual USB sticks and distributed to schools within agreed timeframes ahead of the assessment day. The electronic tests are designed to try and identify what works well to develop students’ science, maths, reading and other transferable skills, and what doesn’t. Instead of knowledge-based questions, PISA is based around real-life problem solving. The data were then uploaded onto the system, where processed by data analysts at RM Results and coded by World Class Arena. Coded data were then sent on to OECD alongside the same information for just over 70 countries and jurisdictions worldwide.
What were the challenges?
Sam: The initial challenge was to bring schools on board in order that they partake in the 2015 PISA Main Study. Following the recruitment of the schools, the main hurdle was the logistical challenge of managing hundreds of schools situated across England, Northern Ireland and Wales. We had around 40 Educational Consultants, like Andy, who not only assisted in preparing schools beforehand but who also acted as test administrators at the schools, facilitating on test day. Each Test Administrator was allocated around 10 schools, which meant that they were often travelling on back-to-back days to attend testing at different schools in a short space of time.
Andy: Another logistical challenge was the distribution of the USB sticks. We needed to make sure the USBs were delivered ahead of test day, but not so far in advance that schools’ had access to the system ahead of their allotted test day. We also had to ensure USBs containing pupil data were moved around the country safely and securely.
What is the outcome of these tests?
Sam: PISA provides the individual schools that take part in the programme with a bespoke report on their school’s performance. The data from schools across the UK and around the world provides invaluable insight into how different education systems compare in terms of achievement and abilities in core areas of assessment at a point in time.
The team at RM Results very much enjoyed being the National Centre for the 2015 PISA Study and would like to thank the Department for Education, World Class Arena and UCL Institute of Education for their help in the delivery of the resulting reports.
Explore the 2015 PISA report
How can I find out more?
Our partners at the UCL Institute of Education are hosting a series of free conferences for school leaders, subject specialists and other stakeholders who would like to hear about the findings from PISA and to discuss the implications for education in England. The conferences will also focus on the findings from another international research study, the Trends in Maths and Science Study (TIMSS).
The agenda for each event will cover:
- Key findings from PISA and TIMSS.
- Example test items from PISA and TIMSS.
- What are the implications of PISA and TIMSS for maths and science curriculum and pedagogy?
- Time for Q&A and discussion.
Conference dates and venues:
- Manchester: Monday 27th February, 3pm-5.30pm
Crowne Plaza Hotel, 70 Shudehill, Manchester M4 4AF
- Bristol: Monday March 13th, 3pm-5.30pm
City Centre Novotel, Victoria Street, BS1 6HY
- Nottingham: Monday March 20th, 3pm-5.30pm
Park Plaza Hotel, 41 Maid Marian Way, NG1 6GD
- York: Tuesday 21st March, 3pm-5.30pm
National STEM Learning Centre, University of York, YO10 5DD
More information and registration details are on the UCL-Institute of Education website; places will be allocated on a first come, first served basis.
If you have any queries about the events please contact Margaret Turner, Membership & Events Team at the London Centre for Leadership in Learning.
In Northern Ireland a PISA seminar will be held in the morning and the afternoon of 7th March at Stranmillis University College, Stranmillis Road, Belfast, Co Antrim, Northern Ireland, BT9 5DY.
If you would like further details of this event or how to book on to attend please contact Deborah Frost.Deborah.Frost@education-ni.gov.uk.