Report spotlight: Question-level analysis

Due to an extended period of remote learning, certain curriculum areas may have had less coverage or may be problematic for particular pupils. Identifying specific areas of the curriculum that individuals or groups of pupils are finding more challenging can inform teaching and help pupils progress.

NFER Tests users gain free access to an online analysis tool, offering simple, yet powerful data analysis and reporting. The question-level analysis report is one of four reports available and is popular with schools looking to gain additional insight into pupil performance.

NFER's question-level analysis report enables you to:

  • gain a finer level of understanding of performance relating to specific questions and areas of the curriculum, at whole class and individual level
  • identify how pupils' skills compare in different curriculum areas against the large-scale nationally representative sample
  • inform whole class, small group and individual interventions
  • target support with resource development, curriculum planning, or staff CPD.

 

Using the question-level analysis report

The image below shows part of an example question-level analysis report generated by the tool for one of the tests in the suite of NFER mathematics assessments.

By considering question 3 (1) which is focused on mathematics statement (as shown in the POS / Focus row), assessing whether pupils can find 1000 more or less than a given number, we can look at the different ways this report might be used.

How to use whole group data 

Question 3 is a 2-mark question for which the average score in this group was 1.24 marks (2). This can immediately be compared to the score of 1.48 (3) achieved by the sample. Gaining this understanding that the group have scored below the nationally representative standardisation sample, combined with  the question-by-question information, will help inform teachers and enable they to respond in the appropriate way. Teachers can adapt their teaching, depending on whether the curriculum area of focus has been covered, or is yet to be covered:

If the topic has not yet been covered If the topic has been covered previously

The teacher could refer to a selection of pupils’ responses to the question and use their findings diagnostically to plan their approach to teaching the topic when they reach that part of their school’s curriculum. Potential and pre-existing misconceptions may be identified in the pupils’ responses and ways of addressing these pre-emptively may then be incorporated into the plans for teaching the topic for the first time. 

The teacher could dedicate some time to revisiting the topic or include the relevant skills and knowledge in their planning for other curriculum areas that are yet to be covered. There may also be value in reviewing the scheme of work from the point at which the topic was first introduced to identify opportunities to incorporate use of the relevant knowledge and skills in other areas. This will help pupils to maintain their skills and make connections between different content .

Since reports for other groups’ and previous years’ test results are available for analysis, additional reports can be generated and the question-level data compared. If the performance in this topic was below the standardisation average in the other groups or previous years, the school may consider research into resources and training that focus on the specific curriculum area. If there are obvious differences in the pupils’ performance in different groups within the school year, or between different school years, the school might consider internal training where more experienced colleagues can share resources and approaches that have worked well for the topic.

How to use omission data

The omission rate for each question tells the teacher the proportion of pupils who made no attempt to answer that question. The omission rate for question 3 is 12% (4), which is quite high for a question that appears early on the test. Higher omission rates are expected for questions towards the end of a test because the longer and more challenging questions appear later. Omissions may be due to pupils being fatigued, and some pupils may not reach the end of the test in the allocated time. A high omission rate observed for an early question, such as this, could prompt the teacher to investigate the reason that pupils chose not to attempt it.

In addition to considering the position of the topic in the school’s curriculum, looking at the question as it appears on the test might give some insight into why pupils omitted a question. For example, the teacher might check whether the vocabulary used and presentation of the question are familiar to their pupils and, depending on their observations, may modify resources to ensure that pupils regularly meet expected vocabulary and can apply their understanding of different topics in a variety of presentations and contexts.

How to use small group data

The online analysis tool also allows the group to be filtered by characteristics such as gender, meaning focused reports can be generated to look at the performance of particular groups of pupils. Support and interventions can be based around the needs of the pupils identified.

How to use individual pupil data

Below the whole class data for a question, each pupil’s mark for that question is listed. This allows teachers to identify areas of strength and specific questions or whole topic areas that cause difficulty for individual pupils. By referring back to a pupil’s responses on the test and understanding the approach a pupil used and where they encountered difficulty, teachers can tailor support appropriately. Children’s access to education and support has been significantly disrupted as a result of the COVID-19 pandemic. In a single school setting, pupils’ experiences may vary widely and ease of access to each pupil’s question level data provides teachers and schools with an invaluable means of monitoring who in their classes is secure and who requires support in different aspects of the subject. Intervention can be planned to address specific skills and fluency, single topics or whole curriculum areas to improve pupils’ confidence and ensure further progress.

Order NFER Tests to gain access to the online analysis tool.