Full BEAM Ahead

First published in the IEA Insider annual newsletter on 3 October 2022.

The Botswana Educational Achievement Monitor project (BEAM) continues to make excellent progress with the completion of the Field Trial in November 2021 followed by the analysis of the data and selection of items and questions for the Main Study. This three–way collaboration has resulted in a high-quality new assessment measure.

Developing a new assessment measure from scratch is no mean feat. Developing it against the backdrop of COVID-19 restrictions and revised schedules only serves to make the task that bit harder. However, the International Association for the Evaluation of Educational Achievement (IEA) and its partner, the National Foundation for Educational Research (NFER), working in close collaboration with the Botswana Examinations Council (BEC), have developed a new set of assessment instruments and questionnaires for Standard 4, aged 9-10. These performed well in the Field Trial and have enabled a strong set of items to be selected for the main data collection in November 2022.

Building on the solid foundations laid in 2020 with the framework development and item and questionnaire development workshops, followed by a concentrated phase of item development and piloting from October 2020 to June 2021, the team then started preparations for the Field Trial. September 2021 was a very busy month for the BEAM project. IEA successfully conducted workshops preparing test administrators and school coordinators, and two workshop parts on data management and data processing. Meanwhile, the NFER teams reviewed the items in light of the pilot scoring data, made amendments and selected the texts and items for the English tests and the items for the mathematics instruments. They also delivered scorer training to the BEC subject teams and senior scorers. In October NFER handed over final versions of all the instruments and the scoring guides which included examples of student responses from the pilot study, as well as the contextual questionnaires.

The Field Trial was successfully administered in 47 primary schools in November and December 2021. BEC recruited a randomly selected sample taking account of geographical location, school type and prior attainment. Many BEC project team members participated in the Field Trial, travelling to schools in remote regions of Botswana, in order to get first-hand experience of pupils’ engagement with the tests and the questionnaires.

Following the test administration, BEC trained the scorers and oversaw the scoring of the open and closed responses. Data processing and item analysis workshops were then conducted by IEA in April 2022. Following this, IEA analysed the items and provided the item analysis results. Reassuringly, the items had largely performed well and in line with expectations. Moving into the item selection phase, IEA convened three workshops in June 2022. With content support from NFER and psychometric input from IEA, BEC were able to select a broad and balanced range of texts (in the case of English) and items (English and mathematics) which also aligned with the rotational design for the BEAM assessment. A similar exercise was conducted with the selection of the final questionnaires. The booklets are now being finalised and will be ready for the Main Study.

When BEC, IEA and NFER embarked on this important capacity-building project in early 2020, we envisaged several in-person workshops as well as IEA and NFER’s involvement in the pilot study and the Field Trial. As a result of the pandemic all three partners have been creative and flexible in overcoming the travel restrictions and have optimised the online workshops, often lasting several days. Where appropriate additional workshops were held to enable in-depth discussion of specific issues such as reasoning items. Regular communication at all levels – e.g., management, subject / questionnaire / sampling / analysis – has been vital in ensuring productive working relationships and keeping the project on track; it’s also been important to have open and constructive dialogue as part of the long-term training function.

With this 3-way commitment, built up over 2½ years, the BEC is well positioned to run the Main Study, and we hope to continue working in partnership in the future.