About the NAEP Science Assessment
The National Assessment of Educational Progress (NAEP) is a congressionally mandated project administered by the National Center for Education Statistics (NCES) within the U.S. Department of Education and is the largest continuing and nationally representative assessment of what our nation's students know and can do in select subjects. NCES first administered NAEP in 1969 to measure student achievement nationally. The NAEP science assessment measures students’ knowledge of three broad content areas—Physical Science, Life Science, and Earth and Space Sciences—and four science practices—Identifying Science Principles, Using Science Principles, Using Scientific Inquiry, and Using Technological Design. These four practices describe how students use their science knowledge by measuring what they are able to do with the science content. Results for grades 4, 8, and 12 are reported for the nation.
In 2019, the NAEP science assessments at grades 4, 8, and 12 transitioned from being paper-based assessments (PBA) to digitally based assessments (DBA). A multi-step process was used for the transition from PBA to DBA, which involved administering the assessments in both formats. Students were randomly assigned to take either the digitally based or paper-based assessment in 2019. The assessment results in this report are based on the combined performance of students who took the paper-based and digitally based assessments. The transition was designed and implemented with careful intent to preserve trend lines that show student performance over time. Thus, the results from the 2019 science assessments can be compared to results from previous years. Read more about the NAEP Digitally Based Science Assessment.
Digitally Based Assessment
NAEP Digitally Based Science Assessment
The NAEP science assessment was administered for the first time in 2019 as a digital assessment at grades 4, 8, and 12. The science DBA was designed to continue reporting trends in student performance while keeping pace with the new generation of classroom environments in which digital technology has increasingly become a part of students' learning. The science DBAs at all three grades were administered on tablets supplied by NCES using a secure, local NAEP network. This allowed the NAEP administrators to create a stable administration environment by bringing in their own equipment that would not be influenced by school-based equipment or school internet connectivity, thereby maintaining consistency across the assessed schools. Students were able to interact with the tablets via touchscreen, with an attached keyboard, or using a stylus provided by NCES.
The science DBA provided students with a variety of on-screen tools to help them interact with the assessment such as zooming capability, different contrast settings, and elimination of response choices. At the beginning of the assessment session, students viewed an interactive tutorial that provided the information needed to take the assessment on tablets; for example, the tutorial explains how to progress through questions, how to indicate answers for multiple-choice questions, and how to use on-screen tools effectively when answering questions. The interactive nature of the tutorial allowed students to familiarize themselves with the digital delivery system before beginning the actual assessment. See how the science DBA was presented to students.
Science assessment development and design
The 2019 science assessment was developed with the same science framework used to develop the 2015 paper-based assessment. Assessment content at all three grades consisted of standalone, discrete questions (e.g., multiple choice and constructed response) that were transadapted from the previous PBA as well as newly developed, digitally based discrete questions that were designed to take advantage of the digital delivery system, including the development of interactive questions using multimedia stimuli and on-screen tools as described in the preceding paragraph. Questions that were previously used in the paper-based format were adapted to fit a tablet screen. While the presentation of the content changed, the content itself was not changed. Most of the 2015 PBA content at all three grades was also used in the 2019 DBA. The goal of adapting questions from paper-based to a digital format included retaining the same measurement targets in the digital version as were in the original paper-based version of the question. At grade 4, of the ten question blocks administered in 2019, seven were transadapted from the 2015 PBA and three blocks were newly developed for DBA. At grade 8, of the 11 blocks administered in 2019, eight were transadapted from PBA and three were newly developed. At grade 12, of the 12 question blocks administered in 2015, nine were transadapted from PBA and three were newly developed.
In addition to standalone questions, the science DBA included interactive, scenario-based tasks (SBTs) that were designed to engage students in solving real-world scientific problems in a digital environment and provide them opportunities to demonstrate a range of knowledge and skills in each of the three science content areas and four science practices. SBTs are comprised of a sequence of connected questions (e.g., multiple choice and constructed response) that are related to a single scenario. Although SBT questions are connected, each task is designed to enable students to progress through the task to completion even if they provide partial or incorrect responses to one or more questions. Individual task questions are designed so that students with a partial understanding could still provide a response and students who answer a question incorrectly still have an opportunity to answer subsequent questions correctly. The science assessment included two types of SBT question blocks:
- Interactive computer tasks (ICTs). ICTs use real-world simulations to engage students in scientific investigations that require the use of science inquiry skills and application of scientific knowledge to solve problems.
- Hybrid hands-on tasks (HHOTs). Students perform hands-on scientific investigations using materials in kits provided by NCES. The “hybrid” in HHOTs denotes that these tasks combine hands-on investigations with digital activities. Students use NCES-supplied tablets to view kit instructions, record results and data, and answer assessment questions.
SBTs administered as part of the grades 4 and 8 assessments each included two extended ICTs (30 minutes) and one 30-minute HHOT. Grade 12 SBTs included one 30-minute ICT, two short ICTs (15 minutes), and two 30-minute HHOTs.
To minimize test-taking burden, students do not take the entire NAEP assessment. Rather, each student answers a small subset of the entire assessment. Total testing time for all three grades was 60 minutes. At grades 4, 8, and 12 students could be administered different combinations of question blocks as follows:
- two 30-minute ICTs;
- two 30-minute discrete question blocks;
- one 30-minute HHOT and one 30-minute ICT;
- one 30-minute ICT and one 30-minute set of discrete questions; or
- one 30-minute HHOT and one 30-minute set of discrete questions;
- at grade 12 only, instead of a single 30-minute ICT, students could be administered two 15-minute ICTs, along with either a 30-minute HHOT, a 30-minute ICT, or a 30-minute discrete question block.
Note that two 30-minute HHOTs were not administered together as part of the 2019 science assessment.
Science assessment transition from paper to a digital format
To estimate overall trend scores and evaluate the effectiveness of the digital transition, the 2015 science PBA at grades 4, 8, and 12 were re-administered in 2019. A multistep process was used for the transition from PBA to DBA in order to preserve trend lines that show student performance over time. The transition process involved administering the assessment in both the DBA and PBA formats to randomly assigned groups of students at each grade in 2019. NCES administered the assessments in both modes—paper-based and digitally based—in all the sampled schools to investigate potential differences in performance between students taking the assessment on a tablet and students taking the PBA. However, in schools with fewer than 21 students, all students were assigned to either DBA or PBA. Each participating student took the assessment in only one mode. The results from the DBA can therefore be compared to those from previous assessment years, showing how students' performance in science has changed over time. Read more about the science transition and mode evaluation.
After the administration of the assessments at each grade, NCES conducted rigorous analyses of the data and aligned the 2019 results to previous assessment years using a two-step process as follows:
- First, common item linking was used to calculate the trend line from 2015 to 2019 based on the PBA results. This kind of linking was possible because the majority of the 2019 assessment questions were also administered in 2015 and showed the same statistical properties.
- Second, common population linking was used to align the 2019 PBA results with the 2019 DBA results. This kind of linking was possible because the samples of students, within each sampled school, were randomly assigned to either the DBA or PBA mode where practical. The sampling and weighting procedures ensured the two samples were sufficiently comparable in their demographic composition.
Once the common population linking aligned the digital results to the paper results on the national level, the analyses evaluated whether the linking allowed for fair and meaningful comparisons for national student groups. These evaluations supported making trend comparisons between DBA and previous PBAs for student groups.
These analyses—common item linking based on paper results and common population linking of paper results to digital results—enabled NCES to successfully maintain the science trend lines while transitioning to the digital assessment in 2019. The 2019 science assessment results in this report are based on the combined performance of students who took the assessment on paper and students who took the assessment on tablets.
NAEP conducted bridge studies to evaluate the effects of the change in administration mode of the 2019 science assessment from paper to a digital format. Mode change effects across student groups were part of the evaluation. Results showed few instances of statistically significant differences in mode effects across grades 4, 8 and 12. At grades 4 and 8, mode effects were observed for students identified and those not identified as students with disabilities (SD). Specifically, at both grades 4 and 8, DBA performance was lower than PBA performance for students identified as SD. Also, at both grades, DBA performance was higher than PBA performance for students not identified as SD. The magnitude of the mode effect was greater for students identified as SD. Therefore, at grades 4 and 8, results that compare performance between 2019 and previous years for students identified as SD should be interpreted with caution. For more details see the science transition and mode evaluation study.
Given the difference between traditional, standalone science questions and SBTs, evaluations were conducted to determine the viability of integrating the SBTs in the science scales reported for each grade. A study was conducted to determine whether the integration of SBTs had any effect on the dimensionality of the science results. The evidence suggested that for the 2019 NAEP science results, questions in SBTs did not measure any distinct dimensions from traditional, discrete science questions. The analysis also suggested that the inclusion of SBTs into the science scale did not change the student group performance estimates for the 2019 NAEP science results. Therefore, SBT performance results were included in the overall NAEP science scale at each grade in 2019. Read more about SBT integration in the science scale.