NAEP Monthly School Survey Linking Study

Survey responses were collected from nationally representative samples of schools with a grade 4 or a grade 8; the collected data are representative of the 45 states/jurisdictions that participated in the survey. School administrators were asked to provide information about the various instructional modes in their schools for the months of January through May of 2021. These same schools were a subset of the schools that later participated in the 2022 NAEP assessments, therefore allowing the linkage between the survey results and performances on the assessments a year later—when most schools were back to school full-time in-person.

School administrators were asked to provide the number of enrolled fourth- and eighth-grade students (rounded to the nearest hundred) by each of the following mode of instruction categories: (a) Remote or online only; (b) Open with both remote/online and in-person instruction (hybrid model); and (c) Open with full-time in-person for all students in their school in the specified month. See the complete Monthly School Survey questionnaire.

Approximately 1,380 to 1,640 schools at grade 4 and 1,290 to 1,550 schools at grade 8 responded to the question about full-time, in-person school instruction across the five months of data collection in 2021. The number of students in 2022 associated with the school responses was approximately 26,800 to 34,100 fourth graders and 29,500 to 35,700 eighth graders across the subjects (table 1).

Table 1.  Number of public schools and students in the 2022 Monthly School Survey Linking Study sample with valid responses to a question about enrollment in instructional modes from the 2021 Monthly School Survey

Grade 4

Grade 8

NOTE: School administrators were asked to provide the number of fourth- and eighth-grade students (rounded to the nearest hundred) enrolled in each of the following mode of instruction categories: (a) Remote or online only; (b) Open with both remote/online and in-person instruction (hybrid model); and (c) Open with full-time in-person for all students in their school in the immediately preceding month. The number of schools is rounded to the nearest ten and the number of students is rounded to the nearest hundred. Some schools or students were included in multiple months but each student only participated in either the mathematics or reading assessment in 2022. See the complete Monthly School Survey questionnaire.

Design and Administration

The 2021 Monthly School Survey

The 2021 NAEP mathematics and reading assessments at grades 4 and 8 were postponed due to the COVID-19 pandemic which began in the spring of 2020, and continued through the planned January to May 2021 NAEP testing window; however, the National Center for Education Statistics (NCES) used the planned 2021 school samples to respond to the Executive Order on Supporting the Reopening and Continuing Operation of Schools and Early Childhood Education Providers by conducting the Monthly School Survey (MSS) during that window. The intended respondents of the MSS were NAEP school administrators. The survey consisted of ten questions covering a variety of topics related to education during the COVID-19 pandemic, such as information about the mode of learning offered and student enrollment by instructional mode. The survey was administered monthly, through a web-based data collection system, from February to June 2021. The survey asked respondents about the immediately preceding month (i.e., the February collection asked about January). The results were reported monthly, shortly after the close of each administration. See the Monthly School Survey Dashboard for more information on the 2021 survey and results.

The 2021 Monthly School Survey was administered to approximately 5,000 public schools that serve fourth- and eighth-grade students throughout the nation. While schools from all jurisdictions were provided the 2021 Monthly School Survey, the following seven jurisdictions did not participate: Alaska, Hawaii, Montana, South Dakota, Utah, West Virginia, and Department of Defense Education Activity. The school sample for the survey was reduced by about half from a typical operational NAEP data collection, primarily due to the pandemic and associated increased costs. At each grade, the target sample sizes varied from 40 to 70 schools per state, depending on the state’s size and whether it contained districts participating in the NAEP Trial Urban District Assessment (TUDA). Ten urban districts participated in the survey. Participating jurisdictions all contribute toward the national-level results presented here.

The 2022 NAEP Mathematics and Reading Assessments

The 2022 NAEP mathematics and reading assessments were administered to representative samples of fourth- and eighth-grade students in the nation, states, the District of Columbia, Department of Defense schools, and 26 participating large urban districts. Samples of schools and students were drawn from each state and from the District of Columbia and Department of Defense schools. The sample of students participating in the TUDA school districts is an extension of the sample of students who would usually be selected by NCES as part of the national and state samples for the NAEP assessment. The schools and students were selected to ensure that the samples are representative of all students nationally and of public-school students at the state/jurisdiction and participating large urban district levels. In addition, the schools were selected to have maximum overlap with the 2021 Monthly School Survey school sample, allowing the linkage between the 2021 Monthly School Survey data and the 2022 NAEP data based on the common set of schools. Scores are reported on NAEP mathematics and reading scales at grades 4 and 8 that range from 0 to 500.

The NAEP Monthly School Survey Linking Study

Using the portion of the 2022 main NAEP operational sample belonging to the same schools that were sampled for the 2021 MSS, the school-level 2021 Monthly School Survey data were linked to the 2022 NAEP school and student data through school identifiers. There was no student-level data (e.g., attendance, enrollment, mode of instruction) collected in the 2021 Monthly School Survey. Therefore, the students’ performance on 2022 NAEP assessments can only be described based on their schools’ responses to the 2021 MSS on the cohorts of students who were in fourth grade or eighth grade in 2021. Private schools were excluded from the linking study because there was no designed overlapping of private school samples between the 2021 MSS and 2022 NAEP samples. Puerto Rico was excluded from the linking study.

Sampling Weights for the NAEP Monthly School Survey Linking Study

The sampling weights allow for generalizing to the target populations for the linking study, which are defined as grade 4 and grade 8 students in public schools that existed in both the 2021 MSS and the 2022 NAEP mathematics and reading assessments. Because six states are missing in the target population, the results are not fully national. Schools eligible for the linking study analysis are public schools that were selected for both the 2021 MSS and the 2022 NAEP assessments at grades 4 and 8. The responding schools are defined as the public schools that participated in the 2022 NAEP assessments and responded to the 2021 Monthly School Survey in months 2 and/or 3 of data collection (March and/or April of 2021) to Question 1 (i.e., “In [previous month], what did your school offer as the mode of instruction or learning model for grade [4 or 8 as appropriate] students?”). Nonresponding schools are eligible schools that did not participate in both the 2021 Monthly School Survey and 2022 NAEP assessments. While this publicly available report shows results for February 2021, the analysis procedures were developed such that secondary users could analyze data from all months with the separate restricted use MSS Data Product.

Months 2 and/or 3 of the Monthly School Survey were chosen to define the responding schools because those months exhibited higher response rates than other survey data collection months (i.e., Months 1, 4, and 5). Monthly School Survey Question 1 was chosen because the question is related to learning mode and the response rate is high relative to other questions.

Linking study sampling weights were derived for the 2022 NAEP students in participating linking study schools. The weighting procedure consisted of a combination of the 2021 Monthly School Survey and the 2022 NAEP weighting processes. The school base weights and replication procedures reflected the Monthly School Survey selection probabilities and the Monthly School Survey variance structure, respectively. School nonresponse adjustments were recomputed using the linking study redefined response status. At the student level, all 2022 NAEP weighting processes (student nonresponse, trimming, and raking) were rerun on the students in the linking study schools.

Table 2 contains counts of the linking study eligible public schools, and the counts and unweighted percentages of the linking study schools that meet the definition of a responding school.

Table 2. Number of eligible public schools, number of responding public schools, and unweighted percentage of eligible public schools that meet responding school definition in NAEP Monthly School Survey Linking Study: 2022

NOTE: The number of schools is rounded to the nearest ten. The 2021 Monthly School Survey (MSS) was administered to approximately 5,000 public schools that serve fourth- and eighth-grade students throughout the nation. The subset of schools that belong to the 5,000 but not the 4,280 (presented in this table) are those that were selected for the 2021 MSS but not the 2022 NAEP assessments, as well as those that were in jurisdictions that did not participate in the 2021 MSS. Schools from the 45 jurisdictions that participated in the 2021 MSS data collection are included in the total number of eligible schools; Alaska, Hawaii, Montana, South Dakota, Utah, West Virginia, and Department of Defense Education Activity did not participate.

Group Score Estimation for the NAEP Monthly School Survey Linking Study

To estimate student group scores for the linking study, latent regression analysis was conducted. The covariates of the latent regression model include variables associated with all three sub-questions of Survey Question 1, a composite variable derived from question 1, the sub-questions of Survey Question 2 that pertains to all students for grade 4 or 8, and Survey Question 6. The Monthly School Survey variables derived from each of the five months’ 2021 survey were included in the latent regression models. In addition, major reporting variables (sex, race/ethnicity, English learner status, eligibility for free/reduced-price lunch status, student disability status, census region, school location, and parental education level [grade 8 only]) collected from 2022 NAEP assessments and their interactions with 2021 MSS variables were included in the model.

Monthly School Survey Question 2 and its corresponding derived survey variables are described below.

Question 2: Enrollment and Mode of Instruction

Survey Question 2 (MSS Q2) asks schools to enter the number of enrolled students by each of the three modes of instruction for all students at grade 4 or 8 in a school and by race/ethnicity. Three variables (percent remote only, percent hybrid, percent in-person) were created categorizing the percentage of students at grade 4 or 8 in a school enrolled in each of the three modes of instruction (0%, 1‒25%, 26‒50%, 51‒75%, 76‒100%). The Monthly School Survey Linking Study does not include any school-level variables derived from the Monthly School Survey sub-questions on mode of instruction by race/ethnicity given that (1) school-level mode of instruction by race/ethnicity percentages are estimated less reliably than those for all students at a grade; (2) Monthly School Survey mode of instruction by race/ethnicity were collected on the cohorts of students who were in fourth grade or eighth grade in 2021. As described above, for this report, the results of mode of instruction by student characteristics are based on student characteristics collected on the 2022 NAEP assessments.

Additional analyses were conducted to evaluate nonresponse bias for both the overall sample and MSS Q2 highlighted in the report. At the overall level, distributions of key student characteristics based on the MSS linking study sample were compared to those from the 2022 operational NAEP sample, and the overlap in response across each pair of months at both the state and national levels were evaluated. These analyses were used to determine the criteria for non-reportable states. An additional layer of analysis was added specifically for MSS Q2 to evaluate the potential for nonresponse bias introduced due to question-level nonresponse. Comparisons were made between mean scores for respondents to Q2 and the mean scores based on the entire MSS linking study sample at each month for states, TUDAs, and major reporting student groups. Large differences that would suggest nonresponse bias were not observed.

Reporting Standards for the NAEP Monthly School Survey Linking Study

Following standard NAEP operational procedures, the minimum requirement for reporting subpopulation statistics is 62 students in a particular group from at least five primary sampling units (PSUs). Further, subpopulation statistics are suppressed if the weighted student response rate for a group is less than or equal to 50 percent. The statistics are flagged if the weighted student response rate is less than or equal to 70 percent.

The participation standards for states and jurisdictions to have their results published are:

• There are at least 25 responding schools for the state/jurisdiction.
• Response rates to Monthly School Survey Question 1 are greater than 50% for at least four of the five survey months. Response rates are weighted by the student sampling weights.

The following states did not meet the standards for having their results to be reported separately: Grade 4: Colorado, Indiana, Massachusetts, Nevada, New Mexico, South Carolina, and Tennessee. Grade 8: Colorado, Indiana, Nevada, New Mexico, North Dakota, South Carolina, and Tennessee. The school and student samples from states that did not meet the standards for their results to be reported still contributed to the national results.

Statistical Comparisons

The differences between group average scores and between percentages discussed in the results take into account the standard errors associated with the estimates. All statistical comparisons are based on independent t-tests that consider both the magnitude of the difference between the group average scores or percentages and the standard errors of those statistics.

All differences reported are statistically significant at the 0.05 level with appropriate adjustments for multiple comparisons. The term "significant" is not intended to imply a judgment about the absolute magnitude or the educational relevance of the differences.

Cautions in Interpretation

The Monthly School Survey Linking Study is an experimental data collection and reporting project. School-reported data about that school’s “environment” is being linked to assessment results a year later. Readers should note that the students represented in the 2021 Monthly School Survey were not the same students who were assessed in the 2022 NAEP grades 4 and 8 mathematics and reading assessments. Furthermore, the 2021 Monthly School Survey results do not contain information about the enrollment status of students in 2022. Readers should also note that when they explore the earlier dashboard and highlights report that those are the percentages of schools from 2021 and cannot be compared to the percentages of students in 2022 presented on this page.

When interpreting the findings presented here, readers are encouraged to keep in mind that these results do not account for multiple factors outside of the survey that may affect the results. Factors such as population density, COVID-19 infection rates, resources, and policies may influence the correlation between learning mode and the student groups reported here. Since the representation of racial/ethnic student groups varies across school locations and regions, it is important to consider the broader demographic patterns in the United States when interpreting the descriptive statistics presented.