Technical Notes for Racial/Ethnic Achievement Gap Tool

The Racial/Ethnic Achievement Gap Tool provides users with the opportunity to observe how various factors are related to the achievement gaps of select racial/ethnic student groups. At present, this tool uses 2019 NAEP reading data to allow users to see how average score gaps between White–Black, White–Hispanic, and White–Asian students change when you consider additional factors that may relate to student performance.

The analyses shown in the tool are exploratory. The tool allows users to examine whether there are associations between select factors or variables collected by NAEP and the White–Black, White–Hispanic and White–Asian average student score gaps. The tool does not assess whether the relationship between factors and scores gaps is causal (i.e., this analysis does not and cannot test whether the select factor causes differential achievement among the student racial/ethnic groups explored here).

NAEP does not provide scores for individual students or schools; instead, it offers results for populations of students (e.g., fourth-graders) and subgroups of those populations (e.g., female students, Hispanic students) as well as results regarding subject-matter achievement, instructional experiences, and school environment. NAEP results are based on school and student samples that are carefully designed to accurately represent student populations of interest.

Variable Information

Variable selection

Exploratory analyses were conducted prior to the final variable selection for the models in the regression analysis. We began with a larger list of potential predictors that could fit into the categories of interest: socioeconomic status (individual); socioeconomic status (school environment); student postsecondary plans and academic behaviors; and student noncognitive factors. Variables that had small associations when controlling for other variables in the model were excluded. These variables were:

NAEPID
Item Description
PARENTS
Parents in the home
UTOL4
School location (i.e., city, suburb, town, or rural)
PCTBLKC
Percent of Black students in school
PCTHSPC
Percent of Hispanic students in school
C087302
Percent of parents attend parent-teacher conferences in school
R850901
How often for English class discuss reading
R850902
How often English class work in pairs/groups to discuss reading
R850903
How often discuss different interpretations of what we have read
R851003
How often teachers ask students to critique author's craft or technique in English/language arts class
R850202
How often interpret the meaning of the passage
RDACT12
How often when reading evaluate/analyze/critique

A handful of interaction terms were of interest from the list of remaining variables. The following interactions were evaluated for potential inclusion in the models:

  1. SQCATR7 (Confidence in reading knowledge and skills index) x B013801 (Number of books in home)
  2. SQCATR7 (Confidence in reading knowledge and skills index) x C087602 (Proportion of last year's graduates attending a four-year college)
  3. B013801 (Number of books in home) x C087602 (Proportion of last year's graduates attending a four-year college)
  4. APIBELA (Taking/took an Advanced Placement (AP) English or language arts course and/or International Baccalaureate Language A1 course) x C087602 (Proportion of last year's graduates attending a four-year college)
  5. SRACE10 (Race/ethnicity) x B013801 (Number of books in home)
  6. SRACE10 (Race/ethnicity) x SQCATR7 (Confidence in reading knowledge and skills index)
  7. SRACE10 (Race/ethnicity) x C087602 (Proportion of last year's graduates attending a four-year college)
  8. SRACE10 (Race/ethnicity) x APIBELA (Taking/took an Advanced Placement (AP) English or language arts course and/or International Baccalaureate Language A1 course)
  9. SRACE10 (Race/ethnicity) x B018101 (Days absent from school in the last month)
  10. SRACE10 (Race/ethnicity) x PARED (Parental education level)

No interactions showed a meaningful increase in explained variance, so none were retained.

Tables 1, 2, and 3 show percentage, average score, and achievement-level results of fourth-, eighth-, and twelfth-grade students in the 2019 NAEP reading assessment for all variables included in the Racial/Ethnic Achievement Gap Tool. Variables in these tables show descriptive statistics for all categories of each variable, even though some categories were combined for purposes of running the regression models. See details about combining categories below.

Two or more categories in three of the predictor variables included in the regression were collapsed: COMPINT, SCHNSLP, and APIBELA (grade 12 only). Categories were collapsed to help with interpretation and because some categories had very small percentages of students. COMPINT was collapsed from four categories to two categories, where "both computer/tablet and Internet" was one category, and all other responses were combined in a second category. SCHNSLP was collapsed from six categories to five categories, where "all students" and "76–100%" were collapsed into the same category. APIBELA (grade 12 only) was collapsed from four categories to two categories, where "Neither" was one category, and all other responses were combined in a second category.

Please note that SCHNSLP is a derived variable constructed specifically for this secondary report.

SCHNSLP was constructed based on responses to variables with NAEP IDs of VH240215, VH240216, and VH240218. SCHNSLP is a 6-category variable constructed as follows:

  • If all responses to above variables are missing, then "Missing"; otherwise
  • If VH240215 = No, then 1 = "School does not participate"; otherwise
  • If VH240216 = All students, then 6 = "All students"; otherwise
  • Categorize VH240218 into 2 = "0–25%", 3 = "26–50%", 4 = "51–75%", 5 = "76–100%". If omit, then "Missing".

Additional details about these survey questionnaire variables that were used to construct the derived variable SCHNSLP are available in the NAEP Data Explorer (NDE) by using the variable's NAEP ID to locate.

TABLE 1 Percentage distribution, average scores, and achievement-level results of fourth-grade students in NAEP reading, by selected variables: 2019
Variable description, ID, and response categories Percentage of students Average scale score Percentage of students
Below NAEP Basic At NAEP Basic At NAEP Proficient At NAEP Advanced
Gender (DSEX)
Male 51 217 37 30 25 8
Female 49 224 30 32 28 10
Race/ethnicity (SRACE10)
White 44 232 22 32 34 13
Black 13 211 45 33 19 4
Hispanic 32 206 48 30 18 4
Asian 4 241 16 26 36 22
American Indian/Alaska Native 2 215 38 32 23 6
Native Hawaiian/Other Pacific Islander 1 210 45 30 20 5
Two or More Races 4 227 27 31 30 12
National School Lunch Program eligibility (SLUNCH3)
Eligible 51 207 47 31 18 3
Not eligible 43 235 19 30 36 15
Information not available 6 232 22 32 34 13
Proportion of students in school eligible for free or reduced-price lunch through the National School Lunch Program (SCHNSLP)
Not participating 15 231 23 31 33 13
0–25% 17 240 15 28 38 18
26–50% 19 227 27 33 30 10
51–75% 14 217 36 33 24 6
76–100% 8 206 48 31 17 3
All students 28 206 49 30 17 4
Have internet access and/or computer/tablet at home (COMPINT)
No computer/tablet/Internet 3 198 58 26 13 3
Computer/tablet no Internet 7 194 61 24 12 3
Internet no computer/tablet 9 214 41 33 21 5
Computer/tablet and Internet 81 226 28 32 29 10
Books in home (B013801)
0–10 12 200 55 30 13 2
11–25 24 210 44 33 19 4
26–100 34 227 26 33 31 10
>100 31 231 25 29 32 15

NOTE: Detail may not sum to totals because of rounding.

TABLE 2 Percentage distribution, average scores, and achievement-level results of eighth-grade students in NAEP reading, by selected variables: 2019
Variable description, ID, and response categories Percentage of students Average scale score Percentage of students
Below NAEP Basic At NAEP Basic At NAEP Proficient At NAEP Advanced
Gender (DSEX)
Male 51 258 32 39 26 3
Female 49 269 22 39 34 5
Race/ethnicity (SRACE10)
White 44 274 17 39 38 6
Black 12 246 45 40 15 1
Hispanic 30 251 38 40 20 1
Asian 5 285 12 30 45 13
American Indian/Alaska Native 1 249 41 40 17 2
Native Hawaiian/Other Pacific Islander 1 248 41 40 18 2
Two or More Races 6 266 24 41 30 4
National School Lunch Program eligibility (SLUNCH3)
Eligible 47 250 40 41 18 1
Not eligible 46 275 17 38 39 7
Information not available 7 276 16 37 40 7
Proportion of students in school eligible for free or reduced-price lunch through the National School Lunch Program (SCHNSLP)
Not participating 14 273 19 37 38 7
0–25% 18 279 14 36 42 8
26–50% 23 268 22 40 33 4
51–75% 15 259 31 41 26 2
76–100% 7 249 41 41 18 1
All students 22 248 41 40 18 1
Parental education level (PARED)
Did not finish high school 7 248 42 41 17 1
Graduated from high school 13 250 40 41 18 1
Some education after high school 14 265 23 45 29 3
Graduated from college 54 273 19 37 37 6
Unknown 12 243 47 37 15 1
Have internet access and/or computer/tablet at home (COMPINT)
No computer/tablet/Internet 1 233 58 32 9 #
Computer/tablet no Internet 2 236 52 35 12 1
Internet no computer/tablet 8 244 46 40 14 1
Computer/tablet and Internet 89 266 25 39 32 5
Books in home (B013801)
0–10 19 241 49 38 13 1
11–25 25 254 35 43 21 2
26–100 33 270 19 42 35 4
>100 23 282 13 33 45 10
Persistence in learning index (SQCATR4)
Low 13 251 39 38 20 2
Moderate 31 260 30 40 27 3
High 56 268 23 39 33 5
Academic self-discipline index (SQCATR5)
Low 19 248 43 39 17 1
Moderate 38 262 28 41 28 3
High 43 272 19 38 37 6
Confidence in reading knowledge and skills index (SQCATR7)
Low 5 220 73 23 4 #
Moderate 30 248 41 42 16 1
High 66 275 16 40 39 6
Performance goals in reading index (SQCATR9)
Low 38 262 28 40 28 4
Moderate 30 263 27 39 30 4
High 32 267 24 39 33 5
Mastery goals in reading index (SQCTR10)
Low 15 255 35 40 23 2
Moderate 27 262 28 39 29 4
High 58 268 23 39 33 5

# Rounds to zero.

NOTE: Detail may not sum to totals because of rounding.

TABLE 3 Percentage distribution, average scores, and achievement-level results of twelfth-grade students in NAEP reading, by selected variables: 2019
Variable description, ID, and response categories Percentage of students Average scale score Percentage of students
Below NAEP Basic At NAEP Basic At NAEP Proficient At NAEP Advanced
Gender (DSEX)
Male 50 279 35 33 27 5
Female 50 292 24 33 35 8
Race/ethnicity (SRACE10)
White 52 295 21 32 38 9
Black 13 263 50 33 16 1
Hispanic 25 274 39 36 22 3
Asian 6 299 21 29 37 14
American Indian/Alaska Native 1 272 41 35 22 2
Native Hawaiian/Other Pacific Islander # 278 34 39 25 2
Two or More Races 3 295 22 32 36 10
Parental education level (PARED)
Did not finish high school 10 269 44 36 19 2
Graduated from high school 16 271 42 35 21 2
Some education after high school 19 284 29 37 30 4
Graduated from college 51 297 21 31 38 10
Unknown 4 255 58 30 11 1
Books in home (B013801)
0–10 21 261 52 33 14 1
11–25 24 277 36 37 25 2
26–100 32 293 22 35 37 7
>100 23 307 14 27 44 15
Have internet access and/or computer/tablet at home (COMPINT)
No computer/tablet/Internet 1 245 64 28 7 #
Computer/tablet no Internet 1 265 46 35 17 2
Internet no computer/tablet 5 260 53 33 14 1
Computer/tablet and Internet 92 288 27 33 32 7
Proportion of students in school eligible for free or reduced-price lunch through the National School Lunch Program (SCHNSLP)
Not participating 16 296 21 31 38 9
0–25% 18 301 18 29 41 12
26–50% 27 288 27 33 33 7
51–75% 15 278 35 36 25 3
76–100% 6 266 47 35 17 1
All students 18 270 44 34 20 3
Proportion of last year's graduates attending a four-year college (C087602)
0–5% 3 265 49 33 16 2
6–10% 4 272 40 37 21 2
11–25% 18 276 38 35 24 3
26–50% 31 283 31 34 30 5
51–75% 21 291 25 32 35 8
Over 75% 14 306 15 27 43 15
I don't know 9 280 35 33 27 4
Applied to a four-year college (B035705)
Yes 60 299 19 31 40 10
No 40 270 43 36 19 2
Submitted the Free Application for Federal Student Aid (FAFSA) (B035702)
Yes 63 294 23 32 36 9
No 37 275 39 35 24 3
Days absent from school in the last month (B018101)
None 34 290 26 33 33 8
1–2 days 41 288 28 33 32 7
3–4 days 17 280 35 33 27 5
5–10 days 7 276 38 33 24 4
More than 10 days 2 256 55 30 15 1
Taking or took Advanced Placement (AP) English or language arts (ELA) course and/or International Baccalaureate (IB) Language A1 course (APIBELA)
Both 2 254 58 18 20 4
AP ELA only 31 305 15 27 44 14
IB Language A1 only 3 283 34 27 29 9
Neither 64 280 33 37 26 3
Persistence in learning index (SQCATR4)
Low 8 275 39 31 25 5
Moderate 27 285 30 33 31 6
High 65 288 28 33 32 7
Academic self-discipline index (SQCATR5)
Low 20 274 40 33 23 4
Moderate 42 287 28 34 32 6
High 38 292 25 32 35 8
Confidence in reading knowledge and skills index (SQCATR7)
Low 3 230 83 14 3 #
Moderate 24 263 49 35 15 1
High 73 297 20 34 38 9
Performance goals in reading index (SQCATR9)
Low 40 284 31 34 29 6
Moderate 30 285 30 33 31 7
High 30 291 25 33 35 8
Mastery goals in reading index (SQCTR10)
Low 21 279 35 34 26 4
Moderate 29 284 31 33 31 6
High 50 291 25 33 34 8

# Rounds to zero.

NOTE: Detail may not sum to totals because of rounding.

Indices related to students' attitudes toward learning

While some survey questions are analyzed and reported individually (for example, amount of books in students' homes), several questions on the same topic are combined into an index measuring a single underlying construct or concept. More information about the 2019 NAEP reading indices and their corresponding questions can be found in the 2019 NAEP reading student (grade 4, grade 8, and grade 12) questionnaires.

The creation of 2019 indices involved the following four main steps:

  1. Selection of constructs of interest. The selection of constructs of interest to be measured through the survey questionnaires was guided in part by the National Assessment Governing Board framework for collection and reporting of contextual information. In addition, NCES reviewed relevant literature on key contextual factors linked to student achievement in reading to identify the types of survey questions and constructs needed to examine these factors in the NAEP assessment.
  2. Question development. Survey questions were drafted, reviewed, and revised. Throughout the development process, the survey questions were reviewed by external advisory groups that included survey experts, subject-area experts, teachers, educational researchers, and statisticians. As noted above, some questions were drafted and revised with the intent of analyzing and reporting them individually; others were drafted and revised with the intent of combining them into indices measuring constructs of interest.
  3. Evaluation of questions. New and revised survey questions underwent pretesting whereby a small sample of participants (students, teachers, and school administrators) were interviewed to identify potential issues with their understanding of the questions and their ability to provide reliable and valid responses. Some questions were dropped or further revised based on the pretesting results. The questions were then further pretested among a larger group of participants and responses were analyzed. The overall distribution of responses was examined to evaluate whether participants were answering the questions as expected. Relationships between survey responses and student performance were also examined. A method known as factor analysis was used to examine the empirical relationships among questions to be included in the indices measuring constructs of interest. Factor analysis can show, based on relationships among responses to the questions, how strongly the questions "group together" as a measure of the same construct. Convergent and discriminant validity of the construct with respect to other constructs of interest were also examined. If the construct of interest had the expected pattern of relationships and nonrelationships, the construct validity of the factor as representing the intended index was supported.
  4. Index scoring. Using the item response theory (IRT) partial credit scaling model, index scores were estimated from students' responses and transformed onto a scale which ranged from 0 to 20. As a reporting aid, each index scale was divided into low, moderate, and high index score categories. The cut points for the index score categories were determined based on the average response to the set of survey questions in each index. In general, high average responses to individual questions correspond to high index score values, and low average responses to individual questions correspond to low index score values. As an example, for a set of index survey questions with five response categories (such as not at all, a little bit, somewhat, quite a bit, and very much), students with an average response of less than 3 (somewhat) would be classified as low on the index. Students with an average response greater than or equal to 3 (somewhat) to less than 4 (quite a bit) would be classified as moderate on the index. Finally, students with an average response of greater than or equal to 4 (quite a bit) would be classified as high on the index.

The following five indices were included in the regression model for grades 8 and 12:

TABLE 4 Items in the index of students' persistence in learning for 2019 NAEP reading assessment at grades 8 and 12
Item ID Sequence Question Response categories
Not at all like me A little bit like me Somewhat like me Quite a bit like me Very much like me
How much does each of the following statements describe you? Select one answer choice on each row.
B034901 a. I finish whatever I begin. A B C D E
B034902 b. I try very hard even after making mistakes. A B C D E
B034903 c. I keep working hard even when I feel like quitting. A B C D E
B034904 d. I keep trying to improve myself, even when it takes a long time to get there. A B C D E
TABLE 5 Items in the index of students' academic self-discipline for 2019 NAEP reading assessment at grades 8 and 12
Item ID Sequence Question Response categories
Never or hardly ever Less than half of the time About half of the time More than half of the time All or almost all of the time
In this school year, how often have you done each of the following? Select one answer choice on each row.
B035001 a. I started working on assignments right away rather than waiting until the last minute. A B C D E
B035002 b. I paid attention and resisted distractions. A B C D E
B035003 c. I stayed on task without reminders from my teacher. A B C D E
B035004 d. I paid attention in class even when I was not interested. A B C D E
TABLE 6 Items in the index of students' confidence in reading knowledge and skills for 2019 NAEP reading assessment at grades 8 and 12
Item ID Sequence Question Response categories
I definitely can't I probably can't  Maybe  I probably can I definitely can
Do you think you would be able to do each of the following when reading? Select one answer choice on each row.
R849601 a. Figure out the meaning of a word you don't know by using other words in the text A B C D E
R849602 b. Explain the meaning of something you have read A B C D E
R849603 c. Figure out the main idea of a text A B C D E
R849604 d. Find text in a reading passage to help you answer a question on a test A B C D E
R849605 e. Recognize when you don't understand something you are reading A B C D E
Do you think you would be able to do each of the following when reading? Select one answer choice on each row.
R850301 a. Recognize the difference between fact and opinion in a text A B C D E
R850302 b. Judge the reliability of sources (for example, how a website might be biased or inaccurate) A B C D E
R850303 c. Critique an author's craft or technique A B C D E
R850304 d. Use evidence from a text to support my answer A B C D E
R850305 e. Identify the author's perspective in a persuasive text A B C D E
TABLE 7 Items in the index of students' performance goals in reading for 2019 NAEP reading assessment at grades 8 and 12
Item ID Sequence Question Response categories
Not at all like me A little bit like me Somewhat like me Quite a bit like me Exactly like me
How much does each of the following statements describe you? Select one answer choice on each row.
R849701 a. I want other students to think I am good at reading. A B C D E
R849702 b. I want to show others that my English/language arts schoolwork is easy for me. A B C D E
R849703 c. I want to look smart in comparison to the other students in my English/language arts class. A B C D E
R851204 d. I want to get better English/language arts grades than most other students in my class. A B C D E
TABLE 8 Items in the index of students' mastery goals in reading for 2019 NAEP reading assessment at grades 8 and 12
Item ID Sequence Question Response categories
Not at all like me A little bit like me Somewhat like me Quite a bit like me Exactly like me
How much does each of the following statements describe you? Select one answer choice on each row.
R849704 a. I want to learn as much as possible in my English/language arts class. A B C D E
R849705 b. I want to become a better reader this year. A B C D E
R849706 c. I want to understand as much as I can in my English/language arts class. A B C D E
R851304 d. I want to master a lot of new English/language arts skills in my class. A B C D E

While the grade 4 student questionnaire also included the above items in the indices, missing rates for SQCATR7, SQCATR9, and SQCATR10 were above 15 percent and contributed to a high overall missing rate for regression analyses that included the indices. For example, only 66 percent of the grade 4 students were retained in the regression that included all indices and all of the variables shown in table 1. Therefore, no grade 4 indices were included as part of the Regression Tool.

Regression Models

Regression models were used to enable comparisons between unadjusted score gaps (i.e., regression coefficients from the model that only includes dummy variables for the racial/ethnic categories, excluding the category for White students as the reference group) to adjusted score gaps (i.e., regression coefficients from a model that includes dummy variables for some combination of predictors in addition to race/ethnicity). Unadjusted score gaps represent the difference between the average scores of two groups of students, and adjusted score gaps represent the estimated score gap once regression analysis controls for the variable(s) selected. For example, the White–Black score gap for Grade 4 is 26.70. It is the regression coefficient for the dummy variable associated with Black when the regression model only includes dummy variables for SRACE10 as predictors (excluding White as the reference group). The gap decreased to 12.02 when the SES variables were added as predictors to the model (i.e., when the gap was adjusted for the SES cluster of variables).

Unadjusted score gaps are shown as statistically significant if the regression coefficient for the dummy variable associated with a particular race/ethnicity category is significant at the α=0.05 level.

The regression analyses were run using the 2019 NAEP reading grades 4, 8, and 12 national reporting sample which included about 150,600 fourth-grade, 143,100 eighth-grade, and 26,700 twelfth-grade participating student records. All analyses used students' scale score as the outcome variable and utilized sampling weights. Coefficient estimates were calculated using 20 plausible values, and standard errors were calculated using 20 plausible values and 62 replicate weights. Missing data was handled by listwise deletion. All predictors included in the model are categorical and were included as dummy variables identifying each group of students for each variable, except for the one group chosen as the omitted category that serves as the reference group.

Comparing Adjusted to Unadjusted Score Gaps

Determining whether score gaps are significantly different between two regression models is not straightforward because the regression coefficients in each model are estimated from the same data, so errors on the regression coefficients for the same predictor may be expected to be positively correlated. With positively correlated or uncorrelated errors, determining statistical significance based on non-overlapping confidence intervals is conservative (Cumming, 2009), meaning that if two 𝑛 % confidence intervals do not overlap, the difference is statistically significant at the \[a = 1-{n \over 100}\] level; if confidence intervals do overlap, differences may not be statistically significant. Note that overlapping confidence intervals do not imply non-significance. The regression tool compares unadjusted score gaps (i.e., regression coefficients from the model that only includes dummy variables for the racial/ethnic categories, excluding the category for White students as the reference group) to adjusted score gaps (i.e., regression coefficients from a model that includes dummy variables for some combination of predictors in addition to race/ethnicity) using 95 percent confidence intervals. Non-overlapping confidence intervals imply statistical significance at the α = 0.05 level.

No multiple comparison adjustments were used in calculating the confidence intervals. Standard NAEP methodology utilizes the Benjamini-Hochberg false discovery rate (FDR) procedure to adjust for multiple comparisons in a single analysis (e.g., analyzing White student performance versus the performance of Black, Hispanic, and Asian students); however, the purpose of the confidence intervals in the regression tool is to compare adjusted versus non-adjusted score gaps within a racial/ethnic group rather than to compare unadjusted gaps across groups. Therefore, adjustments for multiple comparisons are not appropriate in this context.

References

Cumming, G. (2009). Inference by eye: Reading the overlap of independent confidence intervals. Statistics in Medicine, 28, 205–220.

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2019 Reading Assessment.