The National Center for Education Statistics continues its commitment to better understanding what the nations’ students know by releasing a new sample of process data from the math portion of the National Assessment of Educational Progress (NAEP). This website informs all prospective users who plan to work with the NAEP process data. The following sections describe what process data are, how the process data are prepared, and how the final data sets are structured.
Introduction
Digital technology has been increasingly utilized in classrooms over the last decade, and NAEP has made significant strides to align with these practices. In 2017, the
NAEP mathematics assessment was operationally administered for the first time as a digitally based assessment for grades 4 and 8, providing students with an engaging assessment experience corresponding to the delivery mode of many other large-scale assessments.
This transition to digitally based assessment (DBA) has also allowed NAEP to measure content knowledge in innovative ways, employ assistive technology to provide enhanced accommodations for students with disabilities, and gather new types of data that deepen our understanding of what students know and can do, including their utilization of new technologies (e.g., equation editor) to interact with the assessment items.
The data logged within the testing system include student interactions with the assessment platform and the assessment tasks; the data consist of time-stamped records of student-initiated actions (e.g., clicking an assessment feature such as drawing tools) and automatically generated actions by the system (e.g., enter item). The information extracted from this data represents students' interactions during the assessment and processes to arrive at answers, including the time students take to complete specific tasks, the steps or strategies they use to solve problems, and the resources or tools they utilize during the assessment. These new data are called
process data. To further enrich our understanding of what students know and can do in the digital environment, process data from the 2017 NAEP grade 8 mathematics assessment was previously made available for secondary analysis in 2021. Process data from the 2017 NAEP grade 4 mathematics will also be accessible for further examination in 2023.
Digitally Based Assessment in NAEP
NAEP DBAs provide enhanced flexibility in meeting the needs of different students, offering a range of
Universal Design Elements, or built-in functions, which facilitate broader participation without requiring special accommodation sessions. In the digital realm, what used to be considered accommodations in traditional paper-based testing are seamlessly integrated into the universal design framework (e.g., read aloud using text-to-speech, highlighter tool). NAEP also uses assistive technology to offer accommodations for students with disabilities (e.g., extended time). Furthermore, onscreen tools are provided to students to assist them in their problem-solving, such as an in-test equation editor. The goal is for all students to have a seamless assessment administration.
At the beginning of each assessment, NAEP offers a brief, interactive tutorial that allows students to become familiar with the system, its tools, and its functionality. The tutorials are available to the public online. Experience one of the
NAEP DBA tutorials and learn more about the tools available to students.
Some Universal Design Elements and onscreen tools available for the NAEP grade 4 mathematics assessment are described below.
Figure 1. Example item: 2017 NAEP Grade 4, illustration of Universal Design Elements.
Color Contrast/Change Theme | Students have a choice of three color-contrast options, including one high-contrast option. |
---|
Zooming | Text size options are provided for short stand-alone items (e.g., multiple-choice and short constructed-response items) but not for scenario-based tasks. Default zoom option is 100% with a choice of three other options, or “states” (125%, 150%, and 200%). |
---|
Text-to-Speech
| Directions, text, figures, and tables within the assessment can be read aloud using the text-to-speech function.
|
---|
Scratchwork | A scratchwork tool is available for student use. This tool contains an embedded pencil mode and highlighter mode for annotating figures, performing computations, drawing diagrams, and highlighting portions of a question. Eraser mode is used to clear any drawings and highlights.
|
---|
Equation Editor
| An onscreen equation editor is provided for specific items to enter numbers and expressions using the available mathematical symbols.
|
---|
Calculator
| For specific blocks in which a calculator is allowed, an onscreen calculator is provided. Please note that calculator model differs by grade.
|
---|
Try the elements and tools by creating
your own test containing released questions from the 2017 NAEP grade 4 or 8 mathematics assessment.
About NAEP Process Data
The NAEP digital platform collects different types of data during a student’s assessment session. In addition to students’ responses to items in the cognitive assessment and their responses to survey questions, the testing system also logs the actions or events as students interact with the digital assessment. This data can be transformed into process data variables that represent meaningful student interactions with NAEP assessments. For example, process data variables can describe the amount of time that a student spent on an item; usage of the text to speech, zooming, and highlighting tools; and the number of visits to each assessment item. In contrast to the grade 8 data release, the current release of grade 4 data will not include pre-calculated process data variables. The primary rationale behind this decision is to promote researchers' engagement with the data to gain a deeper understanding of the dataset and its nuances and encourage them to derive the variables that serve their specific research questions and needs better.
The table below provides a high-level overview of the data in the files available from the released 2017 NAEP grade 4 and grade 8 mathematics assessment dataset. Both grade 8 and grade 4 releases include the full suite of restricted-use response data product materials, process data files, released cognitive items, and survey questionnaire files.
Restricted-Use Response Data | - Respondent data
- Documentation
- File formats for different software packages
|
---|
Restricted-Use Process Data | - Process data
- Documentation
|
---|
Restricted-Use Complementary Data | - Student demographics and accommodation information
- Student raw and scored item responses
- Item bridge information
- Item file pdf
- Student survey questionnaire pdf
|
---|
A brief description of the data provided above is presented below.
- Respondent data include students’ raw response data for cognitive and survey questionnaire items, student demographics and accommodation information; and teacher and school level variables;
- Process data contain process data logs from each student;
- Student information provided within the complementary data include selected demographic and accommodation variables from the respondent data, and students’ raw (e.g., option A for a multiple-choice item) and scored (e.g., 1 for correct, 0 for incorrect) item responses including total raw score for each block1;
- Item information provided within the complementary data include item ID bridge (to support appropriate merging of data files) and images of the released assessment items from the eNAEP digital test platform;
- Student survey questionnaire within the complementary data asks about information on students’ demographic characteristics, opportunities to learn in and out of the classroom, and educational experiences;
About 2017 NAEP Mathematics Released Process Datasets
The 2017 NAEP grade 4 and grade 8 mathematics assessments included two 30-minute blocks of cognitive items in each digital test form, followed by a 15-minute survey questionnaire that collected information on students’ demographic characteristics, opportunities for learning in and outside of the classroom, and educational experiences. There were 10 cognitive blocks administered at each grade, where each block was paired with every other block, resulting in 50 unique digital test forms. After the assessment administration, the NAEP program released two blocks of cognitive items used in 2017 for both grades 4 and 8. One of the released cognitive blocks at each grade was adapted from the paper-based assessment, and the other released cognitive block was designed for the digitally based assessment. It is important to note that the released blocks may not contain all administered items, which creates a partially released block. The unreleased items are secured for reuse in future assessments. At both grades, one of 50 test forms includes both released blocks, while the other forms may only contain one or no released blocks.
The upcoming grade 4 release and the previous grade 8 release both include block and form level process data. However, there are some differences in the number and specific focus of the data files. Below is a table describing the main differences between the two releases.
Grade 4 | Grade 8 |
---|
Two partially released blocks (MA and MI) | One fully released block (MC) |
Block MA: 13 out of 14 items were released Block MI: 16 out of 20 items were released | Block MC: all 15 items were released |
Block MA: ~ 29,100 students Block MI: ~ 30,300 students | Block MC: ~ 28,200 students |
Specific action focused data for select items:
- Calculator data
- Keypress and Equation Editor data
| Specific action focused data not included |
One Form comprised of Block MI and Block MA | One Form comprised of Block MI (only at the form level) and Block MC |
Form ~ 2900 students | Form ~ 2800 students |
Features data not included | Includes features data (pre-calculated process data variables) |
Below is a table describing the main variables included in the process data files for both Grade 4 and Grade 8.
Variable Name | Definition |
---|
PSEUDO | Anonymized student identification number |
BLOCKCODE | The identifier for the block. |
ACCESSIONNUMBER | The unique identifier for the item. |
ITEMTYPECODE | The type of item (e.g., multiple choice, fill in the blank). |
OBSERVABLETYPE | The executed action (e.g., enter item, math keypress). |
CUMULATIVETIME | The elapsed time, in seconds, starting from the first event, up to and including each event within a block. |
EXTENDEDINFO | Additional information for specific OBSERVABLETYPE actions. |
Get Familiar with NAEP Process Data
This website provides information about the upcoming release of data files from the 2017 NAEP grade 4 mathematics assessment. Researchers will have access to process data collected from two blocks of grade 4 mathematics assessment questions, along with process data from the grade 4 mathematics survey questionnaire (e.g., capturing students' demographic characteristics, learning opportunities in and outside of school, and educational experiences). Process data allows researchers to go beyond analyzing students’ responses to questions as simply right or wrong. Instead, process data enables researchers to delve deeper into research by exploring topics such as the time students spend on questions, pathways through assessment sections, and tools used while solving problems.
To effectively utilize process data, researchers must possess a solid understanding of the data and its characteristics. Before deriving insights from process data, researchers must create meaningful variables. Researchers interested in using the data from the grade 4 release will need to extract their own process data variables, as the dataset of pre-constructed process data variables will not be available (only available in the grade 8 release). The intention is to encourage researchers to gain a more comprehensive understanding of Process Data and to foster the generation of a diverse range of derived variables from NAEP data that can effectively address their research questions. We encourage interested researchers to watch the webinars,
Introduction to NAEP Process Data and
2017 NAEP Grade 4 Mathematics Process Data. These webinars will assist researchers in generating research questions and adopting best practices for data utilization. The webinars cover topics such as the preparation of process data for analysis, challenges associated with extracting meaningful insights from this type of data, as well as research examples and analyses that have utilized NAEP process data. The second webinar also provides information specific to the current grade 4 process data release and introduces the new item-level process data.
Access This Dataset
To explore this process dataset, interested researchers should
apply for a restricted-use license and request access to the files through the National Center for Education Statistics (NCES) website.
Protecting Personally Identifiable Information of Students, Teachers, and Schools in NAEP Data
Under the National Assessment of Educational Progress Authorization Act (Public Law 107-279 III, section 303), the Commissioner of the National Center for Education Statistics (NCES) is charged with ensuring that NAEP tests do not question test takers about personal or family beliefs or make information about their personal identity publicly available. By design, Personally Identifiable Information (PII) is not available at the individual student or school levels. By law, NAEP is not allowed to report results at the individual or school levels.
Direct identifiers (i.e., names of students and schools) are not available in the restricted-use process dataset. Each student is represented with a unique pseudo ID that cannot be directly linked to PII. Although every care is taken during the preparation of data sets, the restricted-use process data files may contain PII. However, if this occurs, researchers must remember that they have promised under penalty of law to keep these identities confidential. Any re-identification of students or schools through the use of this dataset is prohibited by law. Violations are subject to prosecution as a Class E Felony, with penalties of up to five years imprisonment and/or a $250,000 fine.
Frequently Asked Questions: FAQs
-
What are the benefits of using process data?
- Process data provide a wealth of information on how students interact with assessment and the digital platforms that were previously not available in paper and pencil assessments. Coupling process data with the NAEP background data can provide additional insights to better understand student behavior and performance.
-
What security measures do I/my organization need to use this data?
- Please refer to sections 2.4 and 2.5 of the
restricted-use license manual.
-
What software is needed to analyze process data?
- The process data are provided in the text file format that can be analyzed using statistical software such as R, Stata, SPSS, or Python.
-
Is there a way for me to view the original exam questions and interact with the digitally based assessment?
- Exam questions can be viewed using the
NAEP Questions Tool.
-
What is the relationship between a form and a block?
- For each form, NAEP’s assessment design pairs each block with every other block; therefore, the same block is included across multiple forms. For example, at grade 8, ten of the 50 forms included the released block. At grade 4, ten out of 50 forms included the partially released blocks. For more information please visit
https://nces.ed.gov/nationsreportcard/tdw/instruments/cog_blockdesign.aspx
- On the NAEP website, there are some student questionnaire items with an asterisk and a note that states, “*This question was intended to be administered in the student questionnaire, but it was not administered to a sufficient number of respondents to support reporting.”
Will data for those items be in the restricted use data license?
- No, the responses to those survey items are not included in the release product.
-
What are some additional resources to learn about NAEP and NAEP assessments?
- NAEP Sampling Design –
https://nces.ed.gov/nationsreportcard/assessment_process/selection.aspx
- NAEP Methods and Procedures –
https://nces.ed.gov/nationsreportcard/tdw/
- NAEP Questions Tool –
https://www.nationsreportcard.gov/nqt/
- NAEP Digitally Based Assessments –
https://nces.ed.gov/nationsreportcard/dba/
- NAEP Resources for Researchers –
https://nces.ed.gov/nationsreportcard/researchcenter/
1Please note that plausible values are not included in the grade 4 and grade 8 release data files. Student proficiency levels (i.e., basic, proficient, advanced) are not included in the grade 4 release data files.