Script Author: Katja Borchert, Ph.D. (katjab@millisecond.com), Millisecond
Millisecond thanks Dr.Kondé for sharing the original script with the Millisecond Test Library!
Created: January 14, 2024
Last Modified: January 16, 2024 by K. Borchert (katjab@millisecond.com), Millisecond
Script Copyright © Millisecond Software, LLC
The script implements the Unexpected Solutions Task that investigates the verification efficiency of obvious and unexpected solutions to simple mathematical equations as a function of cognitive load. The cognitive load theory served as the theoretical framework for designing the method.
Kondé, Z., Kovács, Z., & Kónya, E. (2023). Modeling teachers’ reactions to unexpectedness. Learning and Instruction, 86. https://doi.org/10.1016/j.learninstruc.2023.101784
The Millisecond library script is based on the original Inquisit script by Kondé et al and edited to provide summary variables.
40 minutes
The test features a virtual math competition in which students compete against each other in teams and individually. Participants have to play the role of a teacher who assesses students' answers to simple math puzzles and announces the winner of the competition. In two stages of the competition, students solve eight simple math equations. The teacher evaluates the students' answers one by one to see if they are correct or not. Four teams of five participate in the team stage of the competition, and the aggregate score of the teams determine the winner. The winner of the individual competition is determined by the sum of the correct answers given by each student.
In a dual-task arrangement, verification of alternative solutions to simple mathematical equations is required
as the primary task, while memorizing task-irrelevant information is the secondary task.
Intrinsic load (IL) – content related load in the primary task
Alternative solutions to mathematical puzzles represents high intrinsic load (type B solution),
low intrinsic load (type A solution) conditions, and control condition (types C and D solution).
• IL is low if the correct answer provides an obvious solution to the equation (e.g. by performing a
single arithmetic operation);
• IL is high if the answer is correct but is the result of some additional operation compared to a
simple one-step solution (e.g. simplification of fractions; changing the representation of the fraction;
complicating the result by performing an unnecessary step; using alternative notation);
• Two false answers represent the control condition.
extrinsic load (EL) – context related load in the secondary task
• EL is high when, as a secondary task, additional, interfering information needs to be remembered while
processing the mathematical information (individual competition);
• EL is low when the memory demand of the secondary task is little (team competition).
The script uses 64 equations, and 256 solutions. Each equation has four alternative solutions, two correct
and two incorrect. Accordingly 128 solutions (types C and D) belong to the control, 64 to the
low intrinsic load condition (type A), and 64 to the high intrinsic load condition (type B).
Half of the equations coupled with related solutions are presented in the low extrinsic load condition
(team competition), and the other half are presented in the high extrinsic load condition (individual competition).
Note that the participants had to verify five answers (given by the five competitors of teams) for each equation,
resulting in 64 x 5=320 answers. While four solutions (two correct and two false) are assigned to each equation,
the fifth answer is a random repetition of one of the solutions presented earlier.
The purpose of using two false responses is to provide a chance of 0.5 that an answer is correct or incorrect.
In total, the probability of each of the alternative solutions is approximately the same.
EXPERIMENTAL SET-UP
(1) INTRO
- general introduction
- introducing the class, the teams, the math problems, the response keys
- practice (1 equation, 5 answers assigned to team members)
(2) Team competition (Low extrinsic load: LL)
• Team A-D (fixed assignments)
- each team consists of five competitors
- 8 equation per team
- 5 answers per equations (correct, alternative correct, false1, false2, + random repetition of one of the four alternatives)
- students and answers are randomly paired
• In the team stage of the competition, a total of 5 x 8 x 4 = 160 answers are given for verification in the first stage.
• Each run of five answers is followed by the follow-up question of team performance (trial.questionL)
- „How many correct answers were received from the team members?”
• The team competition ends with a final question for the winner of the team competition (block.teamEvaluation)
- "Which team do you think gave the most correct answers and win the quiz? Team A, B, C or D?"
(3) Individual competition (High extrinsic load: HL)
• Group A-D (same assignments as for LL condition)
- each group consists of five competitors
- 8 equation per group
- 5 answers per equations (correct, alternative correct, false1, false2, + random repetition of one of the four alternatives)
- students and answers are randomly paired
• In the individual stage of the competition, a total of 5 x 8 x 4 = 160 answers are given for verification in the second stage.
• Each run of five answers is followed by the follow up question for the performance of a randomly selected team member (QuestionB)
• The individual competition ends with a final question for the winner of the individual competition (QuestionCompetitor)
- „Did this competitor below give a correct or false answer?”, presented together with a portrait of a randomly selected student.
• The competition ends with a final question for the individual winner of the competition (QuestionCompetition)
- "Which student do you think win the competition?"
TIMING
The equations are presented for 3000 ms for observation.
The alternative solutions are presented one by one, each paired with a picture of a competitor. There is no time limit for verification. No immediate feedback is given regarding whether the verification was correct.
provided by Kondé et al (2023)
• Equations: 01-64 (see item.equations)
• Obvious correct solutions: 01A-64A ('A' solutions)
• Unexpected correct solutions: 01B-64B ('B' solutions)
• False solutions: 01C-64C; 01D-64D ('C' and 'D' solutions)
• equation and solutions for practice: 24P; 24AP, 24BP, 24CP, 24DP; 13P, 13AP, 13BP, 13CP, 14DP
• The class
• Teams for team and individual stage: TeamA, TeamB, TeamC, TeamD
• Competitors
Source of pictures: https://www.shutterstock.com/hu/image-vector/cute-smiling-faces-people-76786762.
provided by Kondé et al (2023)
All reported latency data are based on the filtered response times (250ms <= response times <= 30000ms)
File Name: unexpectedsolutionstask_summary*.iqdat
| Name | Description |
|---|---|
| inquisit.version | Inquisit version number |
| computer.platform | Device platform: win | mac |ios | android |
| startdate | Date script was run |
| startTime | Time the session was run |
| subjectId | Participant ID |
| groupId | Group number |
| sessionId | Session number |
| elapsedTime | Session duration in ms |
| completed | 0 = Test was not completed 1 = Test was completed |
Main Effect Intrinsic Load |
|
| accIntH | Proportion correct responses in high intrinsic load condition (responses to solution B) |
| corrRTIntH | Median correct response time (in ms) in high intrinsic load condition (responses to solution B) |
| accIntL | Proportion correct responses in low intrinsic load condition (responses to solution A) |
| corrRTIntL | Median correct response time (in ms) in low intrinsic load condition (responses to solution A) |
| accIntC | Proportion correct responses in control condition (responses to solution C and D) |
| corrRTIntC | Median correct response time (in ms) in in control condition (responses to solution C and D) |
Main Effect Extrinsic Load |
|
| accExtH | Proportion correct response in high extrinsic load condition (task 2) |
| corrExtH | Median correct response time (in ms) in high extrinsic load condition |
| accExtL | Proportion correct response in low extrinsic load condition (task 1) |
| corrExtL | Median correct response time (in ms) in low extrinsic load condition |
Low Extrinsic Load (Ll = Task1) |
|
| accIntHExtL | Proportion correct responses in high intrinsic load condition (responses to solution B) in task 1 (LL) |
| corrRTIntHExtL | Median correct response time (in ms) in high intrinsic load condition (responses to solution B) in task 1 (LL) |
| accIntLExtL | Proportion correct responses in low intrinsic load condition (responses to solution A) in task 1 (LL) |
| corrRTIntLExtL | Median correct response time (in ms) in low intrinsic load condition (responses to solution A) in task 1 (LL) |
| accIntCExtL | Proportion correct responses in control condition (responses to solution C and D) in task 1 (LL) |
| corrRTIntCExtL | Median correct response time (in ms) in in control condition (responses to solution C and D) in task 1 (LL) |
High Extrinsic Load (Hl = Task2) |
|
| accIntHExtH | Proportion correct responses in high intrinsic load condition (responses to solution B) in task 2 (HL) |
| corrRTIntHExtH | Median correct response time (in ms) in high intrinsic load condition (responses to solution B) in task 2 (HL) |
| accIntLExtH | Proportion correct responses in low intrinsic load condition (responses to solution A) in task 2 (HL) |
| corrRTIntLExtH | Median correct response time (in ms) in low intrinsic load condition (responses to solution A) in task 2 (HL) |
| accIntCExtH | Proportion correct responses in control condition (responses to solution C and D) in task 2 (HL) |
| corrRTIntCExtH | Median correct response time (in ms) in in control condition (responses to solution C and D) in task 2 (HL) |
Counts |
|
| countIntHExtL | Number of responses in Intrinsic High X Extrinsic Low |
| countValidRTIntHExtL | Number of VALID response times in Intrinsic High X Extrinsic Low (the difference is the number of removed latencies) |
| countIntLExtL | Number of responses in Intrinsic Low X Extrinsic Low |
| countValidRTIntLExtL | Number of VALID response times in Intrinsic Low X Extrinsic Low |
| countIntCExtL | Number of responses in Intrinsic Control X Extrinsic Low |
| countValidRTIntCExtL | Number of VALID response times in Intrinsic Control X Extrinsic Low |
| counTextHExtL | Number of responses in Intrinsic High X Extrinsic High |
| countValidRTextHExtL | Number of VALID response times in Intrinsic High X Extrinsic High |
| counTextLExtL | Number of responses in Intrinsic Low X Extrinsic High |
| countValidRTextLExtL | Number of VALID response times in Intrinsic Low X Extrinsic High |
| counTextCExtL | Number of responses in Intrinsic Control X Extrinsic High |
| countValidRTextCExtL | Number of VALID response times in Intrinsic Control X Extrinsic High |
File Name: unexpectedsolutionstask_raw*.iqdat
| Name | Description |
|---|---|
| build | Inquisit version number |
| computer.platform | Device platform: win | mac |ios | android |
| date | Date the session was run |
| time | Time the session was run |
| subject | Participant ID |
| group | Group number |
| session | Session number |
| blockCode | Name of the current block |
| blockNum | Number of the current block |
| trialCode | Name of the current trial |
| trialNum | Number of the current trial |
| practice | 0 = test (trials) 1 = practice (trials) |
| load | "LL" (low extrinsic load: task1) vs. "HL" (high extrinsic load: task2) |
| loadIndex | 1 = LL; 2 = HL |
| teamIndex | 1 = teamA; 2 = teamB, 3 = teamC, 4 = teamD |
| teamLabel | "A", "B", "C", "D" (the current team working) |
| roundPerTeam | Tracks each round per team (1-8) |
| selectedEquation | The image file of the current equation |
| equationIndex | The index of the current equation (refers to item.equations) |
| countSolutions | Tracks the number of solutions provided (1-5) => order of solution presentation |
| selectedCompetitor | The image file of the currently selected competitor (team member) |
| suggestedSolution | The solution currently suggested by the team member |
| solutionCat | "A" = obviously correct (low intrinsic load) "B" = surprisingly correct (high intrinsic load) "C" = incorrect (ctrl) |
| correctResp | Stores the currently correct response key |
| response | The response of participant (scancode of response button) |
| responseText | The label of the response key |
| correct | Correctness of response (1 = correct, 0 = error) |
| latency | Response latency (in ms); measured from: onset of solution |
| rt | The filtered response latency: if below or above the cut-offs, this variable will contain "invalid", otherwise it contains the response latency |
| winningTeam | Contains the teams with the highest score (LL task) |
| teamChampion | Contains the team member(s) with highest score for the current team (HL task) |
| overallChampion | Contains the student(s) with the highest scores across teams and tasks (LL + HL task) |
| stimulusItem | Presented stimuli in order of hard-coded order |
The procedure can be adjusted by setting the following parameters.
| Name | Description | Default |
|---|---|---|
Timing Parameters |
||
| equationPresentationDurationMS | The duration (in ms) that each equation is presented | 3000 |
| highLoadQuestionDurationMS | The duration (in ms) that the HL load question is presented | 1500 |
Responsekeys |
||
| leftResponseKey | The response key on the left side of the keyboard | "S" |
| rightResponseKey | The response key on the right side of the keyboard | "L" |
| correctResponseKey | parameters.rightResponseKey 'L' is assigned to indicate 'correct' | |
| incorrectResponseKey | parameters.leftResponseKey 'S' is assigned to indicate 'incorrect' | |
Performance Metrics Cleaning Parameters |
||
| minValidRtMS | The minimum valid RT that should be considered for RT analyses is 250ms | 250 |
| maxValidRTMS | The maximum valid RT that should be considered for RT analyses is 30000ms (per suggestion by Dr. Kondé) | 30000 |