Digital ink and differentiated subjective ratings for cognitive load measurement in middle childhood.

Publisher: Wiley-Blackwell Country of Publication: England NLM ID: 0370636 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 2044-8279 (Electronic) Linking ISSN: 00070998 NLM ISO Abbreviation: Br J Educ Psychol Subsets: MEDLINE

Publication: <2012-> : Chichester : Wiley-Blackwell
Original Publication: Edinburgh : Scottish Academic Press

Cognition* ; Ink*; Child ; Adult ; Humans ; Adolescent

Background: New methods are constantly being developed to adapt cognitive load measurement to different contexts. However, research on middle childhood students' cognitive load measurement is rare. Research indicates that the three cognitive load dimensions (intrinsic, extraneous, and germane) can be measured well in adults and teenagers using differentiated subjective rating instruments. Moreover, digital ink recorded by smartpens could serve as an indicator for cognitive load in adults.
Aims: With the present research, we aimed at investigating the relation between subjective cognitive load ratings, velocity and pressure measures recorded with a smartpen, and performance in standardized sketching tasks in middle childhood students.
Sample: Thirty-six children (age 7-12) participated at the university's laboratory.
Methods: The children performed two standardized sketching tasks, each in two versions. The induced intrinsic cognitive load or the extraneous cognitive load was varied between the versions. Digital ink was recorded while the children drew with a smartpen on real paper and after each task, they were asked to report their perceived intrinsic and extraneous cognitive load using a newly developed 5-item scale.
Results: Results indicated that cognitive load ratings as well as velocity and pressure measures were substantially related to the induced cognitive load and to performance in both sketching tasks. However, cognitive load ratings and smartpen measures were not substantially related.
Conclusions: Both subjective rating and digital ink hold potential for cognitive load and performance measurement. However, it is questionable whether they measure the exact same constructs.
(© 2023 The Authors. British Journal of Educational Psychology published by John Wiley & Sons Ltd on behalf of British Psychological Society.)

Anmarkrud, Ø., Andresen, A., & Bråten, I. (2019). Cognitive load and working memory in multimedia learning: Conceptual and measurement issues. Educational Psychologist, 54(2), 61-83. https://doi.org/10.1080/00461520.2018.1554484.
Ayres, P. (2006). Using subjective measures to detect variations of intrinsic cognitive load within problems. Learning and Instruction, 16(5), 389-400. https://doi.org/10.1016/j.learninstruc.2006.09.001.
Ayres, P., & Paas, F. (2012). Cognitive load theory: New directions and challenges. Applied Cognitive Psychology, 26(6), 827-832. https://doi.org/10.1002/acp.2882.
Badarna, M., Shimshoni, I., Luria, G., & Rosenblum, S. (2018). The importance of pen motion pattern groups for semi-automatic classification of handwriting into mental workload classes. Cognitive Computation, 10(2), 215-227. https://doi.org/10.1007/s12559-017-9520-2.
Barz, M., Altmeyer, K., Malone, S., Lauer, L., & Sonntag, D. (2020). Digital pen features predict task difficulty and user performance of cognitive tests. Paper presented at the proceedings of the 28th ACM conference on user modeling, adaptation and personalization, Genoa, Italy. https://doi.org/10.1145/3340631.3394839.
Berninger, V. (1991). Overview of “bridging the gap between developmental, neuropsychological, and cognitive approaches to reading”. Learning and Individual Differences, 3, 163-179. https://doi.org/10.1016/1041-6080(91)90006-M.
Cha, G. E., & Min, B. C. (2022). Correlation between unconscious mouse actions and human cognitive workload. In CHI conference on human factors in computing systems (pp. 1-7).
Chambers, C. T., & Craig, K. D. (1998). An intrusive impact of anchors in children's faces pain scales. Pain, 78(1), 27-37. https://doi.org/10.1016/S0304-3959(98)00112-2.
Chambers, C. T., & Johnston, C. (2002). Developmental differences in children's use of rating scales. Journal of Pediatric Psychology, 27(1), 27-36. https://doi.org/10.1093/jpepsy/27.1.27.
Chen, F., Zhou, J., Wang, Y., Yu, K., Arshad, S. Z., Khawaji, A., & Conway, D. (2016). Robust multimodal cognitive load measurement. Springer.
Cierniak, G., Scheiter, K., & Gerjets, P. (2009). Explaining the split-attention effect: Is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load? Computers in Human Behavior, 25(2), 315-324. https://doi.org/10.1016/j.chb.2008.12.020.
Drotár, P., Mekyska, J., Rektorová, I., Masarová, L., Smékal, Z., & Faundez-Zanuy, M. (2016). Evaluation of handwriting kinematics and pressure for differential diagnosis of Parkinson's disease. Artificial Intelligence in Medicine, 67, 39-46.
Eitel, A., Kühl, T., Scheiter, K., & Gerjets, P. (2014). Disfluency meets cognitive load in multimedia learning: Does harder-to-read mean better-to-understand? Applied Cognitive Psychology, 28(4), 488-501. https://doi.org/10.1002/acp.3004.
Eysink, T. H. S., de Jong, T., Berthold, K., Kolloffel, B., Opfermann, M., & Wouters, P. (2009). Learner performance in multimedia learning arrangements: An analysis across instructional approaches. American Educational Research Journal, 46(4), 1107-1149. https://doi.org/10.3102/0002831209340235.
Green, S. B. (1991). How many subjects does it take to do a regression analysis. Multivariate Behavioral Research, 26(3), 499-510.
Kalyuga, S. (2011). Cognitive load theory: How many types of load does it really need? Educational Psychology Review, 23(1), 1-19. https://doi.org/10.1007/s10648-010-9150-7.
Kirschner, P. A., Ayres, P., & Chandler, P. (2011). Contemporary cognitive load theory research: The good, the bad and the ugly. Computers in Human Behavior, 27(1), 99-105. https://doi.org/10.1016/j.chb.2010.06.025.
Klepsch, M., Schmitz, F., & Seufert, T. (2017). Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load. Frontiers in Psychology, 8, 1997. https://doi.org/10.3389/fpsyg.2017.01997.
Klepsch, M., & Seufert, T. (2020). Understanding instructional design effects by differentiated measurement of intrinsic, extraneous, and germane cognitive load. Instructional Science, 48(1), 45-77. https://doi.org/10.1007/s11251-020-09502-9.
Korbach, A., Brünken, R., & Park, B. (2017). Measurement of cognitive load in multimedia learning: A comparison of different objective measures. Instructional Science, 45(4), 515-536. https://doi.org/10.1007/s11251-017-9413-5.
Korbach, A., Ginns, P., Brünken, R., & Park, B. (2019). Should learners use their hands for learning? Results from an eye-tracking study. Journal of Computer Assisted Learning, 1-12, 102-113. https://doi.org/10.1111/jcal.12396.
Krieglstein, F., Beege, M., Rey, G. D., Ginns, P., Krell, M., & Schneider, S. (2022). A systematic meta-analysis of the reliability and validity of subjective cognitive load questionnaires in experimental multimedia learning research. Educational Psycholy Review, 34, 2485-2541. https://doi.org/10.1007/s10648-022-09683-4.
Leahy, W. (2018). Case studies in cognitive load measurement. In R. Z. Zheng (Ed.), Cognitive load measurement and application: A theoretical framework for meaningful research and practice (pp. 199-223). Routledge/Taylor & Francis Group.
Lee, H., & Mayer, R. E. (2015). Visual aids to learning in a second language: Adding redundant video to an audio lecture. Applied Cognitive Psychology, 29(3), 445-454. https://doi.org/10.1002/acp.3123.
Leppink, J., Paas, F., Van der Vleuten, C. P. M., Van Gog, T., & Van Merriënboer, J. J. G. (2013). Development of an instrument for measuring different types of cognitive load. Behavior Research Methods, 45(4), 1058-1072. https://doi.org/10.3758/s13428-013-0334-1.
Leppink, J., Paas, F., Van Gog, T., Van der Vleuten, C. P. M., & Van Merrienboer, J. J. G. (2014). Effects of pairs of problems and examples on task performance and different types of cognitive load. Learning and Instruction, 30(2014), 32-42. https://doi.org/10.1016/j.learninstruc.2013.12.001.
Lin, T., Xie, T., Chen, Y., & Tang, N. (2013). Automatic cognitive load evaluation using writing features: An exploratory study. International Journal of Industrial Ergonomics, 43(3), 210-217. https://doi.org/10.1007/s12559-015-9343-y.
Luria, G., & Rosenblum, S. (2012). A computerized multidimensional measurement of mental workload via handwriting analysis. Behavior Research Methods, 44(2), 575-586. https://doi.org/10.3758/s13428-011-0159-8.
Marsh, H. W. (1986). The bias of negatively worded items in rating scales for young children: A cognitive-developmental phenomenon. Developmental Psychology, 22(1), 37-49.
Maxwell, S. E. (2000). Sample size and multiple regression analysis. Psychological Methods, 5(4), 434-458.
Metcalfe, J., & Finn, B. (2013). Metacognition and control of study choice in children. Metacognition and Learning, 8(1), 19-46. https://doi.org/10.1007/s11409-013-9094-7.
Oviatt, S., Hang, K., Zhou, J., Yu, K., & Chen, F. (2018). Dynamic handwriting signal features predict domain expertise. ACM Transactions on Interactive Intelligent Systems (TiiS), 8(3), 1-21. https://doi.org/10.1145/3213309.
Oviatt, S., Lin, J., & Sriramulu, A. (2021). I know what you know: What hand movements reveal about domain expertise. ACM Transactions on Interactive Intelligent Systems, 11(1), 1-26. https://doi.org/10.1145/3423049.
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1-4. https://doi.org/10.1207/S15326985EP3801_1.
Park, B., Korbach, A., & Brünken, R. (2015). Do learner characteristics moderate the seductive-details-effect? A cognitive-load-study using eye-tracking. Journal of Educational Technology & Society, 18, 24-36.
Prange, A., & Sonntag, D. (2022). Modeling users' cognitive performance using digital pen features. Frontiers in Artificial Intelligence, 5, 787179. https://doi.org/10.3389/frai.2022.787179.
Ransdell, S., & Levy, C. M. (1999). Writing, reading, and speaking memory spans and the importance of resource flexibility. In M. Torrance & G. Jeffery (Eds.), The cognitive demands of writing: Processing capacity and working memory in text production (pp. 99-113). Amsterdam University Press.
Reitan, R. M. (1992). Trail making test: Manual for administration and scoring. Reitan Neuropsychology Laboratory.
Rheem, H., Verma, V., & Becker, D. V. (2018). Use of mouse-tracking method to measure cognitive load. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 62, pp. 1982-1986). SAGE Publications.
Rosenblum, S., Ben-Simhon, H. A., Meyer, S., & Gal, E. (2019). Predictors of handwriting performance among children with autism spectrum disorder. Research in Autism Spectrum Disorders, 60, 16-24.
Rosenblum, S., & Dror, G. (2016). Identifying developmental dysgraphia characteristics utilizing handwriting classification methods. IEEE Transactions on Human-Machine Systems, 47(2), 293-298. https://doi.org/10.1109/THMS.2016.2628799.
Rosenblum, S., & Livneh-Zirinski, M. (2008). Handwriting process and product characteristics of children diagnosed with developmental coordination disorder. Human Movement Science, 27(2), 200-214.
Rosenblum, S., & Luria, G. (2016). Applying a handwriting measurement model for capturing cognitive load implications through complex figure drawing. Cognitive Computation, 8(1), 69-77. https://doi.org/10.1007/s12559-015-9343-y.
Ruiz, N., Taib, R., Shi, Y., Choi, E., & Chen, F. (2007). Using pen input features as indices of cognitive load. In Proceedings of the 9th international conference on multimodal interfaces (pp. 315-318). https://doi.org/10.1145/1322192.1322246.
Schaaff, K., Degen, R., Adler, N., & Adam, M. T. (2012). Measuring affect using a standard mouse device. Biomedical Engineering, 57, 761-764.
Schmeck, A., Opfermann, M., van Gog, T., Paas, F., & Leutner, D. (2015). Measuring cognitive load with subjective rating scales during problem solving: Differences between immediate and delayed ratings. Instructional Science, 43(1), 93-114. https://doi.org/10.1007/s11251-014-9328-3.
Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47(10), 1173-1181.
Schrader, C., & Kalyuga, S. (2020). Linking students' emotions to engagement and writing performance when learning Japanese letters with a pen-based tablet: An investigation based on individual pen pressure parameters. International Journal of Human-Computer Studies, 135, 102374.
Schroeder, N. L., & Cenkci, A. T. (2018). Spatial contiguity and spatial Split-attention effects in multimedia learning environments: A meta-analysis. Educational Psychology Review, 30(3), 679-701. https://doi.org/10.1007/s10648-018-9435-9.
Sepp, S. (2019). Meaningful hand gestures for learning with touch-based I.C.T. Doctor of Philosophy thesis, School of Education, University of Wollongong. https://ro.uow.edu.au/theses1/903.
Skulmowski, A., & Rey, G. D. (2020). Subjective cognitive load surveys lead to divergent results for interactive learning media. Human Behavior and Emerging Technologies, 2(2), 149-157. https://doi.org/10.1002/hbe2.184.
Smits-Engelsman, B. C. M., & Van Galen, G. P. (1997). Dysgraphia in children: Lasting psychomotor deficiency or transient developmental delay? Journal of Experimental Child Psychology, 67, 164-184. https://doi.org/10.1006/jecp.1997.2400.
Snijders, P., Tellegen, P. J., & Laros, J. A. (2005). SON-R 5½-17. Snijders-Oomen-Non-Verbal intelligence test (3rd ed.). Hogrefe.
Stebner, F., Kühl, T., Höffler, T. N., Wirth, J., & Ayres, P. (2017). The role of process information in narrations while learning with animations and static pictures. Computers & Education, 104, 34-48. https://doi.org/10.1016/j.compedu.2016.11.001.
Sweller, J. (2010). Element interactivity and intrinsic, extraneous, and germane cognitive load. Educational Psychology Review, 22, 123-138. https://doi.org/10.1007/s10648-010-9128-5.
Sweller, J., Ayres, P., & Kalyuga, S. (2011). Intrinsic and extraneous cognitive load. In Cognitive load theory. Explorations in the learning sciences, instructional systems and performance technologies (Vol. 1). Springer. https://doi.org/10.1007/978-1-4419-8126-4_5.
Sweller, J., van Merriënboer, J., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251-296. https://doi.org/10.1023/A:1022193728205.
Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31(2), 261-292. https://doi.org/10.1007/s10648-019-09465-5.
Tang, M., Ginns, P., & Jacobson, M. J. (2019). Tracing enhances recall and transfer of knowledge of the water cycle. Educational Psychology Review, 31, 439-455. https://doi.org/10.1007/s10648-019-09466-4.
Van Gemmert, A. W. A., & Van Galen, G. P. (1998). Auditory stress effects on preparation and execution of graphical aiming: A test of the neuromotor noise concept. Acta Psychologica, 98, 81-101. https://doi.org/10.1016/S0001-6918(97)00049-8.
Willems, D., & Niels, R. (2008). Definitions for features used in online pen gesture recognition. NICI, Radboud University Nijmegen.
Witte, T. E., Haase, H., & Schwarz, J. (2021). Measuring cognitive load for adaptive instructional systems by using a pressure sensitive computer mouse. In International conference on human-computer interaction (pp. 209-218). Springer, Cham.
Wu, Z., Lin, T., & Tang, N. (2016). Explore the use of handwriting information and machine learning techniques in evaluating mental workload. International Journal of Technology and Human Interaction (IJTHI), 12(3), 18-32. https://doi.org/10.4018/IJTHI.2016070102.
Yu, K., Epps, J., & Chen, F. (2011). Cognitive load evaluation of handwriting using stroke-level features. In Proceedings of the 16th international conference on intelligent user interfaces (pp. 423-426). https://doi.org/10.1145/1943403.1943481.
Yung, H. I., & Paas, F. (2015). Effects of computer-based visual representation on mathematics learning and cognitive load. Journal of Educational Technology & Society, 18(4), 70-77.
Zu, T., Hutson, J., Loschky, L. C., & Rebello, N. S. (2020). Using eye movements to measure intrinsic, extraneous, and germane load in a multimedia learning environment. Journal of Educational Psychology, 112(7), 1338-1352. https://doi.org/10.1037/edu0000441.

01JD1811 Bundesministerium für Bildung und Forschung

Keywords: assessment; cognitive load measurement; extraneous load; intrinsic load; primary school; smartpen

Date Created: 20230326 Date Completed: 20230802 Latest Revision: 20230802

20231215

10.1111/bjep.12595

36967475