Hostname: page-component-745bb68f8f-f46jp Total loading time: 0 Render date: 2025-02-06T10:14:12.057Z Has data issue: false hasContentIssue false
  • English
  • Français

Latent Feature Extraction for Process Data via Multidimensional Scaling

Published online by Cambridge University Press:  01 January 2025

Xueying Tang ,
Zhi Wang ,
Qiwei He ,
Jingchen Liu Open the ORCID record for Jingchen Liu [Opens in a new window]  and
Zhiliang Ying
Xueying Tang
Affiliation:
University of Arizona
Zhi Wang
Affiliation:
Columbia University
Qiwei He
Affiliation:
Educational Testing Service
Jingchen Liu*
Affiliation:
Columbia University
Zhiliang Ying
Affiliation:
Columbia University
*
Correspondence should be made to Jingchen Liu, Department of Statistics, Columbia University,1255 Amsterdam Ave, New York, NY10027, USA. Email: jcliu@stat.columbia.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Computer-based interactive items have become prevalent in recent educational assessments. In such items, detailed human–computer interactive process, known as response process, is recorded in a log file. The recorded response processes provide great opportunities to understand individuals’ problem solving processes. However, difficulties exist in analyzing these data as they are high-dimensional sequences in a nonstandard format. This paper aims at extracting useful information from response processes. In particular, we consider an exploratory analysis that extracts latent variables from process data through a multidimensional scaling framework. A dissimilarity measure is described to quantify the discrepancy between two response processes. The proposed method is applied to both simulated data and real process data from 14 PSTRE items in PIAAC 2012. A prediction procedure is used to examine the information contained in the extracted latent variables. We find that the extracted latent variables preserve a substantial amount of information in the process and have reasonable interpretability. We also empirically prove that process data contains more information than classic binary item responses in terms of out-of-sample prediction of many variables.

Keywords

response processlog file analysisPIAACmultidimensional scaling
Type
Theory and Methods
Information
Psychometrika , Volume 85 , Issue 2 , June 2020 , pp. 378 - 397
Copyright
Copyright © 2020 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Electronic supplementary material The online version of this article (https://doi.org/10.1007/s11336-020-09708-3) contains supplementary material, which is available to authorized users.

References

Borg, I., & Groenen, P. J. (2005). Modern multidimensional scaling: Theory and applications. New York, NY: Springer. Google Scholar
Gómez-Alonso, C., & Valls, A. Torra, V., & Narukawa, Y. (2008). A similarity measure for sequences of categorical data based on the ordering of common elements. Modeling decisions for artificial intelligence, Berlin: Springer. 134– 145. CrossRefGoogle Scholar
Greiff, S., Niepel, C., Scherer, R., & Martin, R. (2016). Understanding students’ performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files. Computers in Human Behavior, 61, 36– 46. CrossRefGoogle Scholar
He, Q., & von Davier, M. (2015). Identifying feature sequences from process data in problem-solving items with n-grams. In van der Ark, L. A., Bolt, D. M., Wang, W.-C., Douglas, J. A., & Chow, S.-M.(Eds.), Quantitative psychology research (pp. 173–190). Cham: Springer. https://doi.org/10.1007/978-3-319-19977-1_13. CrossRefGoogle Scholar
He, Q., & von Davier, M. Rosen, Y., Ferrara, S., & Mosharraf, M. (2016). Analyzing process data from problem-solving items with n-grams: Insights from a computer-based large-scale assessment. Handbook of research on technology tools for real-world skill development, Hershey, PA: Information Science Reference. 749– 776. Google Scholar
Herranz, J., Nin, J., & Sole, M. (2011). Optimal symbol alignment distance: A new distance for sequences of symbols. IEEE Transactions on Knowledge and Data Engineering, 23 (10), 1541– 1554. CrossRefGoogle Scholar
Karni, E. S., & Levin, J. (1972). The use of smallest space analysis in studying scale structure: An application to the California psychological inventory. Journal of Applied Psychology, 56 (4), 341– CrossRefGoogle Scholar
Klein Entink, R., Fox, J. -P., & van der Linden, W. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, (1), 21– 20037635 CrossRefGoogle Scholar
Kroehne, U., & Goldhammer, F. (2018). How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items. Behaviormetrika, 45, (2), 527– 563. CrossRefGoogle Scholar
Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10, (8), 707– 710. Google Scholar
Lord, F. M. (1980). Applications of item response theory to practical testing problems, New York, NY: Routledge. Google Scholar
Meyer, E. M., & Reynolds, M. R. (2018). Scores in space: Multidimensional scaling of the WISC-V. Journal of Psychoeducational Assessment, 36, (6), 562– 575. CrossRefGoogle Scholar
Qian, H., Staniewska, D., Reckase, M., & Woo, A. (2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35, (1), 38– 47. CrossRefGoogle Scholar
Robbins, H., & Monro, S. (1951). A stochastic approximation method. The Annals of Mathematical Statistics, 22, (3), 400– 407. CrossRefGoogle Scholar
Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications, New York, NY: Guilford Press. Google Scholar
Shoben, E. J. (1983). Applications of multidimensional scaling in cognitive psychology. Applied Psychological Measurement, 7, (4), 473– 490. CrossRefGoogle Scholar
Skager, R. W., Schultz, C. B., & Klein, S. P. (1966). The multidimensional scaling of a set of artistic drawings: Perceived structure and scale correlates. Multivariate Behavioral Research, 1, (4), 425– 436. CrossRefGoogle ScholarPubMed
Subkoviak, M. J. (1975). The use of multidimensional scaling in educational research. Review of Educational Research, 45, (3), 387– 423. CrossRefGoogle Scholar
Takane, Y. (2006). 11 applications of multidimensional scaling in psychometrics. Handbook of Statistics, 26, 359– 400. CrossRefGoogle Scholar
van der Linden, W. J. (2008). Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics, 33, (1), 5– 20. CrossRefGoogle Scholar
Wang, S., Zhang, S., Douglas, J., & Culpepper, S. (2018). Using response times to assess learning progress: A joint model for responses and response times. Measurement: Interdisciplinary Research and Perspectives, 16, (1), 45– 58. Google Scholar
Zhan, P., Jiao, H., & Liao, D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71, (2), 262– 286. CrossRefGoogle ScholarPubMed
Supplementary material: File

Tang et al. supplementary material

Tang et al. supplementary material 1
Download Tang et al. supplementary material(File)
File 514 Bytes
Supplementary material: File

Tang et al. supplementary material

Tang et al. supplementary material 2
Download Tang et al. supplementary material(File)
File 3.5 KB
Supplementary material: File

Tang et al. supplementary material

Tang et al. supplementary material 3
Download Tang et al. supplementary material(File)
File 3 KB