چکیده:
یکی از نوآورانه‌ترین ابزارهای محققان برای بهبود آموزش و ارزشیابی به‌خصوص درمورد آموزش‌های سرنوشت‌ساز که حاصل اتحاد روان‌شناسی شناختی و سنجش آموزشی است استفاده از رویکردهای تشخیصی شناختی است، اما متاسفانه این رویکردها مورد توجه و استفادۀ اندک پژوهشگران حیطۀ آموزش و ارزشیابی زبان، به‌خصوص دستور زبان، در ایران قرار گرفته است. از مسائل اساسی در جریان سنجش تشخیصی شناختی، تعیین مهارت‌ها و ریزمهارت‌های شناختی لازم برای پاسخ به پرسش‌ها و همچنین، انتخاب مدلی بهینه است. اهداف اصلی پژوهش کیفی و کمی حاضر تعیین این مهارت‌های شناختی از طریق تدوین ماتریس کیو، تعیین مدل بهینۀ تشخیصی شناختی (CDM) برای آزمون دستور زبان کنکور کارشناسی ارشد مجموعۀ زبان انگلیسی و درنهایت، شناسایی نقاط قوت و ضعف آزمودنی‌هاست که 5000 نفر از شرکت‌کنندگان سال 1396 بودند. یافته‌های حاصل از بررسی ادبیات موجود، ارزیابی متخصصان و پروتکل‌های تفکر با صدای بلند در فاز تدوین ماتریس کیو نشان می‌دهد، هر کدام از پرسش‌های آزمون دستور زبان به بررسی دو تا چهار مهارت از شش خرده‌مهارت تصریف زمان فعل، کاربرد فعل صحیح، کاربرد عبارات اصطلاحی، کاربرد تعدیل‌کننده‌ها، توافق و کاربرد حروف اضافه می‌پردازند. بررسی مدل‌های DINA, DINO , G-DINA از میان مدل‌های تشخیصی شناختی موجود نشان داد که مدل G-DINA با داده‌ها برازش مطلوب‌تری دارد و درنتیجه، تحلیل‌های تکمیلی بر روی عملکردهای این مدل انجام شد. درنهایت، با توجه به تحلیل عملکرد آزمون‌دهندگان مسلط و غیرمسلط، کاربردها و پیشنهاداتی برای آموزش و سنجش زبان برای آزمون‌های سرنوشت‌ساز همچون کنکور کارشناسی ارشد و همچنین، پژوهش‌های مشابه ارائه شده است.
One of the most innovative tools for researchers in order to improve the quality of education and assessment is the application of Cognitive Diagnostic Approaches (CDA) which is the result of the unification of cognitive psychology and educational measurement; unfortunately, they are scarcely utilized by (grammar) language education and assessment researchers (in Iran). Among the most important decisions to be made in the process of cognitive diagnostic assessment are determining the (sub-)skills required to respond correctly to each question as well as choosing an optimal cognitive diagnostic model. The present qualitative and quantitative study aims to develop a Q-matrix in order to identify such cognitive (sub-)skills, to determine the optimal cognitive-diagnostic model (CDM) for the grammar section of MA entrance examination for English majors, and to pinpoint mastery and non mastery states of the examinees who were 5000 MA entrance participants. The findings from the literature review, informants’ and experts’ evaluations, and think-aloud protocols in the Q-matrix development phase revealed that each MA examination grammar item taps into between two to four of the six attributes of verb tense, correct verb usage, idiomatic expressions, modifiers, agreement, and preposition. Evaluation of three alternative models [i.e. the Deterministic-Input, Noisy-And (DINA), Deterministic Input, Noisy-Or-gate (DINO), and Generalized DINA (GDINA)] from among the existing cognitive diagnostic models indicated that G-DINA was the best fit for the Grammar data. Considering the performance of master and non-master participants, the study concludes with suggestions, implications, and applications of the findings for high-stakes language education and testing
1. Introduction
Cognitive diagnostic assessment (CDA) is designed to measure specific cognitive skills of students, so as to provide information about their cognitive strengths and weaknesses (Leighton & Gierl, 2007). Previous research on CDA in language testing mostly focused on reading comprehension and listening sections, whereas less attention has been paid to grammar. Moreover, in most of the previous studies (Baghaie & Ravand, 2015; Clarck, 2013; Jang, 2009; Lee & Sawaki, 2009; Ravand, 2015; Ravand, Barati & Widhiarso, 2013) just a specified cognitive diagnostic model (CDM) was fitted to the language test data while searching for an optimum CDM was generally overlooked.
Given the importance of high-stakes tests such as university entrance examinations, the current research aims to apply CDA to an Iranian high-stakes English grammar test to specify the underlying skills required to answer the test items correctly; furthermore, it intends to detect strengths and weaknesses of the students based on the identified skills. In doing so, searching for an optimum CDM was adopted to find the best fitting model to the second language grammar test data.
2. Literature Review
Cognitive diagnostic models are confirmatory multidimensional latent variable models with complex structures. These models let researchers propose exact hypotheses about the nature of cognitive processes that students use in response to test items (Rupp et al., 2010). Three classes of saturated, compensatory, and non-compensatory models of CDA are available to researchers. That is, a- the saturated model titled generalized deterministic inputs, noisy “and” gate (G-DINA), b-compensatory models (e.g., the Deterministic Input Noisy Output “OR” gate (DINO) and the additive CDM (ACDM)) which allow for other skills to contribute to the chance of responding to an item correctly, and c- the non-compensatory models (e.g., The Deterministic Input Noisy Output “AND” gate (DINA) and the reduced reparametrized unified model (RRUM)), where possessing all cognitive skills is necessary to answer a test item correctly (de la Torre, 2011).
In recent years, many CDM studies were conducted on data from different fields of studies including language assessment (Alavai & Ranjbaran, 2018; Alderson et al., 2015; Baghaie & Ravand, 2015; Clarck, 2013; Jang, 2009; Li, 2011; Lee & Sawaki, 2009; Minaei et al., 2014; Moghadam et al., 2015; Park & Cho, 2011; Ranjbaran & Alavi, 2016; Ravand, 2015; Ravand et al., 2013; Yie, 2016, 2017). Although Park and Cho (2011) applied CDA on the English grammar for Korean EFL learners, only Yie (2017) searched for an Optimal Cognitive diagnostic model in a second language grammar test data. The latter study is very similar to the present study in many respects but the eventual cognitive diagnostic model.
3. Methodology
A retrofitting approach (Jang, 2009) to CDA was adopted to reach the cognitive diagnostic model. In doing so, at the first stage of the CDM, a Q matrix (de la Torre, 2011) was qualitatively developed based on the findings from the literature review, the viewpoints of an English language expert panel, and the cognitive processes extracted from college students' think-aloud protocols. The resulting Q-matrix provided all of the required skills needed to answer all of the grammar test items of the Iranian MA entrance examination for English majors. Then, DINA, DINO, and the G-DINA models were empirically fitted to the grammar test data of 5000 participants through the CDM package of R (George et al., 2016).
4. Conclusion
The findings of this study showed that the saturated G-DINA model was the best fitting model for the grammar data. The compensatory DINO model also fitted the data, yet the non-compensatory DINA model did not fit the grammar item response data based on the absolute model fit indices.
In line with the results of Park and Cho (2011), this study also confirms that the six underlying skills including 1-verb tense, 2- correct verb usage, 3- idiomatic expressions, 4- modifiers, 5- agreement, and 6- preposition encompass almost all of the required grammar skills. Moreover, verb tense skill was identified as the weakness of the students, while idiomatic expressions skill was a strength point. Altogether, in a second language context, it seems that even the students majoring in the English language do not master all of the required skills of grammar. The study concludes with suggestions, implications, and applications of the findings for high-stakes language education and testing
(
, 
, 
)
