In order to see if the aims of the project were met or not, several evaluation methods were examined and conducted, mainly by the Evaluation Teams of the project. The evaluation methods included (1) questionnaires administered prior to and after the evaluation session to the participants from the target audience, (2) a protocol analysis to observe and analyze the process of using the courseware, and (3) a game-design expert interview to try to clarify the rules of judgments in the course of game design, applied to the development of this courseware.
Despite of a review of prior research studies, the project group did not feel comfortable as to how to go about conducting the evaluation. It should be noted that the methods of evaluation per se were to be examined their validity through the usage in this study. Brief descriptions of the methodologies and the results are as follows:
Pre and post session questionnaires were administered to all of the 39 participants (mostly college students) of the evaluation study. Pre-session questionnaire asked participants‘ biographical data, experiences of learning on computer, characteristics of the most memorable computer courseware using 17 5-point Likert scale items, and TV game experiences and impression of the most memorable TV game, with 18 Likert items. Post-session questionnaire asked English learning experiences, and reactions, opinions and feelings about the courseware "The Secret of Aunt Mariko", using a total of 81 Likert items.
Among the items concerning the English learning aspects of this courseware, two items on participants‘ subjective judgment of improvement of listening comprehension (both the contents and the words of narrations and questions) were rated higher than other items (such as remembering dialog patterns). A comparison between game aspects of this courseware and impression of the most memorable TV game revealed that both were rated similarly, suggesting that this courseware was considered to be one of the best "games" they ever played.
Many of the participants expressed their continuing motivation to spend more time on this courseware, to recommend this courseware to their friends, and to wish if it could be used in their schools. This verbal commitment was supported by frequent observations of voluntarily choosing to remain using the courseware after the evaluation session when they were told that they could do so.
Protocol analysis, or discourse analysis, refers to recording and categorizing learners‘ conversation while studying courseware (Yoden, 1991). In the process of the formative evaluation of this courseware, some of the participants were asked to use the courseware in a pair, by orally exchanging feelings and ideas before making any choices. The evaluation process was videotaped, which was to be analyzed into sentences and classified using predetermined utterance categories. The amounts of utterance in each category were compared with each other to see the kinds of mental activities frequently occurred during the learning process.
The results from an analysis of two pairs of participants indicated that (1) the operation of the courseware was clearly understood, which was judged by few conversation concerning how to proceed to the next screen; (2) the courseware was affectively stimulating, judged by frequent utterance in affective nature, such as curiosity, joyfulness, and surprises; (3) the courseware required the users to remember the contextual information to make choices, and (4) the courseware required them to concentrate on what the characters said in English, also judged by frequent conversations making sure contextual information and repeating questions in English for themselves.
This method of evaluation was adopted because it captures the process of learning, unlike the other forms of tests or questionnaires collecting information at either the entrance or the exit of leaning activities. It brought useful information in a quantitative form as to how the participants reacted to the courseware. It is noted, however, that the time consuming processes of analyzing all the utterance for over one-hour sessions and then manually categorizing them must be shortened without loosing validity of the data. It should also be noted that possibility of affecting users motivation by employing this evaluation method (i.e., to study in a pair being directed to overtly respond) deserves an attention, especially when enhancing motivation is the goal of the project .
The Theory Team conducted an interview with the chief engineer of Development Team, to try to extract his expertise in game design. Because it was mostly due to his expertise if the product was interesting and appealing, and because the theory could not play a prescriptive role in this project, it seemed a natural step to conduct an interview so as his decisions to be stated in as an objective form as possible. Borrowing from the method of structured interviews in the field of knowledge engineering, a three-hour interview was conducted using predetermined questions and free discussion with an access to the courseware. A part of the results from the interview is shown in Table 3.
---
Table 3. Insert about here
---
This paper described a case study of motivational design applied to a development project of multimedia courseware, "The Secret of Aunt Mariko". The usefulness of the ARCS model and motivational design was demonstrated, at least in a descriptive sense. It helped to categorize motivational attributes of the courseware, which were originated from a game designer‘s rules of thumb, unconsciously from his own experiences in game design. The next step of project‘s theoretical concern is to narrow the focus of motivational enhancement, in order to position the future product in light of the ARCS four categories.
Although some innovative evaluation methodologies provided indications supporting project‘s success, further investigations are in order, as to how to improve evaluation process per se. We still feel uncertain how much or how little our success truly was, as well as the reason why.
As Keller (1987) has pointed out about a systematic motivational design, we should "not expect it to be completely mechanical or algorithmic; it still requires judgment and benefits from experience, intuition and creativity (p. 1-2)." It is apparent from this study that we can learn many things from joint projects with entertainment experts, if we are to make our own instruction more appealing. It can only be said at this point that investigations in motivational design are in order in various settings of educational research.
联系客服
