Scott W. Bonham, Aaron Titus, Robert J. Beichner and Larry Martinto be published in Journal of Research on Computing in Education, Fall 2000Abstract
The growth of the Internet, and in particular the World Wide Web, is already influencing the wayscience is taught and will undoubtedly do so to greater extent in the future. One important facet of this isthe development of web-based assessment and testing systems. These systems also provides a valuablenew tool to the Education Research community; a tool that combines the ability of multiple-choicediagnostic tests to handle large numbers of subjects with some of the greater flexibility and additionalinformation that other methods offer. On the other hand, some of the particular strengths of this tool alsogive rise to some unique disadvantages. In this paper the strengths and weaknesses for education researchare discussed, and some suggestions for its use are presented.
comprehensive overviews of the use of suchsystems can be found elsewhere (Goldberg &Salari, 1997; Titus, Martin, & Beichner, 1998;Zhao, 1998). The various systems listed in theappendix differ greatly among themselves,ranging from interactive course material that anindividual professor has created for his studentsto comprehensive systems incorporating
databases, chat rooms, and multimedia. Someof them are systems developed by groups withinuniversities for in-house use, while other
university-developed systems are available foruse by others. Commercial offerings come fromsome companies where the system is their mainproduct, and from other companies—inparticular textbook publishers—where thesystem is provided as a service to theircustomers. This paper focuses on
comprehensive systems in which password-authenticated homework web pages are deliveredand graded by a central server.
The system developed and used at NorthCarolina State University (NCSU) is described atthis point as an example of a comprehensiveWATS (NCSU, 1998). WebAssign uses a
Sybase database in which homework questionsare stored, assignments are organized, andgrades are recorded. Class rolls and studentinformation are downloaded from the
University’s registration and records database.The majority of the questions it contains are
standard problems from various physics books—currently eleven different texts are represented inthe database. The database includes surveyquestions and questions from nationally normedphysics diagnostic tests (Engelhardt, 1997;
Introduction
The explosive growth of the Internet ismaking available radical new means of
communication that affect such diverse areas asbusiness, entertainment and education. Whileolder methods of accomplishing tasks continueto be used, the Internet offers unique advantages.In areas of education it offers a medium that hasthe potential to be more responsive to students,to encourage greater participation in their ownlearning, and to give greater access to differentsources of information than traditional methodsoffer (Brooks, 1997; de La Beaujardiere et al.,1997; Khan, 1997). In this paper the focus is onthe marriage of computerized testing systemswith the World Wide Web to produce web-basedassessment and testing systems (WATS). Thesesystems are on the verge of becoming widelyadopted at the university level in physics andother courses. A list of many of the WATSknown to the authors may be found in theappendix. At least one of these systems,WebAssign, is also currently used in severaldozen secondary schools. In the past, new typesof technology, such as audio and video
recording, have given rise to new methods ofconducting education research. In the same way,the coming wide spread use of WATS will openup possibilities for educational research.Surprisingly, this area has not received muchattention in the literature (Campbell, 1997).The adoption of WATS for student work inphysics is being driven in part by the promise ofreduced grading load and the provision of moreimmediate feedback to students. More
Halloun & Hestenes, 1996; Redish, Saul, &Steinberg, 1998). WebAssign is also being usedfor courses in computer science, mathematics,business and statistics. Questions present
students with buttons for multiple choice (bothexclusive and non-exclusive format), text boxesfor numerical answers and fill-in-the-blank
questions, and/or text area boxes for free-format(essay) questions. The instructor constructsindividual assignments by selecting from theavailable questions, setting starting and endingtimes for the assignment availability, and settinggrading and feedback options. The students logon and are authenticated by the same usernameand password as their campus computer account.They receive a page generated by Perl script,which lists the currently available assignmentsand a list of the past-due assignments for whichthey may now view the answer key. Upon
selecting an assignment, the system generates anHTML document containing the questions. Anexample assignment is shown in Figure 1. Thestudents enter their answers in the form’s boxesor select among the buttons and submit the pagefor grading. The multiple choice and numericalanswers are automatically graded and studentsquickly receive another page with the incorrectanswers marked wrong. The answers are inboxes where they can be changed and—if
permitted by the instructor—may be resubmittedan unlimited number of times until theassignment deadline. The free response
questions are collected for later grading by theinstructor or other designated persons. Studentsmay view their current homework grades at theirconvenience. The instructor may view students’overall scores as well as individual responses.They may also perform administrative tasks fortheir class such as adding students to the rosterand granting individual students extensions onhomework assignments. Question formatting isin HTML, so pictures, audio, video, Java applets,and links to other sites may be included.
Another feature of the system is the ability to userandomized numbers so that each studentreceives a different set of numbers for a givenproblem. In Figure 1 the problem has
randomized numbers for the velocity and time,as can be seen by the lighter color of these
numbers (on the screen the randomized numbersappear in red). In the physics courses at NCSU,students receive short assignments each classday.
Figure 1: A sample assignment page fromWebAssign. This problem is from (Halliday,
Resnick, & Walker, 1997), Copyright © 1997, JohnWiley & Sons, Inc. Reprinted by permission.
Education research via WATS may be
conducted with different degrees of interventionin the learning process. The approach with leastintervention is to look at data generated bystudents completing regular assignments.Essentially, this is passive observation with aWATS. In one sense, this is a limited type ofobservation; the researcher will only have
information about students’ interactions with theWATS. However, the computer log files willhave a complete, accurate record of this type ofinformation for all students and all assignments,so from another perspective this is a verycomprehensive method of observation. Asecond approach of conducting research is
through the inclusion of special exercises in theassignments, which is a limited type of
intervention, since this will appear to the subjectsas part of the regular work. These could includeitems from standard diagnostic tests or otherquestions to probe students’ knowledge, andcould be either given as separate assignments orembedded in the regular course work. A thirdapproach is through the use of surveys and self-reported data, which represents the greatest levelof intervention by the researcher. For example,after working a problem the student could beasked to report in a text box how he/she went
2
Homework relationship180160140Total scoreattempt an assignment two to three times.
Significant numbers of submissions beyond thishas very little affect on homework scores beyondreducing the numbers of students with lowscores.
An investigation via the second approach atNCSU has investigated the influence ofmultimedia on students’ success at solving
physics problems. “Video-enhanced” problemswere created by supplying a video clip depictingthe motion described in a traditional physicsproblem with the exercise. The WATS was usedboth to deliver these problems and to gather dataon students’ performance on the video-enhancedproblems compared to the same problems thatonly had a still picture. Including video withtraditional problems significantly affectedstudents’ performance on some types of
problems but not others (Titus, 1998). This workwas followed up by the creation of “multimedia-focused” problems in which informationnecessary to solve the problem is embeddedwithin a Java applet animation (Christian, 1996-1998; Christian & Titus, 1998). Using theanimation, students had to make relevantmeasurements in order to solve the problem,incorporating elements of a laboratoryexperiment. The WATS was used to
automatically deliver the multimedia-focusedproblems to students randomly selected to be thetreatment group and traditional problems to thecontrol group. As a follow up to this study,students were asked to complete an on-linesurvey about their experiences with the
multimedia focused problems—an example ofthe third method described above (Titus, 1998).WATS have the potential to bring significantchanges to courses in which they are used. Withtheir power to collect and process data,
computers and networks are used to mediate
Computerized data collectionflexibility in how information may be presentedthe ways in which students may respondindividualization of the materialevery detail of the subject’s interactions recordeddecreased contactvolume of dataevery detail of the subject’s interactions recordeddecreased contactvolume of data1201008060402000100200300400500600Total submissionsFigure 2: Scatter plot of total homework scores versustotal number of submissions for students in the firstsemester engineering physics course. The vertical linerepresents (approximately) the minimum number ofsubmissions to have attempted every assignment.
about solving it. The flexibility of the Webenvironment means that there are many
combinations and permutations of these meansthat could conceivably be used for research.As an example, Figure 2 presents a resultfrom an investigation via the first researchmethod. This is a scatter plot of student
homework scores vs. the number of submissionsfor the entire semester of the introductory
engineering physics course. Each correct answerwas worth one point. The vertical line representsthe minimum number of submissions to haveattempted every assignment. (This number—40—is an approximation; there was somevariation in number of assignments betweensections.) Dots to the right of this representstudents who made enough submissions to haveattempted every assignment. The relationship issomewhat surprising; the main effect of
increased submissions is elimination of the low-scoring students. Most students, on average,
Internet settingflexibility in time and locationalready used in the class being studiedtightly integrated into a regular courseProsConsgreater variation of the research environmenttightly integrated into a regular coursecomplications due to the technologyTable 1: Summary of implications for the use of WATS as an education research tool from its use of the Internetand of computerized data collection.
3
instructor-student interaction in a very flexibleelectronic environment (Khan, 1997). Thesesame aspects also provide a new tool for
education research, both for evaluating this newmethod of teaching and for broader issues. Thesuitability of WATS for education researcharises naturally from the ability to individuallytailor material and responses for each studentand its ability to keep track of every activity ofthat student. In the following sections many ofthe issues related to the use of web homeworksystems for education research are discussed.These issues can be separated into two broadcategories. First discussed will be how use ofthe Internet can significantly change the settingin which the subjects work, the type of workdone, and related implications. Second is adiscussion of the data that can–and cannot–beobtained by this method, the type of data and itsquantity. Following this, the relationship
between this new tool and other existing researchtools is briefly discussed and suggestions aboutusing this tool are made.
The settings of the work
A significant advantage for educationresearch that WATS offer is the removal ofmany restrictions on studies. There is muchgreater flexibility of when and where thesubjects may participate, accompanied by acorresponding loss of control of some elementsof the study. In general, the students will alreadybe using the system for their class work and theinvestigation will be integrated with this to acertain extent. However, the heavy reliance ontechnology also introduces potentialcomplications.
A major characteristic of anything done viathe World Wide Web is flexibility in time andlocation. Information can be sent and retrievedfrom any location in the world at any time ofnight or day as long as there is access to theInternet. What this means for the educationresearcher is that his or her subjects can, inprinciple, work in any location and at any timethat is the most convenient. The investigator andthe subjects need not be in the same room oreven on the same continent. The minimumrequirements for a subject to participate in agiven location—a computer with Internet
access—have become widely available (NSTA,1998), which reduces greatly the extent and cost
of carrying out large studies in multiple
locations. The subjects also have the possibilityof working in a familiar setting—such as athome—as opposed to an unfamiliar laboratory.As can be seen, this tool offers a valuable
method of looking at realistic learning situations.Of course this flexibility comes with a cost.The ability of the subjects to work anywhere andanytime necessarily introduces a greatervariation of the research environment. Somesubjects will be alone and others may collaboratewith fellow students. Even if instructions aregiven about this, they might be difficult toenforce. Some may be in a quiet place whileothers may be surrounded by many distractions.Some may work on it late at night when tiredwhile others log in first thing in the morning.Some students will be very diligent about thework, while others may play at it betweenbrowsing through web ‘zines and the latestDilbert cartoons. If this is too great of aproblem, it can be addressed in part by
restricting when and where subjects may work.Subjects could be required to work in certainlocations, perhaps by restricting the main serverto accept requests only from certain IPaddresses, such as a specified computer
laboratory. To further increase control, thesedesignated locations could be proctored toprevent collaboration and to keep people on-task. Security could be addressed by posting aphoto of the student logged in at the top of thescreen, which would make it easy to detect
cheaters. The server could also restrict the timesat which it will accept submissions. As may beeseen, WATS offer different levels of flexibilityversus control for educational research.An important aspect of using a WATS is thatusually it would be already used in the classbeing studied. Under most circumstances, theinvestment to set up a WATS and for subjects tolearn it would be impractical were it only usedfor education research. However, once the
system has been introduced for class use and thestudents have become familiar with it, theadditional cost to use it for research is small.There is little additional work required for theinfrastructure, the database, or training thesubjects. Almost all of the work would be inplanning, developing the materials and analyzingresults, which are the heart of any researchproject.
4
One way to recruit subjects would be to askfor volunteers to carry out certain tasks and thenreward them in some way for their participation.This could easily be done using web-basedevaluation. Much of this process could be
automated, should this be desirable. These extraassignments could appear either with the regularhomework or separate from it, and could begiven with extra instructions and restrictions asneeded. However, one of the unique features ofusing web evaluation as a research tool is theability to be tightly integrated into a regularcourse. In this way, all of the students in acourse, by virtue of doing work for that course,would be participating in the study. This isattractive for several reasons. First, this can
provide a large number of subjects, as WATS areparticularly attractive for large-enrollmentcourses. Second, this would provide a less
biased sample than the use of volunteers is likelyto provide (Campbell & Stanley, 1963; Gay,1980). Third, students will not be very aware ofbeing studied and observed. This means that thestudy will happen in a very realistic (albeit high-tech) setting where there is nothing to remindthem that the computer at which they are doingtheir homework is doubling as an observationtool. Even if they are told at the beginning, theywill tend to forget. This, of course, raises ethicalissues of coerced participation and the rights ofthe subjects to know. This will necessarilyrequire some sort of oversight to ensure that therights of the students are not violated and willplace restrictions on the design of studies. Onerestriction will be how control groups are set upso that no students are unfairly penalized. Forexample, in the study with multimedia-focusedproblems described above, the treatment groupand control groups were reversed every otherweek, ensuring that all students had equalopportunity (or frustration) of working onmultimedia-focused problems. Another
restriction on investigations will be that all thematerial involved is appropriate in both contentand difficulty level for the students’ goals oflearning the material and performing
successfully in the course. For example, caremust be exercised when including standardizeddiagnostic exams as part of course material, asmany are designed to have average scoresaround 50%, far lower than typical classroomexams.
The use of computerized systems as thesetting for educational research also introducesthe potential for complications due to thetechnology. This is particularly true when theinvestigator and subject have little directcontact—the normal case for WATS. Onereason is the potential for bias related to thetechnical abilities of the subject or theinvestigator. A subject that is not as
knowledgeable or as comfortable is more likelyto be intimidated or confused by the task at handthan a more knowledgeable subject is. In mostcases, the investigator will not be present to clearup the difficulties, and may never know to whatextent the subject’s response represents theirknowledge of the subject and to what extent theirtechnical aptitude. A poorly designed system orweb page could aggravate this. Another
potential problem due to the heavy reliance ontechnology is the effect of technical problems.Again, this is particularly true of WWW systems,where no one has full responsibility for all thetechnology used. Problems could include:crashes of the main server, crashes of the
student’s computer, bugs in the program or theassignment, incompatibilities with browsers andInternet service providers (especially when usingadvanced features like Java), and limited accessto the Internet by subjects. All of these issuescan negatively affect the validity of the study.Knowledgeable planning and judicious use ofadvanced features can go a long way tominimizing these problems, but these issuesmust be taken into account when interpreting theresults, particularly if control and treatmentprocedures use different features of the WATS.
The data that can be collected
In WATS, both the instructor and the
students interact directly with the computerizedweb system. This becomes a powerful filter asto the nature of the interactions and theinformation available for the researcher. A
WATS is very flexible in the content that can bedelivered. Computers can easily collect andstore every detail of the interaction. However,the nature of human-computer interaction
restricts the types of information that can easilybe collected.
One of the significant characteristics of theWorld Wide Web is its ability to be used in avariety of ways. Examples can be seen thatrange from personal self-expression to
5
commercial transactions and from publicannouncements to entertainment. The webcontext of WATS means that there is greatflexibility in how information may be presentedto students and in the ways they are asked torespond. Nearly anything that can be sent indigital format can be delivered to the students.This information could include text, images,video, sound, animations and simulations.Information could come from the instructor,fellow students, or other third-party sites. Thishas several potentially major benefits both forinstruction and education research. First is theability to address a greater variety of learningstyles (e.g. visual-oriented vs. text-oriented).Second is the “real world” aspect of the
problems presented. Third is that a greater levelof participation can be required of students.Students could be presented with a video or ananimation from which they must obtaininformation in order to be able to solve theproblem. Students could be given a problem inwhich they must set up and run a simulation inorder to determine the answer. Students couldbe given a problem requiring the use of
information from several different web sourcesin its solution. An example is the multimedia-focused problems discussed above, which
combines some aspects of traditional homeworkproblems with laboratory work. This couldserve as an alternative or supplement to
traditional laboratories, especially when there aredistance, time or budgetary constraints on thelatter.
Likewise, WATS have, at least in principle,few restrictions on the ways in which studentsmay respond. As described above, the currentsystem can present questions in a variety offorms. Students can select among severaloptions, either in an exclusive multiple-choiceformat or in a non-exclusive checkbox format.These need not be in the traditional form—
selecting a region of an image map also works asa multiple-choice problem but in a much morenatural format. Students can fill in boxes eitherwith numerical answers that are checked againstan internal answer or with a word that is checkedagainst one or more correct answers. These areall automatically graded. Free response (essay)questions can also be asked, which in some casescould be graded automatically, or they could becollected and then analyzed by the instructor oranother person. Although currently somewhat
limited as far as automatic grading, submissionsin the form of graphical drawings also have greatpotential. Finally, the web system could acceptsubmission of computer files or code fragments.The use of the web and custom-generation ofpages by Common Gateway Interface protocol(CGI) scripts also allows individualization of thematerial presented to each student. This iscurrently used in randomizing some of thenumerical values students receive. Customgeneration of pages can present students withdifferent material depending on their class,group, or any other classification—includingtreatment and control group membership.Students may be presented with differentmaterial or problems depending on otherinformation, such as past history. This is
exploited in adaptive testing, where the subjectsreceive a series of questions, and each onedepends on the responses to previous ones
(Trentin, 1997). In a well-designed format, thiscan probe students’ understanding and providethe researcher with many of the benefits of aninterview.
One of the greatest benefits for educationresearch is that, by the very nature of the
computer-mediated interaction, every detail ofthe subject’s interactions with the system can berecorded. Unlike observations, interviews,written samples and most every other researchmethod, the primary data from web-basedsystems is already in digital form where it caneasily be searched and processed. A log of everysubject’s activity on the system—requests forpages and submissions for grading—can beeasily maintained. Furthermore, client-sideapplications such as Java applets can be writtenso as to send information back to the serverabout the subject’s every action. In this way, alarge amount of detailed information from everysubject’s interaction with the system can easilybe obtained and processed.
The use of computer-mediated interactiondecreases the contact between the investigatorand the subjects. This has a positive aspect asthis reduces the chance of investigator bias andhelps to ensure that all students are treated alike.On the down side, it greatly reduces the
collection of some types of information. In sucha system, the primary data from a given subjectis in the form of lists of requests, question idnumbers, submission numbers, etc. This is not a
6
very intuitive form in which to have data. Itrequires a certain amount of electronic
manipulation to even begin to sift out usefulinformation. This form also captures only onefacet of the subject’s activity—that with thecomputer. Thus, it would be difficult to capturea comprehensive picture of the learning processby means of a WATS. This tool does not lenditself very well to a qualitative study and oftenwill need to be supplemented with interviews orother means in order to interpret the data. Theremoval of direct interaction between
investigator and subject also removes manychannels that might alert the investigator toproblems during the study, such as body
language. Thus, it may be discovered only afterthe data is collected—or worse, never realized—that design flaws or technical problems havecompromised the study’s validity.
Another characteristic that sets this approachapart from many other research tools is the sheervolume of data that can easily be collected andprocessed. This follows from many of thecharacteristics described above. The use of theweb removes many of the geographical andtemporal constraints on who could be used as asubject. Generally this tool would piggyback ona system used for classroom teaching, so withlittle additional overhead cost and one couldeasily obtain large numbers of subjects throughthe class enrollments. A large variety of workcan be presented to the students, and their everyinteraction with the system can be recorded.Since this interaction is entirely in digital formatand computer-mediated, data collection requireslittle additional effort to go from ten subjects to athousand. For the same reason, it is fairly easyto include surveys and other types of feedbackalong with the problem solving. Being alreadyin digital form, it is also easy to supplement thisdata with information from other databases, suchas the university records department, or trackstudents through all the classes in which they usethe WATS. Education research through WATSfaces few logistical and economical constraintsfor carrying out large, detailed studies. This isusually a desirable quality, as larger sample sizestend to increase reliability of results
The sheer volume of data that can easily becollected of course also has potentially negativeaspects. The ease in which WATS generatesdata means that the focus of research must be asmuch on the selection and screening of
information as on the acquisition of it. The factthat the investigator can quickly assign
additional material means that it would be easyto overlook its cost in terms of workload on thesubjects. This could result in annoying thesubjects (or worse) and thereby damaging thevalue of the study. Unfortunately, this is also atype of problem that the investigator may be lesslikely to notice during the study. Also, any timedatabases contain detailed information aboutmany people there are ethical issues of privacyand right-to-know, which only become moreimportant as the size increases. There may evenbe political implications. Will a university
continue supporting a research effort that showssubstantial differences in performance in classwork between gender or racial groups? Finally,the sheer size of the data collected can causesignificant problems in interpretation. Most ofthe data will be uninteresting lists of requests andsubmissions, and so the researcher must findways to sort through this and glean that which isinteresting. Even so, one is likely to wind upwith a large aggregate of facts about each of thesubjects, while at the same time the data itself isnot very helpful in giving the researcher a goodover-all picture of what is going on. This wouldaggravate the problem of dealing with largeamounts of data and could wind up wastingstudents’ time. The volumes of data that can becollected by this tool can be a great boon formany research projects, but by the same tokenrequire good understanding and careful planningby the investigator to avoid the pitfalls of dealingwith this quantity of data.
Suggestions about the use of WATS asa research tool
WATS offer some unique advantages for useas an education research tool, but they alsopresent a number of significant drawbacks. Allof these need to be taken into consideration
when considering its appropriateness for a givenstudy and throughout the planning process. Thestrengths of this tool relate largely to itsflexibility, ease of data collection and thecloseness it can approach (some types) of realcourse settings. Closely connected to theflexibility and computational power are themajor weakness of this tool: the researcher’sreduced control of the experiment, reducedcontact with the subjects, and reduced channels
7
of information. WATS are a tool that has greatstrengths and significant limitations. Bothshould be carefully considered in the design ofstudies employing it.
Here the authors suggest several types ofresearch studies that could benefit greatly fromthis tool.•
Large, quantitative studies. WATS arebetter suited than any other research tool formany types of large-scale studies. Thisincludes traditional multiple-choice (MC)exams and questionnaires. The latter can
easily be delivered via WATS, which can alsoaccept and score answers that are beyond theability of the MC format (Zandvliet &
Farragher, 1997). Since the person directlyenters the data, there is no need for opticalreaders or other data entry methods. A well-designed screen layout is more intuitive thanan optical scanning sheet, which could reduceerrors in data entry. Already computers arereplacing pencil-and-paper exams (Bugbee,1996) and the use of the Web will furtherfacilitate this. Just as with MC forms, Webquestions can be administered either in aproctored environment or at the convenienceof the subjects. The one significant drawbackis the technology—having the computers forthe subjects to work on. However, if the
technological infrastructure is already in placefor other purposes, this could be even cheaperthan paper MC tests.
•
Widely dispersed subjects. Again, the webis in many cases superior to the other methodsof dealing with a geographically separatedpopulation, i.e. mailing forms or conductingtelephone surveys. It is faster, provides moreconsistent information and—provided thetechnological infrastructure is already in
place—is cheaper than other means. It wouldbe particularly useful in evaluating distance-learning students using the web; already thereare many courses being offered in thismedium (CASO, 1998; Collis, 1997;
Donahue, 1996; Faulhaber, 1996; Leonard,1996). Other kinds of studies could alsobenefit. For example, a proposed diagnostictest could be given to subjects from aroundthe country. A third use would be
undertaking a study that involves a narrowlydefined but physically scattered group (e.g.professional enhancement resources available
to African-American middle-school scienceteachers located in rural communities).•
Monitoring of courses. For courses thatalready use Web-based evaluation systems forhomework, WATS are the easiest, fastest andcheapest means of monitoring students’progress and getting feedback. Scores,
patterns of use, and level of participation arealready recorded in a format that can be
analyzed and linked to other data. Patterns ofuse could be tracked over time and the rangeof scores on individual exercises is easilyviewed. Some of this monitoring should takeplace in courses using WATS to compensatefor the loss of contact and direct feedbackbetween instructors and students. Computerscripts could be set up to automatically
analyze student answers, looking for patternsthat indicate conceptual or other types ofdifficulties. The system could then alert theinstructor and/or student of this pattern, oreven automatically direct the student toadditional resources.
•
Evaluating the use of WATS. WATS arevery new and are a different approach to theway in which students do homework. Theimplications of this for their learning is notentirely understood, and is an area that needsto be researched. All WATS courses modifysome aspects of the pedagogical process. Forexample, students using WATS usuallyreceive more immediate feedback on theirhomework but only their final answers areevaluated, which is a trade-off compared tohanding in written work. Although the
relative merits have been debated, no studieshave been carried out to find out under whatcircumstances this is a net gain for thestudents. The flexibility of question
presentation and answer collection presents avariety of ways curricula can be changed,leading to similar to those raised above. TheWATS themselves will prove to be animportant tool in gathering this sort of
information, though not necessarily the onlyone.
•
Evaluating the use of multimedia. In onesense, this is merely an extension of the abovepoint, but it has even greater potential impacton instruction than the automatic grading.The ability of the Web to deliver diverseinformation has the potential to completely
8
change the appearance and use of homework.A great deal of time and effort will be
expended on this, but only some of the effortswill really improve students’ learning—andsome may actually interfere with it. Muchguidance needs to be given to these efforts.An important component will be how realstudents use multimedia in real classes. Thereis no other tool that can match Web systemsfor unobtrusively monitoring large numbers ofstudents as they do their work in real
situations. This also has potential beyond justevaluating multimedia use in WATS. Javaapplets written to send back information to theserver about how they are being manipulatedcould be used on many different types ofpages. A public site like that of a sciencemuseum could have its use monitored by asimilar scheme as described here. While otherresearch tools will also be of value, WATSwill be an essential element of understandinghow students interact in multimedia problem-solving environments.•
Complex studies. Sometimes it would bedesirable for the course of the treatment or theevaluation to depend on a large number offactors. One may wish to give a studentdifferent instruction based on a combinationof performance on a pretest and a self-described learning style. Adaptive testing,where the subject receives different questionsdepending on answers to earlier ones, may bea desirable tool. Computer adaptive testingcan validly duplicate paper and pencil testingif it is well designed (Bugbee, 1996; Trentin,1997; Zandvliet & Farragher, 1997). Thesetype of studies can be easily and flawlesslyadministered by a well designed WATS, evenwith the large numbers of subjects that wouldbe required to obtain statistical results fromsuch complex studies. Furthermore, all theinformation about exactly what sequence thateach subject went through would be recordedas part of the log file.
•
Complementing other tools. As has beensuggested above, the particular strengths andlimitations of research using WATS are verydifferent from some of the other educationresearch tools. For example, in comparison tocollecting student web responses, a video-taped interview about problem solving is farmore invasive, more geographically
constrained, harder to do with large numbersand more challenging to analyze. On the
other hand, it provides a much richer, personalpicture of what is going on than a web systemcould ever provide. In this way, web systemstudies and studies with more qualitative
tools, such as interviews and observations, canprovide strong complements in developing acomprehensive picture of student learning. T
his is by no means intended to be acomprehensive list, but rather to provide an ideaof the types of education research projects whichcould benefit from the use of WATS as a
research tool. In general, these projects are thosewhere flexibility is a desirable quality, wherecomputational power is important to deal withlarge or complex studies, and/or where thetechnology itself is a major focus of the study. P
ractical Suggestions A
few practical suggestions on implementingresearch via WATS are offered here. This iscertainly not a complete guide to carrying outsuch studies but rather represents issues that theauthors have encountered up to this point. Forthe most part, these can be summed up as “planahead and be realistic about how the systemworks”.•
Know how the students use the system.WATS allow students to interact with thesystem in a variety of ways, some of themunexpected. A poll given in the Fall
semester of 1997 at NCSU found that overhalf of the students frequently printed outassignments, presumably to work them outon paper and then enter them into thecomputer at a later time. By informal orformal means, the researcher needs toobtain a realistic picture of how the
students actually go about doing the work.•
Plan ahead. Any good research projectneeds to be carefully planned so that it willachieve its goal, and research via WATS isno exception. What is different is the easewith which one can plunge into researchactivity without making careful plans, andthe mountain of unintelligible data that canbe effortlessly collected without a clearpicture of the goals.
•
Maintain a good relationship with thesystem administrator. Unless the
9
researcher also happens to be the systemadministrator, these two people will oftenhave different backgrounds, outlooks andgoals. The level of cooperation providedby the system administrator will affect theusefulness of the WATS for research.Since in most cases, the research will bepiggybacking on a system originally
designed for other purposes, the researchermust make every effort to not be just
another demand on the (often over-worked)system administrator.•
Judicious use of advanced features.Unless the subjects will always be usingdesignated computers, the HTML for givenpages or problems needs to be written witholder browsers in mind. For example,
Applets written in Java 1.1 will run only onExplorer 4.0 or higher, and at the time ofwriting there is no Macintosh browser thatcan run them. Compatibility acrossplatforms also must be considered; fontsused may need to be limited to those
available on all platforms (Macintosh, PCand UNIX machines). Similar
consideration may need to be given toavoiding the inclusion of large amounts ofgraphics, in deference to students with slowmodems.
•
Minimize interference with the on-goingclasses. This is particularly relevant whenthe researcher is not the instructor. Thegoal of the instructor and students is theireducation, and the research must not hindertheir progress. One issue that has arisen inthe use of WebAssign at NCSU has beenconfusion among students over whethersurveys posted on the web count towards
the total homework score—even after astatement to the contrary was posted at thetop of every survey.•
Be realistic about authentication. Onecan never be 100% sure that a givensubmission came from the student whosename is on it. Students may work together,or might give their password to someoneelse. This must be taken into consideration;just as with pencil-and-paper work, there isthe possibility of copying or submittingwork in another person’s name.
In conclusion, web-based assessment andtesting systems offer the education researcher anew and, in some aspects, unique tool. It offerstremendous advantages including flexibility ofsetting and format, the ability to collect largeamounts of data, and tying in closely with
courses. By the same token, computer mediated-communication also reduces the investigator’scontrol of the experimental setting and directcontact with the subjects. This tool can providesome types of studies with great benefits.Careful planning can minimize many of thesignificant pitfalls of this tool and monitoringthat recognizes the unique aspects of WATS.Like any other product of technology, WATSoffer education research great benefits whenproperly understood and wisely used.
Acknowledgments: The authors would liketo thank the NCSU Department of Physics for itssupport for developing a WATS and allowing itsuse for research purposes, and John Risley andPeg Gjertsen for their work in turning it into areality. One of the authors (SWB) would like tothank the National Science Foundation for itssupport of this work through PostdoctoralFellowship DGE-9714546.
10
Brooks, D. W. (1997). Web-teaching: a guide to designing interactive teaching for the World Wide Web.
New York: Plenum Press.Bugbee, A. C. (1996). The Equivalence of Paper-and-Pencil and Computer-Based Testing. Journal of
Research on Computing in Education, 28(3), 282-299.Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research.
Chicago: Rand McNally College Publishing Company.Campbell, T. (1997). Technology, Multimedia, and Qualitative Research in Education. Journal of
Research on Computing in Education, 30(2), 122-129.CASO. (1998). Internet University, [WWW]. Available: http://www.caso.com/.
Christian, W. (1996-1998). WebPhysics, [World Wide Web]. Davidson College. Available:
http://WebPhysics.davidson.edu/.
Christian, W., & Titus, A. (1998). Developing Web-based Curricula Using Java Physlets. Computers in
Physics, 12(3), 227-232.Collis, B. (1997). Pedagogical Reengineering: a Pedagogical approach to course enrichment and redesign
with the WWW. Educational Technology Review, Autumn/Winter1997(8), 11-15.de La Beaujardiere, J. F., Cavallo, J., Hasler, A. F., Mitchell, H., O'Handley, C., Shiri, R., & White, R.
(1997). The GLOBE Visualization Project: Using WWW in the Classroom. Journal of ScienceEducation and Technology, 6(1), 15-22.Donahue, S. (1996). Programs and Resources in Distance Education. Journal of the American Society for
Information Science, 47(11), 870-874.Engelhardt, P. (1997). Examining students' understanding of electrical circuits through multiple-choice
testing and interviews. Unpublished Ph.D. dissertation, North Carolina State University.Faulhaber, C. B. (1996). Distance Learning and Digital Libraries: Two Sides of a Single Coin. Journal of
the American Society for Information Science, 47(11), 854-856.Gay, L. R. (1980). Educational Evaluation & Measurement. Columbus: Charles E. Merrill Publishing
Company.Goldberg, M. W., & Salari, S. (1997). An Update on WebCT (World-Wide-Web Course Tools) - a Tool for
the Creation of Sophisticated Web-Based Learning Environments. Available:http://homebrew1.cs.ubc.ca/webct/papers/nauweb/full-paper.html.Halliday, D., Resnick, R., & Walker, J. (1997). Fundamentals of Physics. (5 ed.). New York: John Wiley &
Sons.Halloun, I., & Hestenes, D. (1996). Views About Sciences Survey: VASS. Paper presented at the Annual
Meeting of the National Association for Research in Science Teaching, St. Louis, MO,.Hecht, E. (1998). Physics: Algebra/Trig. (2 ed.). Pacific Grove: Brooks/Cole Publishing Company.Khan, B. H. (Ed.). (1997). Web-based instruction. Englewood Cliffs, N.J.: Educational Technology
Publications.Leonard, D. C. (1996). Using the Web for Graduate Courses in Technical Communication with Distant
Learners. Technical Communication: Journal of the Society for Technical Communication, 43(4),388-401.NCSU. (1998). WebAssign, [World Wide Web]. Available: http://wwwassign.physics.ncsu.edu/demo/.NSTA. (1998, May/June). Survey Shows K-12 Internet Use Is Up. NSTA Reports!, pp. 18.
11
Redish, E. F., Saul, J. M., & Steinberg, R. N. (1998). Student expectations in introductory physics courses.
American Journal of Physics, 66(3), 212-224.Serway, R. A., & Faughn, J. S. (1995). College Physics. (4 ed.). Fort Worth: Saunders College Publishing.Titus, A. (1998). Investigation of the Effectiveness of Animation on Solving Physics Problems. Unpublished
Ph.D. dissertation, North Carolina State University.Titus, A., Martin, L., & Beichner, R. (1998). Web-based Testing in Physics Education: Methods and
Opportunities. Computers in Physics, 12(2), 117-123.Trentin, G. (1997). Computerized Adaptive Tests and Formative Assessment. Journal of Educational
Multimedia and Hypermedia, 6(2), 201-220.Zandvliet, D., & Farragher, P. (1997). A Comparison of Computer-Administered and Written Tests.
Journal of Research on Computing in Education, 29(4), 423-438.Zhao, Y. (1998). Design for Adoption: The Development of an Integrated Web-Based Education
Environment. Journal of Research on Computing in Education, 30(3), 307-328.Zitzewitz, P., Davids, M., Neff, R., & Wedding, K. (1992). Merril Physics: Principles & Problems. (1992
ed.). Westervill, Ohio: Glencoe/McGraw-Hill.
12
Appendix: A partial list of Web-based assessment and testingsystems
SystemCAPACompanionWebsites
Course ManagementSystemCyberProfGateway TestingSystem
homework serviceinteractive problemsMallardMicroTestpractice quizzesQuestion MarkTopClassVirtual-Uweb evaluation andfeedbackweb quizzesweb-basedhomeworkWeb@ssessorWebAssignWebCTWWWAssign
DeveloperMichigan State UniversityPrentice HallJohn Wiley & Sons
University of Illinois (Center forComplex Systems Research)University of Nebraska -Lincoln.(Dept. of Mathematics and Statistics)Univeristy of Texas-Austin(Department of Physics)
University of Illinois (Department ofPhysics)
University of Illinois (Department ofElectrical and Computer Engineering)Chariot Software Group
Addison Wesley/Benjamen CummingsQuestion Mark ComputingWBT Systems
Simon Fraser University
University of North Carolina-Charlotte(Department of Physics)McGraw-Hill
University of North Carolina-Ashville(Department of Physics)Computer PREP
North Carolina State University(Department of Physics)WebCT
North Park University
13
Web address (http:// )capa2.nscl.msu.edu/homepage/www.prenhall.com/pubguide/www.wiley.com/college/cms/www.howhy.com/home/www.math.unl.edu/webtests/hw.ph.utexas.edu/overview.htmlwebug.physics.uiuc.edu/courses/ie.htmlwww.cen.uiuc.edu/Mallard/www.chariot.com
occ.awlonline.com/bookbind/pubbooks/campbell_awl/www.qmark.comwww.wbtsystems.com/virtual-u.cs.sfu.ca/vuweb/http://www.physics.uncc.edu/cgi-bin/CGI/CGI4edu.htmlwww.mhhe.com
www.ctl.unca.edu/bennett/Phys222/default.asp
www.webassessor.comwwwassign.net/info/http://www.webct.com/
www.northpark.edu/~martin/WWWAssign
因篇幅问题不能全部显示，请点此查看更多更全内容