Note: If you browse this site it will use temporary 'session cookies' to assist navigation and keep an anonymous record of page visits. More info…

1. Introduction

1

How can technology contribute to the summative assessment of problem solving skills – a requirement of the “Mathematical Practices” in the US Common Core State Standards and many other national curricula – in an otherwise conventional timed assessment?

As discussed in the ISDDE working group report in this issue (ISDDE, 2012), conventional summative testing tends to promote tests comprised of many short, closed items with multiple choice or short, constructed answers. The economies of large-scale computer-based testing are biased in favour of this model, despite its weakness in testing problem-solving. In contrast, there are copious examples of rich classroom activities using technology to develop process skills in mathematics and science (see, for example: Boon, 2009, Figueiredo, van Galen & Gravemeijer, 2009). There are also efforts to produce radically different types of summative assessment based on extended, immersive, virtual reality investigations (such as the Virtual Performance Assessment Project at Harvard). In this article we consider a “third way” – computer-delivered tests comprised of self-contained 5-10 minute tasks, more ambitious than the typical test item, but more structured and closed than most “classroom investigations”. In the novice-apprentice-expert model being developed in the US in response to the Common Core Standards, most of these would be classified as “apprentice” tasks (see section 2 of ISDDE, 2012, in this issue).

The World Class Tests project had the freedom largely to define its own syllabus and, specifically , to focus on problem solving skills without the usual obligation to assess the wider mathematics curriculum. On the other hand this project was required, after an initial research phase, to deliver externally marked assessments in quantity. These were published and administered by an awarding body. This places it in a slightly unusual position between pure “insight” research projects, which might study a few tasks in great detail, and regular assessment production.

2. The World Class Tests project

2

The brief

The World Class Tests were the central part of the UK government-funded World Class Arena programme, intended to provide support for ‘gifted and talented students’. A particular focus was to identify, engage and challenge those students whose ability might not be apparent from their performance on day-to-day classroom activities (so-called ‘submerged talent’).

The product, as originally conceived by the Government in 1999, would consist of computer-delivered assessment tests for students at ages 9 and 13, available ‘on-demand’ (requiring a bank of questions equivalent to producing four sittings per year). Early in the tendering process, this was altered to include a mix of computer- and paper- based tests, sat twice a year.

There would be two separate sets of tests: “Mathematics” and “Problem solving in mathematics, science and technology”. This article concentrates on the development of computer-based tasks for the “problem solving” strand and the issues arising from this process.

Educational principles

2

3. The role of the computer

3

Although the original brief called for an entirely computer-based assessment, the consensus of the designers was that the “state of the art” of computer-based testing and automatic scoring would require highly structured questions with constrained response formats, precluding the type of open-ended, unstructured tasks which are an essential component of problem solving – and of the Balanced Assessment philosophy. The arguments for this were similar to those presented in the ISDDE Working Group Report (ISDDE, 2012). QCA, the government agency, accepted this and it was therefore decided that each test should consist of two parts – one using pencil-and-paper and another delivered by computer.

In addition to the pencil-and-paper-only tests, the computer-based tests would also be accompanied by a paper workbook. For the mathematics tests, these were used purely to provide space for rough working. In the case of problem solving, however, some on-screen questions would instruct the students to write the response in their workbook. This was seen as the only way that students could respond to questions which required a description (possibly including mathematical notation) or demonstrate that they could, autonomously, choose to represent data as a chart or table without being given an on-screen form which defined the format for them.

Although probably untenable in the long term for a “computer based” assessment, this did provide a valuable interim solution as task styles developed. It was also the only way that tasks could be trialled in the early stages of the project, before the data collection infrastructure was in place. Towards the end of the project, as experience was gained by the designers, the dependence on the answer books was waning. Had task development continued, the answer books would probably have been dropped or, as with the mathematics tests, relegated to “rough work” which would not be marked.

The availability of the written paper-based tests meant that the computer tests did not have to waste effort replicating tasks that were known to work well on paper, and could concentrate on ideas that exploited the computer to the full. The answer booklet for the computer test meant that the computer could be used to present contexts and information in an interactive format without sacrificing the ability to ask less structured, investigative questions.

Qualities that made a task particularly suitable for use in the computer-based component included:

• The use of animation or interactive graphics to present concepts and information that would be hard to communicate, in simple language, on paper;
• The provision of a substantial data set, for students to explore with searching or graphing tools;
• Use of simulated science experiments, games and other “microworlds” - allowing question types that would be impossible on paper; and
• Other types of questions that were more suited to computer than paper – for example, questions that naturally suggested a “drag and drop” interface.

The main constraint was that the test was to be assembled from self-contained, 5 to 15-minute tasks. Although such tasks are long compared to those typically found on current mathematics tests, it is quite short for the sort of open-ended investigations suggested by the criteria above. As well as the direct limitation on task length, this meant that any on-screen “tools” that the student was expected to use within a task had to be extremely simple and intuitive to operate, otherwise valuable assessment time would be wasted on on-screen tutorials and practice before each task.

As the tests were to be scored and graded conventionally, each task also required a systematic, summative scoring scheme so, even without the constraints of capturing the answer on computer, there needed to be a definite “outcome” against which performance could be reliably assessed.

The other constraint was that tasks had to be produced in significant quantities (over the course of the project, 110 computer based tasks were developed, each representing 5-15 minutes of assessment and usually involving some sort of interactive animation or simulation). This limited the amount of software development effort that could be devoted to an individual task.

4. Illustrative examples of tasks

4

Table 1: “Heuristic inference” scoring for the Sunflowers task

Amount of A and B for best height achieved	Inference	Score
11 ≤ A ≤ 12	Has held B constant while varying A Has tried 0 or <1 for B Has searched for maximum using integers +1	+1
11.0 < A < 12.0	Has used decimal fractions.	+1
0 < B < 1	Has used decimal fractions less than 1	+1
0.3 ≤ B ≤ 0.4	Shows some sort of systematic search for B Has held A constant	+1
0.30 < B < 0.40	Has gone to 2 decimal places. +1	+1
A = 11.5, B = 0.36	Full marks!	+1

Mathematical games

1

Exploring rich data sets

2

Use of the workbooks

3

5. The development process

5

Initial design

The design philosophy was that design should start with valid and engaging tasks that would allow candidates to show “what they know, understand and can do” (Cockroft, 1982). Small-scale school trials of the tasks took place at an early stage to ensure that students could engage with the task and demonstrate progress. The scoring  schemes were developed continually throughout the trials to ensure that they reflected the type and variety of valid responses produced by real students, not simply the designer's anticipated solution, and could be applied reliably by the scorers. The balancing instruments described above were then used to assemble a test that adequately sampled the assessment domain.

This approach differs from most test development, which is typically centred on detailed, but abstract, specifications of the curriculum areas to be covered, around which the tasks are constructed. This is straightforward, but can lead to the sort of fragmentation and contrived contexts often seen in assessments such as the General Certificate of Secondary Education examinations in England (see section 5.2 of Pead, 2010).

The above context-led technique would be impractical if applied universally, so some tasks were inevitably written to address gaps in coverage or balance as the test was assembled.

Ideas for computer-based tasks arose in various ways. They were developed in brainstorming sessions; invented by individual designers and other contributors; adapted from past projects or “appropriated” from tasks under development for the paper test. It was then up to the computer task designer to develop the ideas into a workable specification.

At this point, one of the challenges of computer-based task development became apparent: traditional paper-based tasks at this stage of development would have been drafted, with clip-art graphics and rough diagrams where needed, ready for further discussion and refinement, initial approval by the clients and informal trials. For computer-based tasks, though, all that was available were sketches of the screen layout, the wording of the question and technical notes on how any interaction or animation would work. Tasks in this state could not be trialled in school. Even soliciting feedback from colleagues and clients proved difficult when the task had significant graphical, animated or interactive elements that any reviewer would have to visualise based on the outline specifications.

Specification, commissioning and implementation

2

Programming of tasks was conducted by a third party, so the next step was to specify each task in detail for the programmers.

The specification had to cover such aspects as:

• Details of the artwork required – this needed tight specification due to the danger of introducing additional clues or distractions: it is surprisingly easy to inadvertently include a 'red herring' in an image;
• Details and timings of any animation required;
• How on-screen objects should respond to various inputs, covering:
  • Suggested algorithms where objects have to move according to mathematical rules, or where the computer must play or referee a game;
  • The range of possible inputs for type-in fields (e.g. text, integers, decimals, including the number of decimal places). Should the candidate be warned of/prevented from entering invalid values?; and
  • Rules for drag-and-drop elements – where on the screen do objects start? How many are available? How they can be removed? Should they automatically align to a grid?
• Details of what data should be captured and stored so that it could be marked;
• Details of how the task should be paginated and whether some elements should appear on all pages. This could be crucial, because of the limited amount of information that can be presented on each screen; and
• Eventually, specifications for the algorithms needed to score responses automatically, although this stage came after the initial implementation, once a manual scoring scheme had been designed.

In the context of a 10-minute assessment task, where the candidate must be able to rapidly grasp the concept without additional help, the considerations above can be critical and are hard to separate from the educational design. For example, the task designer might design, on paper, a “cloze” question[4] comprising a text with missing words (or numbers) and a list of possible words to go in the gaps. The student would copy the correct words into the gaps. A programmer might decide to implement this by displaying the candidate words on icons which the student could drag and drop onto the gaps in the text. This is not necessarily the same problem, since the new design means that you can only use each word once – a constraint which is not present in the paper version. Even if the correct answer only uses each word once, it is possible that a common mistake involves re-use of a word, so denying the student that option could affect the nature of the task.

From the point of view of a software designer aiming to produce robust and easy to operate software, checking the validity of data and dealing gracefully with any unexpected inputs is an important consideration. Adding constraints and checks to the user interface which restrict the domain of possible responses with which the software must cope is therefore an attractive technique[5]. This might make the task simpler to score by preventing ambiguous inputs but could also make the task easier by alerting the candidate when they entered a wrong answer. The educational designer must be involved in deciding how such constraints might alter the question. So, in the above “cloze ” example, the designer must remember to specify whether there should be more than one of each icon, something which they might not consider in a paper-based task.

Typically, the first implementation of a task by the programmer had serious faults and one or two rounds of improvement requests were required to arrive at a version ready for trials. This was not simply due to mistakes by the programmer, but often because the designer wished to refine details having seen the first working version. Good communication between the educational designers, graphics designers and programmers was essential here, and the strictly partitioned approach imposed by the World Class Tests project structure, where (for instance) change requests sometimes had to be submitted in writing without face-to-face contact with the programmer, was not ideal.

As the project progressed, it was often found to be simpler for the designer to produce partial working prototypes which implemented the critical interactive aspects and included draft graphics and animations, which could be fine-tuned before submission.

In the initial stages, the delivery “shell” which allowed the candidate to log on and navigate through the questions was also under development, as was a “library” of standard buttons, input boxes and other controls. An example of the sort of issue that arose here was whether it should be possible for a candidate to return to a previous question to review, and possibly modify, their responses. This is something that would be taken for granted on paper, but which is only possible on computer if it has been specifically provided for in the test delivery software.

Trial and refinement

3

Each task was scheduled to go through at least three rounds of trials:

• “Informal”, closely observed trials with a small number of students to ensure that they could engage with the task and to identify any bugs or shortcomings in either the task content or its technical implementation. These trials were often conducted with students working in pairs, with no attempt made to present balanced tests or gather psychometric data. Working in pairs encouraged students to discuss their thinking (and, sometimes, express their frustrations) without the observer having to interrupt with questions.
• “Formal” trials, with around 50 students taking each task, to establish that the tasks were performing well in an assessment environment and producing an adequate spread of results. These trials remained focussed on individual tasks. The resulting student work was used to refine the scoring schemes and to inform the assembly of the tasks into balanced tests.
• “Pre-test” trials of complete, balanced tests – aiming for around 200 students per test – intended to provide statistical data for calibrating the tests.

A major tension was that, for the first two rounds of trial to be worthwhile, it had to be possible to rapidly revise and re-trial a task. There was a conflict between the need to schedule school visits for informal trials in advance and the requirement to commission any revisions from the developers. A flaw in a task might become obvious the first time a child tried to complete it, but whereas a paper task could be redrafted overnight, it was often impossible to revise the software in time for the next scheduled visit. Combined with the delays in task commissioning noted above, and the problems with getting infrastructure in place for trials (discussed below) this meant that it was often impossible to put computer tasks through the full, three-stage, iterative trial and refinement cycle, and many tasks skipped the “formal trials” step.

Some design challenges

4

Finding the task in the context

The desire for rich and interesting contexts has to be balanced with the constraints of the assessment. Many appealing subjects emerged from brainstorming sessions – such as Muybridge's famous pictures of galloping horses, or analysis and comparison of demographic data from many countries – but identifying a self-contained, 5-15 minute task set in that context often proved difficult.

One of the hardest decisions for a lead designer was when (and how) to diplomatically reject a contributed idea, into which a lot of research and effort had already been put and which would make a wonderful extended investigation, on the grounds that no well-defined, score-able task had been identified.

Eliminating trial and error

When designing interactive test items based around a microworld or simulation, a key challenge is finding questions which genuinely probe the students' understanding of the situation and which cannot be answered with a simplistic “trial and improvement” approach in which the student uses the simulation to check possible answers.

Tactics used to eliminate or reduce trial and improvement include:

• Written explanation – ask students to describe their strategy/justify their findings, or to support/refute some suggested hypotheses.
• Simple challenge – ask students to “beat the computer” and rely on the time constraints of the test to discourage brute force/trial and error solutions.
• Logging and analysis – record every interaction between the student and computer and then try to analyse this data to spot promising patterns and sequences. This requires complex coding and could be fragile: a few random interactions not indicative of the students' thought processes could disguise a valid response. Generally, a large corpus of trial data would be needed to validate such an approach.
• Heuristic inference – Table 1 shows a possible scheme for scoring  the Sunflower task (Figure 3) which infers the sophistication of reasoning and strategy shown by the student based solely on their best result, without recourse to their written work or their sequence of trials. Likewise, with Factor Game (Figure 5) the final score was taken to be indicative of the level of understanding: most students could beat the computer eventually; a “high score” of 30 suggested that the student grasped the idea of factors and multiples; 35 implied they had made some progress towards a strategy for improving their score while the optimum score of 40 was unlikely to be achieved without a well-developed strategy. This has the advantage of being easy to program and fairly easy to justify – but the approach does not lend itself to all tasks.
• Extension problems – after exploring an interactive scenario, such as a computer game, the student is asked to demonstrate their understanding by making inferences or predictions about an extended or generalised variant, with no simulation available. This technique was also used in Factor Game, where the final challenge is to suggest the optimum opening moves in a game with 50 cards instead of 10. In other cases, an arbitrary limit was set on the range of inputs accepted by the simulation and the final question lay outside that domain.

6. Technical and Logistical Challenges

6

Technical issues

The project started before widespread access to broadband internet connections could be taken for granted. Consequently, most of the tests were delivered on CD and had to be installed on individual computers. The data then had to be extracted from the individual computers and returned by email or mailed on floppy disc.

This proved to be a major challenge – especially in schools with networked systems that prevented individual machines from writing to their local hard drives. Although this potentially meant that administration and data collection could be centralised, the diversity of networking systems and lack of technical support made installation complicated. Even on stand-alone systems there was a high incidence of lost data when teachers were asked to manually copy and return data. The agency performing the programming and delivery software design was also somewhat naïve about the level of technical proficiency that could be expected from teachers (such as their ability to copy files by dragging and dropping rather than opening them in a word processor and re-saving).

Whatever the problems with internet delivery of assessment, the possibility of “zero-install[6]” delivery and automatic return of data is attractive in the light of the experiences with World Class Tests.

Project management issues

2

The early years of the project were somewhat fraught, and there may be some lessons to be learned for future projects. Some of the issues included:

• Structure of the project – the organisation, as conceived, was heavily compartmentalised – with two groups contracted to work on the educational design, a third contractor handling the software development and a fourth (appointed later) responsible for “delivering” the tests. This seemed to be founded in a publishing metaphor: manuscript -> editor/designer -> publisher/distributor; which assumed that the hand-over between each stage was routine and well understood. Initially, this led to designers being unaware of the constraints of the delivery system and programmers not understanding the aspirations of the designers.
• Task specification and approval – as discussed above, when tasks involve substantial interactive elements, programmers must be supplied with more than the question text and a sketch of the artwork. The workload of specifying the tasks, testing implementations and specifying revisions had been underestimated, and largely fell on one or two people. This delayed the commissioning of new tasks from the programmers – who were expecting a steady flow of routine work.
• Prototyping – in a non-routine project such as this, it is hugely ambitious to expect to go directly from paper specification to final implementation. Quickly prototyping partly-working examples, so ideas could be rapidly refined – or possibly rejected – was found to be more efficient. The prototypes proved an effective way to communicate the design to the final programmers.
• Technical oversight –the project had several stages of internal and external review to ensure the educational validity of the materials. There was, initially, no corresponding oversight of the technical issues or agreement between the designers and programmers as to what the constraints or expectations of the system were. An internal committee was eventually set up, but its source of authority was unclear.
• Timing – although the overall timescale – two years until the first live sittings - was appropriate, the contract mandated a large scale trial just a few months after the effective start of the project. This would not have been unreasonable for paper based tests which could easily be piloted in draft form. In contrast, but delivery of computer tasks required substantial infrastructure development as well as programming of the actual tasks, and the attempt to meet this requirement largely failed. Multiple rounds of trial, feedback, revision and calibration are critical to developing a robust and valid test but, in a computer-based project, need to be reconciled with the fact that a substantial amount of programming needs to be completed before any materials can be trialled.
• Short-term contracts & rights – this affected the programming side in particular – with no ongoing commitment to continue the contract after the initial two years and all intellectual property rights assigned to the client, there was little commercial incentive to invest time in building a solid technological infrastructure which might then have been taken over by the lowest tenderer at the end of the contract.

7. Outcome of the project

7

The project produced a bank of 5 complete tests at each of ages 9 and 13, which have been successfully administered, marked, moderated and graded on a commercial scale, setting it apart from “blue sky” eAssessment projects that develop and deeply research a handful of ambitious exemplar tasks.

Students in the target ability range were able to make progress on the tasks, producing a good spread of scores which adequately discriminated between different levels of performance.

Development of new test items was stopped in 2003, but test sittings continue with the existing materials – see www.worldclassarena.org. On that site it says: “Since the first test session in 2001, over 18,000 students in over 25 different countries worldwide such as Australia, Hong Kong, New Zealand, Saudi Arabia, Slovenia, the United Arab Emirates, the United Kingdom and the United States have taken the tests.”

In the later stages of the project, it was realised that students who had never encountered these types of problem in the classroom found the tests particularly difficult. Consequently, some of the test development effort was diverted to produce teaching materials based around adapted and extended versions of previous test questions. The approach used was that students would tackle the task individually or in pairs, and then engage in a classroom discussion in which they compared their techniques with other groups, and against specimen solutions provided with the materials. The tasks chosen were, intentionally, challenging so many students would only make progress after sharing techniques.

The classroom materials were published by nferNelson, including 6 modules under the title Developing Problem Solving (Crust, Swan, Pead et al. 2005).

More details of the design philosophy of these tests can be found in Computer-based assessment: a platform for better tests? (Burkhardt & Pead, 2003 ).

8. Conclusions

8

The World Class Tests project illustrates several ways in which the computer can deliver rich, open tasks involving simulated experiments, “microworlds” puzzles and games, significantly expanding the domain of task types and contexts which can be included in a formal, external assessment.

The main success of World Class Tests was in using the computer to deliver microworld-based tasks in a mixed computer and paper assessment. However, half of the assessment in World Class Tests was still in the form of paper-and-pencil tests, in addition to which the problem-solving computer tests relied partly on a paper answer booklet. While the challenges in producing a completely paperless test may have been soluble on a task-by-task basis, the design and programming load of scaling this to adequately sample the subject domain and deliver 2-4 test sittings a year would have been considerable.

The greatest implication for the technical and pedagogical processes of computer-based assessment design is the clear need for two, usually separate, areas of expertise to work together to ensure that the technical aspects of the product reflect the pedagogical principles on which it was based. Task designers accustomed to handing over their paper manuscripts for conventional typesetting and printing need to become involved in key decisions over animation, interactivity and response input methods, while programmers need to learn how their decisions can impact on pedagogical issues and know when to refer a technically-driven change back to the designer. If programmers are to work from detailed specifications then it must be recognised that developing these specifications is a new and significant phase of development not present in a traditional paper-based product cycle.

There are also challenges for design research models which rely on multiple, rapid cycles of trial and refinement.  This is straightforward when the “refinement” step means a few changes to a paper document; less so when it entails specification, commissioning and testing of software changes.

Acknowledgements

ii

Although the author was the lead designer and editor for the computer-based problem solving portion of the World Class Tests, the tasks were a team effort and include the work of Richard Philips, Malcolm Swan, Rita Crust, Jim Ridgway, Sean McCusker, Craig Turner and others. The Flash implementations shown here were produced by Vivid Interactive and Doublestruck Ltd. The World Class Tests are available from World Class Arena Ltd. (http://www.worldclassarena.org/).

Footnotes

iii

References

iv

© ISDDE 2012 - all rights reserved