Note: If you browse this site it will use temporary 'session cookies' to assist navigation and keep an anonymous record of page visits. More info…

Design approaches in the behavioural sciences

2

For over a hundred years, calls have been made to examine the relationship between the quest for fundamental understanding and the quest for applied use. Psychologist Hugo Münsterberg (1899) and educational philosopher John Dewey (1900) both spoke of a linking science, which would connect theory and practical work. Taking these ideas further, Robert Glaser (1976) laid out the elements of a psychology of instruction, calling for science of design in education. And more recently, Donald Stokes (1997) provided a fresh look at the goals of science and their relation to application for use, in his highly-acclaimed book entitled, “Pasteur’s Quadrant: Basic science and technological innovation”.

The quest for fundamental understanding is typically shaped by the empirical cycle; this is well known to both researchers and laypersons, and is used to explore, to describe, to explain or to test, and leads to theory development. De Groot (1969), identified five main phases in the empirical cycle: observation (data collection); induction (formulating hypotheses); deduction (making testable predictions); testing (new empirical data collection); and evaluation (linking results to hypotheses, theories and possibly new studies). The quest for applied use tends to be less broadly understood in a formal sense, though many would recognize commonalities in trajectories fed by the desire to solve problems in practice. For example, van Strien (1986, 1997) described this process as the regulative cycle, consisting of the following five phases: problem identification; diagnosis; planning; action and evaluation.

While the need for bringing the empirical and regulative cycles together has been evident for over a century, it only started to gain momentum in the educational sciences in the 1960’s, and has mostly grown in the last few decades. In the field of education, this approach has come to be known as design research (also design-based research and engineering research). The motives stem from both research, where experts sought to conduct studies under real-world circumstances with the aim of yielding more relevant and useful knowledge (Brown, 1992; Collins, 1992), as well as from design practice, where experts expressed the need for scientific insights to inform the creation of educational resources and programs (Stenhouse, 1975; Walker, 1992; van den Akker, 1999). Some design research models portray a process in which the empirical and regulative cycles are parallel but separate (e.g. Ejersbo et al, 2008), and others advocate that they should be integrated (McKenney & Reeves, 2012), but all design research models we have encountered do yield both scientific outputs (theoretical understanding) and practical ones (a designed product).

Several special issues of highly respected journals have addressed design research, including Educational Researcher (2003, 31(1)), Journal of the Learning Sciences (2004, 13(1)), and Educational Psychologist (2004, 39(4)). A number of books devoted to the topic examine areas such as: design research conceptualization (Akker, Gravemeijer, McKenney, & Nieveen, 2006), methodological considerations (Kelly, Lesh, & Baek, 2008), and even guides and tools for conducting design studies (McKenney & Reeves, 2012; Reinking & Bradley, 2008; Richey & Klein, 2007). These resources have been pivotal in garnering support for an approach that is viewed by many to be a viable route to increasing the relevance of research and the impact of design efforts. However, despite growing support in the professional literature for design research, the approach is still underutilized. This may be due to the fact that design research (like any other genre of research) is fitting in some circumstances, but not suited to all kinds of research questions. Yet this may also be a function of how researchers and designers are educated.

Design and research in education

3

Design research is a genre of research in which the iterative development of educational products[1] provides the setting for scientific inquiry. In this genre of research, educational product development serves as a case of that which is being studied. The research is framed to inform product development both proximally and distally. From a proximal perspective, the research informs the internally-focused specific product design. From a distal perspective, issues related to the design task are problematized and characterized with an external orientation; that is, in such a way that they exemplify a manifestation of the phenomenon to be studied and could therefore be informative to others interested in that phenomenon. By aligning these two perspectives, findings from carefully-conceived and well-structured research may contribute to both product improvement and scientific understanding simultaneously. Design research is sometimes confused with research-based design. Both are concerned with a design process that is fed by both theoretical and empirical inputs. However, design research is also undertaken in such a way as to yield scientific understanding that can inform the work of others. This has implications for nearly every part of the process.

Among the wide variety of backgrounds, many of those active in pioneering and applying educational design research seem to have arrived through an interest in psychology, the learning sciences or instructional design. Although most design studies are carried out in multi-disciplinary teams, few researchers have strong conceptual understanding of the marriage between the design discipline and scientific research traditions. Rather, most have strong exposure to either (a) research methodologies or (b) instructional design methods, theories and/or practices. In this contribution, we present an example of a rare program that aims to prepare graduates for careers as research-based designers and/or design researchers.

Educating the empirically-grounded design professional

4

The University of Twente proudly describes its educational programs as facilitative of the research and design orientation, as indicated on the website for prospective students, “You will be trained to be a professional - someone who can develop valuable knowledge and is able to find practical applications for that knowledge”. The goal of teaching students about research, design and organizing these processes is held across most disciplines at this university.

Within the field of education, the University of Twente offers both bachelor and master programs to educate the empirically-grounded design professional. The master-apprentice pedagogical approach is prominent throughout these programs, which revolve around a set of ‘design studios’ and supporting coursework. Inspired by the Studio Experience at the University of Georgia (Rieber, 2000; Rieber, Orey, & King, 2006), as well as research confirming that ‘design studio projects’ are among the most effective routes to educating novice designers (Demirbas & Demirkan, 2007), Twente’s students develop their competencies through authentic tasks, both individually and collaboratively, supported by modeling and coaching from their instructors. Students are increasingly expected to act as independent, self-regulating, autonomous professionals. Across the curriculum, education in these programs has been shaped by four main concepts. The remainder of this section briefly describes each main concept, and gives examples of how that idea has been translated into practice.

1. Initiation in professional roles

Twente’s education programs are designed to prepare graduates for the roles of academically qualified professional educational designer, researcher, and advisor. Students are expected to acquire theoretical knowledge; to build design/development and research/evaluation expertise; and to develop the creative ability to choose and apply a combination of those skills.Their assignments require them to fulfill design, research, and consultancy tasks in authentic situations, whereby some contexts are familiar and others are new to them. The designer and advisor orientations are proportionally stronger in the bachelor program whereas graduate students have a greater focus on research.

Working on authentic assignments helps students bridge theory and practice. Students learn design skills through working on authentic assignments (mostly in groups), which are intentionally varied. Assignments examples include the design of: multimedia educational products; written advice for organizational (re)structuring, school policy or human resource development policy; or design project grant proposals. The passage below is excerpted from the course description for a mandatory second-year course (Endedijk & Hendriks, 2012).

In this course educational design within the domains of Organisation and Management, and Human Resource Development (HRD) is put central. Design activities in these domains focus on the organisation and management conditions that will result in organisational goals. Such issues usually have a complex and context specific character. This can be the entire organisation or a specific part of functions of the organisation. Some problems are easier to solve than others. In this course, design processes and models within the area of organisation and management and HRD will be discussed. The design type, or the product within this area, will also be discussed. By guest lectures and analysing previous final assignments, specific characteristics of designing in this area of profession and the different stakeholders in this process will be defined. Also the specific design context and available characteristics of the organisation are defined, like the size, the type, the history and the current culture, specific organisation processes and the prevailing laws and rules. Students work in groups of three or four on an assignment involving a current organisation or policy case, for which they need to write an advice. This concerns a problem in normal education or in the area of business trainings, or HRD. This course prepares students for a role as a consultant or advisor, but also a researcher and designer. The assignment will be executed in a real organisation, so the advice not only fits to the content, but also suits the organisation.

2. Focus on instrumental and alternate design approaches

Influenced by its behaviorist roots, the body of literature on educational design generally reflects a limited view of design approaches, characterized by a higher prevalence of instrumentalism than is actually found in design practice. Analyses of professional design practices have shown that design approaches are much more diverse than suggested in literature, due to differences in the kinds of design products to be created, design contexts, and the designer’s personal preferences and amount of experience. Based on empirical study of design practices, Visscher-Voerman and Gustafson (2004) distinguish four design paradigms and underlying rationalities: instrumental or ‘planning-by-objectives’; communicative or ‘communication to reach consensus’; pragmatic or ‘interactive and repetitive try-out and revision’; and artistic or ‘creation of products based on connoisseurship’.

Twente’s program emphasizes a variety of design approaches, reflecting not only what is found in literature, but also in actual design practice. At the same time, experience has demonstrated that novice designers without any experience, profit most from learning to apply a sequential and structured problem-solving design approach. Therefore, it is considered important to use the instrumental approach as a basic ‘pedagogical tool’. For this reason, initial exposure to design ideas emphasizes an instrumental orientation because it provides novices with clear guidelines and steps for all kinds of design activities that could be conducted in a variety of design processes and contexts (Visscher-Voerman, 1999). It can also serve as a baseline against which theoretically different and innovative notions can be discussed. Over time, students are taught approaches to design challenges from a variety of perspectives, and instruction is specifically geared to develop sensitivity to different design process options and become more flexible in choosing one approach.

This aspect of the program in Twente has been strongly influenced by the works of Romiszowski and Plomp. Romiszowski (1981) offered sets of heuristics for decision-making during instructional design projects in his book, Designing Instructional Systems. He distinguished among three dimensions of design methodology (the nature of the design activity; the phase of the design process; and the system level), and also discriminated between approaches for an ‘education context’ or a ‘training context.’ An important aspect of Romiszowski’s work that is still influential today is questioning the points of departure, e.g. ‘Is instruction the solution?’

Students develop their own design approaches as they progress through the program. Earlier in the program, they are confronted with smaller, more well-defined design tasks. By the time they reach the master’s program, they are given larger, more open design challenges. For example, one required master level course centers completely around the task of developing and submitting a design proposal for government funding. They must present a design that is convincing (justified through literature); innovative (capitalizes on new ideas and new technologies); marketable (will attract the target group – inservice teachers) and feasible (as shown through budget estimates using the current personnel and resources at the university). After an initial presentation of the Request for Proposals, they work in small teams, and are offered coaching and feedback only four times throughout the entire process.

3. Introduction to generic and domain-specific practices

Studies of design practice have shown that design processes are highly influenced by various factors in the design context, such as the type of problem to be solved and the type of intervention to be designed. These factors have been incorporated into design-specific models, and as such those models are quite helpful in addressing domain-specific accents, interests, needs, and wishes regarding design approach and interventions. Design models in the domain of educational media, for instance, provide clear guidelines for user-interface design or design of web-sites, and in so doing, create a preference for pragmatic approaches.

In view of helping students develop the competencies envisaged, it is considered of major importance to not only highlight the main domain-specific design models and approaches, but to also address commonalities and differences between the various domain-specific approaches, the models, and the four design paradigms. In the Twente program, teachers from different departments share their domain-specific expertise and approaches through their courses. At the same time, a core course module features specific design and research interests, themes and activities in relation to a generic model for educational design. After students are acquainted with a generic model for educational design, a series of guest lectures is given by leaders of the following research units: curriculum; research methods, measurement and analysis; instructional technology; human resource development; and educational organization and management. Students then write papers describing key themes in each domain; discussing similarities and differences across domains; and reflecting on their findings.

4. Explicitly linking educational design and research

Since its inception, the Twente program has included courses on research methodology to help inform design methodology. Research methodology courses deal (amongst other things) with the development and use of a consistent ‘research chain of reasoning’ (cf. Krathwohl, 2004) which is the core of the aforementioned empirical cycle. Twente’s educational designers are taught to conduct a systematic investigation, for example, during initial analysis of tasks, problems, or context. This includes searching for more accurate and explicit connections of that analysis with state-of-the-art knowledge from literature; formulating a guiding problem statement; collecting data to inform understanding of the problem at hand and reasoning logically towards a solution based on design guidelines derived from the preliminary investigation.

Students are taught basic research methodologies with attention given to both quantitative and qualitative methods. Design studio projects and capstone projects provide ample opportunity to integrate research and design. For example, in the bachelor Design Studio 2, students are guided through an analysis phase and through three cycles of design and evaluation (screening, expert appraisal, and try-out) while developing a web quest. In their fifth and final design studio, students use research to inform their design work for an external client, such as:

• Twente’s Promise: An interactive program (and commensurate teacher support) for secondary level students to develop awareness and motivation for the four core principles of the Earth Charter

• University of Twente: A proposal and/or product to support uniformity and open-access concerning course evaluations at the University of Twente.

• World Wildlife Federation: A lesson chest including teacher guides that can be used in second or third grade in any Dutch primary school to promote wildlife and environmental awareness

• Dutch National Institute for Curriculum Development: A short lesson series (and commensurate teacher support) for secondary level students of economy using the context-concept approach

Educating the design researcher

5

In addition to preparing the empirically-grounded design professional, the University of Twente is increasingly concerned with not only offering (under)graduate students exposure to research, but also preparing them for careers in research. This section describes how a solid foundation of research skills is inculcated through the same four concepts mentioned in the previous section.

1. Initiation in professional roles

Several specific parts of the program emphasize the development of knowledge, skills and attitudes commensurate with the role of researcher. In addition to courses offering basic data analysis (primarily quantitative, but also qualitative), students learn research skills while working on authentic (research) tasks. For example, in their second year, students work in a research studio, analyzing data and reporting findings in a scientific article. For their final assignments, students conduct a full empirical cycle, from formulating a research question and developing a theoretical framework to designing instrumentation, collecting data, analyzing data and producing a research article. The annual collection of student articles is made accessible to all students and staff through the Bachelor Proceedings each year. Inspiring and informative, students browse the collection especially when considering applying to the program, and when gearing up for their final assignments. In exceptional cases, students are encouraged to present their papers at conferences, or submit their work to journals. The work of Ebbeler, Schildkamp and Downey (2012) is such an example. In this case, the student (Ebbeler) was supervised at a distance (Schildkamp) while conducting her study in England, hosted by the Southampton Education School (Downey). A short paper based on Ebbeler’s final bachelor assignment was presented at the annual meeting of the American Educational Research Association in 2012.

2. Focus on instrumental and alternate design approaches

This program in educational design emphasizes the importance of designing based on (a) insights from literature and (b) empirical investigations. In several courses, students are taught how to search data bases for literature and to judge the quality of the literature found. Also, they develop several analysis skills, both qualitative and quantitative, to be applied to the data they, or others, have gathered in practice. Through the program, students develop sensitivity for the fact that different research methods are suited to different kinds of research questions. Connections are made to particular research methods that might be suitable in different stages of design. For example, it might be more informative to engage in mostly qualitative, brief, in-depth studies in early stages of design, focusing on the usability of a product and more long-term, large-scale, quantitative studies towards the end of a project, gaining information about the learning effects of a designed program. The need for differing approaches is clear in light of the variation in master thesis titles (Visscher-Voerman, 2012), through which various stages of development and diverse contexts of design creation and testing, are evident:

• Learning support for management development in a health care setting

• Construction of a pre-math test applying the dynamic programming decision method

• The issue of fidelity: What is needed in 3D military serious games?

• Best design to stimulate learning professionals in an online self-directed learning environment

• Measuring effects of guided questioning support on inquiry performance in ZAPs

• Improving the quality and outreach of technical and vocational education through ICT integration: The case of Ethiopia

3. Introduction to generic and domain-specific practices

In the methodological courses, students are provided with generic research tools, both quantitative and qualitative. While conducting a specific assignment in practice, they are coached to set up their research so that it can build on existing domain-specific insights. For example, generic recommendations may be given about conducting interviews, related to e.g. contacting respondents, question framing techniques or member-checking. In addition, specific guidelines are usually given during field assignments, such as, what works well (or not) in primary school settings, when is a good time to approach teachers, what teachers are likely to expect, and so on.

4. Explicitly linking educational design and research

The Twente program also attends to the design activities that are inherent in the research process. For example, data collection instruments must be designed; a treatment in an experiment must be designed. Such designs must of course be well-argued and clearly described. Another explicit link with research projects stems from those that are explicitly focused on gathering knowledge about ‘promising practices’ or ‘high-quality products’ that could serve to feed design work on new projects. For example, a research question such as ‘What are characteristics of an effective program for life-long learning for target group X in setting Y?’ may result in design heuristics that could be used in new designs of future life-long learning programs. The master’s thesis produced by Fickweiler (2007) provides a detailed account of the design and implementation of curriculum materials for use by un(der) qualified educators in Indian urban slums. It demonstrates the tight connection between design and research, yielding both a concrete product and guidelines for similar products in similar settings.

Observations

6

As stated previously, a solid understanding of the marriage between the design discipline and scientific research traditions is not common among designers, nor among researchers. We now offer a critical assessment of Twente’s Education program in general, as well as the extent to which it fills this void. When looking at the current program, several observations can be made. Firstly, we notice that in design courses, students are exposed to design and evaluation more than to implementation. This can be explained by the fact that course projects are usually too short for students to be involved in the actual implementation process. Instead, students often write implementation plans that guide their design. However, actually seeing a design through to implementation (and observing what happens) would be a powerful mechanism for meeting the goals of the program.

Secondly, the research courses delivered in the program expose the students to traditional research methods with a relatively large time investment in quantitative research, such as quantitative data gathering methods (surveys) and analysis (SPSS). This is justified by the fact that students consider these subjects rather difficult to master, and foundational skills are necessary. However, students often experience difficulties understanding the relevance of these research approaches vis a vis their design activities. Broadening the repertoire in the methodological toolkit to better balance both qualitative and quantitative methods might serve the students well, and also provide opportunities to link research and design skill sets more explicitly, (e.g. in capstone assignments).

Thirdly, a large aspect of the ‘academic’ nature of this program is exemplified by the considerable emphasis on reflection. Reflection can serve several purposes: (a) through proactive reflection, it can be seen as an opportunity to optimally shape a situation and pre-consider design solutions; (b) it can help make designers become aware of their own individual strengths, shortcomings, interests, and basic assumptions; and (c) it provides the design researcher opportunities to think over their work and its outcomes, in order to produce design heuristics. While the first two functions are explicitly featured in the Twente program, this last element could be strengthened, especially in the current design-oriented courses. At present, students are taught to critically reflect on what went well and wrong in a given design or research situation and how that can be explained by theory. However, comparatively little emphasis is given to the kind of initial positioning of a project that could eventually yield learnings for use in other, similar, circumstances (design heuristics). While admittedly, theory-forming is more complex than simply collecting related design heuristics, such reflection is an important step in understanding the cause and nature of a certain phenomenon. One course (which, course evaluations demonstrate is highly appreciated by students) is entitled, systematic reflection. Here, students are guided through critical reflection processes with the goal of sharpening insights. It could be advisable to extend the work in this course toward design heuristics.

However, as the observations above imply, there is still room for improvement. Perhaps it would be more fitting to say that, in our program, the relationship of design and research currently better resembles that of engagement, than marriage. The common ground and mutual admiration are apparent. However, what might be done to increase the commitment to supporting each other?

Discussion

7

Several steps can be recommended for enhancing the cross-fertilization of research and design orientations in Twente’s education program. For one, precise clarification in coursework could emphasize the fact that the research chain of reasoning also reflects academic skills which are vital to educational engineering processes. Moreover, it would seem prudent to devote increased attention to the aforementioned linking science, through which the empirical and regulative cycli come together: design research.

The generic model depicts two main outputs from an intertwined empirical and regulative cycle, a practical one and a theoretical one. Related to the proximal orientation mentioned previously, the practical output is a designed intervention, which may be a process, a product, or (most often) a combination of the two. Examples of interventions Twente students might (partially) create include: learning objects; (electronic) learning environments; learner workbooks; teacher professional development programs; teacher guides; school data management systems. The intervention matures with the completion of each design research cycle. The model shows the process as iterative but does not prescribe one set pathway for each iteration; rather, the model depicts many potential routes that could be taken. Implementation considerations play a role throughout the entire process, typically increasing over time. From the distal perspective, the theoretical output of design research is an empirically founded set of design heuristics which can inform similar endeavors. Such heuristics underpin design and are refuted, validated or refined based on the research findings. These too, mature with each design research cycle.

Due to the explicit intertwining of the design and research work into one, integrated cycle, it would seem that this model may serve as a starting point for re-thinking several elements of the Twente education program. For example, the processes of implementation and diffusion are prominent, as is the notion that reflection (with the aim of generating design heuristics) is integral to the overall process.

We close with four brief recommendations for enhancing Twente’s education program, inspired by the generic model for conducting design research in education, above. First, we consider chunking: breaking down (research or design) tasks into well-defined sub-units. In accordance with the conviction that students benefit from exposure to authentic design and research work, one of the skills staff and students might need to further develop is that of chunking their ongoing research and design (sub)tasks into course- or assignment-sized chunks (cf. McKenney & Reeves, 2012). This approach could facilitate greater student participation in ‘real’ research or design endeavors.

Second, we recommend that students be offered opportunities to broaden their (research) methodological toolkits. Specifically, we think that students would benefit from a richer variety of skills for both quantitative and qualitative research. As stated previously, researchers as well as designers need to be creative. But researchers can hardly think creatively about how to study a phenomenon if they have a limited notion of how this can be done. As the platitude goes, “When the only tool you have is a hammer, all the world starts to look like a nail”. A solid grounding in myriad methods for inquiry (including the ‘rules’ that govern each method) is certainly necessary for conducting sound design research. This is because, by nature of the approach, design research involves asking and answering different types of questions at different stages of inquiry (e.g. questions to probe a problem, understand how a solution works, or assess the quality of an intervention).

In addition, we consider positioning research so that design heuristics can be produced. In the design studios, and in reflection courses, attention should be given not only to the proximal orientation of solving the educational problem at hand, but also to the distal orientation: What – outside of the immediate context – makes this problem worth solving in the first place? Why is this design challenge interesting? If students are conducting project work alongside or within authentic research work, it should be possible to coach them toward articulating the theoretical contribution of their design work (i.e. design heuristics). Recent work by Sandoval (2013) may be helpful in this regard; he provides a model for mapping and testing conjectures through educational designs. Linking to authentic research projects also means that ad hoc design cases would have little place in the curriculum, as they might be more difficult to justify (from a scientific perspective) than those flanking a well-established research agenda.

Finally, we consider the notion of societal relevance. The societal relevance of research in general and educational research in particular has been a topic of much discussion in recent years. Starting in 2006, the Dutch NWO established ERIC: the council for Evaluating Research in Context, to increase awareness of and to help assess the societal relevance of research. Recently, John Q. Easton, the new director of America’s Institute of Education Sciences (IES) announced the priority of moving beyond ‘What Works’ to sharpening the field’s understanding of how the research-and-development process works in education; citing the need for research to be relevant and usable as key motivators for shifting the focus. Viewed by many to be a viable route to increasing the societal relevance of research, introducing design research as a prominent paradigm in Twente’s program would ushers in increased consciousness for relevant and useful research, along with the skill set to engage in this kind of inquiry.

Closing remarks

8

We hope that the ideas presented in this paper may serve to fuel discussions that could lead to improvements in designer education in general, and Twente’s education program in particular. We feel that some aspects may be addressed by a more prominent emphasis on design research, a science which integrates the empirical cycle and the regulative cycle. In so doing, we hope to develop a broad methodological repertoire while also inculcating a socially-conscious value system in our next generation of educational designers and researchers. Ideally, this understanding would not only feed the educational programs at the University of Twente, but serve the needs of the ISDDE community, as well.

Footnote

9

References

10

© ISDDE 2013 - all rights reserved