Field Studies in TEL
Viktoria Pammer-Schindler, Graz University of Technology, Austria
Introduction
There are two different types of field studies in technology-enhanced learning (TEL) research:
- Field studies that observe learning- or teaching-related processes in the field, i.e. in real environments (as opposed to in a laboratory). Note that while typically technology already plays some role in the environments into which TEL researchers investigate, this isn’t necessarily the case: A field study could also investigate an environment without technology in order to develop knowledge that is relevant for TEL research.
- Field studies in which researchers investigate an educational technology intervention in the field. This means that researchers place a new technology, or a new way of using technology, into a real-world environment and then study it. Within this text we refer to such field studies as “interventionist field studies”. Note, however, that most scientific texts don’t use this name, as typically it is clear from the description of a particular field study whether it tests an intervention or not.
What is the role of field studies in design-oriented TEL research?
The role of both kinds of field studies in TEL research is to understand a real-world environment (Type 1 from above) or an intervention in the context of a real-world environment (Type 2). In this section, we discuss specifically the role these two types of field studies have in design-oriented research.
First, let’s briefly define the term design-oriented research, so that this term can be related to the term design-based research, as introduced in a different chapter in this book. The goal of educational technology research could be to:
- Develop educational technology as artefact that serves learning, teaching or the administration of such processes, and with this artefact to make a contribution to practice. The designed artefact is an important output for reaching this goal.
- Develop knowledge that is important for designing educational technology: This could be knowledge about educational technology that generalises beyond a single artefact (e.g., what constitutes a good human-computer interface of an intelligent tutoring system, what data and data analytics methods are suitable to describe a learner’s progress, etc.), knowledge about methods for designing educational technology or concepts that are useful for understanding and developing educational technology. This goal doesn’t necessarily mean that an artefact needs to be designed. Sometimes, the activity of designing educational technology is part of the activity of generating such design-oriented knowledge: in this understanding, the designed artefact is a means through which research happens.
Educational design-based research and action research typically pursue both goals, but the emphasis may be different (more or less emphasis on the artefact as having an impact, more or less emphasis on contributing to the existing body of literature).
In this chapter, we use the term design-oriented research to broadly refer to research that has the purpose to generate design-oriented knowledge (Goal 2 above), no matter which approach is taken to generating this knowledge, i.e. independent of used research method or whether an actual design activity takes place. We use the term design-based research to refer to research that pursues both of the above goals, and very importantly, assigns the designed artefact with value on its own.
It could now be asked, whether there is any educational technology research that isn’t design-oriented? At the limits, this is indeed the case, when research doesn’t clarify how results inform design, design methods, or impact design-oriented concepts. For instance, you could use technology to analyse educational data in order to gain insights about learning processes, without specifying what results mean for design, or how this analytical technology could be used in structuring learning or teaching.
Further, in the context of this book, we assume that the designed artefact has some technical element. This isn’t necessarily the case in all design-based research, design research, or action research in wider educational research: (Edelson, 2002) specifically mentions “curriculum, software, professional development, school-organization, and school-community collaborations” as possible types of designed artefacts. In this paper we therefore use the term design-based educational technology research to refer to design-based research in which the designed artefact concerns educational technology.
Field studies appear in both roles in design-based and design-oriented educational technology research: Firstly, as a way to gather knowledge about a particular social context (a real-world environment that is of interest), whether there is educational technology in it already or not. Secondly, as a way to test an educational technology intervention in the field. In design-based research, the assumption is typically that interventions need multiple iterations to reach sufficient maturity for really being applied in the real-world. For design-oriented research, we argue differently: When the goal is to contribute to existing knowledge in the field, a single design iteration can be sufficient for research if (and only if!) a single design iteration generates a contribution, i.e. knowledge that wasn’t there. In order to support design-oriented educational technology research, data collection, data analysis and interpretation of results need to connect to existing state-of-the art in educational technology research in order to be able to inform research – hence, needs to refer literature that already says something about what is a good design of educational technology, or a good way of using it.
Reflection activity: When you review the two different types of goals in educational technology research, what are the goals of your own research (PhD, current research stream, etc)? To what kind of real-world environment do your designs aim to make a contribution, if any? Will – and how – answers to the research questions be useful to future educational designs (e.g., for developing curricula, lesson plans, educational technology, etc.)?
What research questions can be answered with field studies?
Field studies can serve all research functions, use all research designs, and answer all research questions that are discussed in the chapter Design Based Research.
The salient quality of field studies is, that they can give insight into the socio-technical system(s) in the targeted real-world environment: Who are the actors? What concepts and tools do they use, and how do they make use of them? What do they try to achieve? How do actors interact with each other, and what are the roles they take and the distribution of work/activities across actors? What is valued? Such knowledge can be useful as an insight prior to design (Field study of Type 1) as well as useful in relation to how an (educational technology) intervention works within the real-world environment.
Reflection activity: What aspects of your target real-world environment are unclear to you? Are there resources that you can use to collect more knowledge before going into the field (e.g., books, scientific publications, talking with selected practitioners)? What is the essence that is open – and for which you need a field study (for both types)?
What are challenges, particularly in interventionist field studies?
Field studies are always challenging to implement well, as researchers have less control over the study than in a laboratory study. Interventionist field studies have the additional challenge, that in order to accommodate the intervention, the target environment of use (the field study environment) has to adapt to the intervention – some changes will have to be made to the routine operations of actors, e.g., how learning or teaching is done.
Particularly in interventionist field studies, many things can go wrong – precisely because many elements and events in the field are outside researcher’s control, which is the whole point of a field study. It is therefore important to capture a wide range of different types of data, analyse them using a wide range of analysis methods, and to ask about different aspects of those learning- or teaching-related processes and outcomes that the research is interested in (the four below are inspired by (Kirkpatrick & Kirkpatrick, 2010), with a number of modifications):
Usage/compliance with the intervention: How do actors in the field actually use the intervention? Don’t assume that actors will use the intervention as specified by the researchers! This doesn’t have to be due to malevolence or laziness of the actors; everyone in the field will have goals that are not related to publishing a paper about the field study. If the prescribed way of using the intervention sufficiently interferes with people’s goals, they will try to find a workaround. That might also be an improvement over the way researcher’s thought the intervention should work.
Learning or teaching process: How is the learning or teaching process with the intervention, how is it different from the previous/normal learning or teaching process?
Learning outcome/performance outcome: What do learners actually learn, using the intervention? If the intervention targets professional behaviour, such as teaching practice, or the practice of managing an educational institution: What is the work outcome of study participants, using the intervention?
Impact: In all educational systems, some aspects are especially valued. These could be aggregates of indicators that measure learning or teaching processes, or learning and performance outcomes, such as the number of drop-outs, the number of graduates etc. While it is important for educational technology research to also critically reflect those indicators that already are in place, and to suggest meaningful indicators, interventionist field studies also could investigate how an intervention impacts those indicators that are in place. We are more careful in suggestion that such impact should be measured as part of an interventionist field study, as a causal influence of the intervention on this level might not always be reasonable to expect, and especially as existing indicators might not reflect the impact that a particular intervention has. However, the recommendation still is to think of what an intervention’s impact at an educational strategic level could be.
Reflection activity: If you are planning, or have carried out, an interventionist field study – which of these aspects would be interesting to you? How could you measure, usage, the learning or teaching process, and the outcome? How could you measure – even if only conceptually – how and whether your intervention has an impact on key indicators that are in use in the target environment of your intervention?