This retrospective essay covers the period from 2001-2008, during which the research group at Utah State University (USU) focused on designing, developing, and evaluating a National Science Digital Library (NSDL.org) web-based service, called the Instructional Architect (IA.usu.edu). Later in this period, the focus was on disseminating the IA service in school contexts by developing and implementing formal and informal teacher professional development opportunities. These efforts have been funded by a series of National Science Foundations grants.
This essay is presented as three sections. In the first section, we describe our efforts to build a simple software system, the Instructional Architect, deploy it with users, and integrate it with the NSDL core technical infrastructure. In the second section, we describe our efforts to better understand the target context of educators, and to develop sustainable and scalable teacher professional development models. The final section reflects on how the IA fit within the NSDL program. Each section also includes a subsection describing evaluation strategies.
This essay also reflects shifts in our thinking over this period. Early efforts reflected a kind of technological determinism (i.e., ‘if we build it, they will come’). This eventually shifted to a more socio-technical approach. An unspoken assumption of early work was that teachers and their students would access and use such technologies in unproblematic and seamless ways. Unfortunately, the history of educational technology suggests that this is seldom the case (Cuban, 2001). Instead, after spending time with ‘real’ people (teachers and their students) in ‘real’ contexts (classrooms), it became clear that we needed to better understand the complex ways in which systems cross institutional boundaries (Agre, 2003).
The Instructional Architect
The Instructional Architect (IA) is an end-user authoring service designed to support the instructional use of online resources in the National Science Digital Library and on the Web. The IA enables users (particularly teachers) to discover, select, sequence, annotate, and reuse online learning resources stored in digital libraries to create instruction (e.g., lesson plans, study aids, homework – collectively called IA projects). In this way, the IA is intended to increase the utility of online learning resources for classroom educators (Recker, 2006).
We begin the description of the Instructional Architect with two examples created by teachers using our tool (see Figures 1 and 2). The foreground of each figure shows one of the teacher’s selected online resources. The background shows the output of using IA: a web page containing the content created by the teacher, consisting of activities and annotations for online resources (referred to by links). Note how the level of detail in the projects varies; the project in Figure 1, intended for middle-school students, provides detailed activities for the students, whereas the project in Figure 2 (intended for kindergarten students) seems to be more of a lesson plan sketch.
As is apparent from the figures above, teacher-created projects are fairly simple. Teachers are not web designers, nor should we expect them to be. Instead, they are professionals attempting to efficiently and effectively address classroom and learning issues.
Indeed, much of the functionality of IA could be recreated with blog software coupled with a social bookmarking system. However, as previously noted, by following a user-centered design process, we believe the system better meets the basic requirements of teachers who wish to use digital library technology to quickly and easily meet classroom demands.
From the home page of the Instructional Architect, users can 1) browse projects, 2) register as a new user, or 3) login as a registered user or guest (with reduced functionality).
- Browse.Users can access IA projects by performing keyword searches or by browsing these IA projects by subject area, grade level, author’s last name, or title (see Figure 3).
- Register. Users can create a free account, which provides them secure access to their saved resources and IA projects.
- Login. After the user logs in, the IA offers three major usage modes. First, with the ‘My Resources’tool, users can search for resources in the NSDL Data Repository. Queries are sent to the NSDL REST-based search interface (Lagoze et al., 2006). Metadata records for matching resources are displayed to users in an abbreviated form (including title, author, brand, description, and date). After browsing these results and viewing resources, users can select desired resources for further use. Users can also select any Web resource including interactive and Web 2.0 content (such as RSS feeds and podcasts), and add it to their list of saved resources. Users can also organize their selected resources into folders (see Figure 4).
Second, with the ‘My Projects’ tool, users can create web pages in which they they select a look and feel for their project, input selected online resources, and provide accompanying text in order to create learning activities (called ‘IA projects’).
Finally, users can share their IA projects by ‘Publishing’ them and setting permissions on them, such as a) user-only view, b) users and their students (student view), or c) public view (anyone browsing the IA site). Users can also add basic metadata about their IA projects, including subject area, grade level, and core curriculum standard. These are then used to support browse and search of existing IA projects, as described above.
Early design and evaluation efforts (2001-2002) focused on measuring usability and utility, referred to as ‘developmental evaluation.’ This included a needs assessment and interface design and development of the IA. Each design cycle was followed by an evaluation that helped inform the design of the subsequent phases. Participants included graduate students as early testers, pre-service teachers, and expert teachers.
Methods included literature reviews, focus group interviews, and expert review of prototype interfaces, early testing by members of the target audience, and analysis of code changes by constituents. Early recommendations included a search tool, and combination tool, and a reflection tool. In addition a more in-depth case study approach was conducted with 8 in-service teachers in Utah. They provided input on how Internet resources were currently used in their teaching practice, and contributed to the needs assessment.
At that time, the design, development, and evaluation of our project were hampered by the fact that the NSDL was co-evolving with our project. This meant that technical standards were in flux, resulting in system instability. In addition, the library collections were simultaneously being seeded and grown, resulting in uneven and sometimes sparse holdings. The latter caused no small amount of frustration among our classroom teachers as they attempted to search for interesting and relevant learning resources. To address this problem, we worked with other educational digital libraries, including SMETE.org, DLESE.org, and the National Library of Virtual Manipulatives (nlvm.usu.edu) to devise means to query their metadata. We were able to greatly benefit from the maturity of these projects.
In 2005-2006, as the NSDL gained maturity, our project worked on tighter technical integration with the NSDL Core Technical Integration. This included queries of the NSDL search service, pilot implementation of community sign-on (CSO) via Shibboleth, and co-branding. At the same, however, the NSDL as a whole, seems to suffer a bit from ‘wheel reinvention.’ For example, many projects are developing tools with similar functionality to the IA. Partly due to the ‘not invented here’ syndrome, projects wanted functionality that differs slightly from what the IA provides. Hence, they found it easier to simple built their own. In general, the NSDL as a whole needs to consider strategies that avoid ‘tool silos’.