≡ Menu

An NSDL Retrospective: The Case of the Instructional Architect by Mimi Recker

Download PDF: An NSDL Retrospective: The Case of the Instructional Architect


This retrospective essay covers the period from 2001-2008, during which the research group at Utah State University (USU) focused on designing, developing, and evaluating a National Science Digital Library (NSDL.org) web-based service, called the Instructional Architect (IA.usu.edu). Later in this period, the focus was on disseminating the IA service in school contexts by developing and implementing formal and informal teacher professional development opportunities. These efforts have been funded by a series of National Science Foundations grants.

This essay is presented as three sections. In the first section, we describe our efforts to build a simple software system, the Instructional Architect, deploy it with users, and integrate it with the NSDL core technical infrastructure. In the second section, we describe our efforts to better understand the target context of educators, and to develop sustainable and scalable teacher professional development models. The final section reflects on how the IA fit within the NSDL program. Each section also includes a subsection describing evaluation strategies.

This essay also reflects shifts in our thinking over this period. Early efforts reflected a kind of technological determinism (i.e., ‘if we build it, they will come’). This eventually shifted to a more socio-technical approach. An unspoken assumption of early work was that teachers and their students would access and use such technologies in unproblematic and seamless ways. Unfortunately, the history of educational technology suggests that this is seldom the case (Cuban, 2001). Instead, after spending time with ‘real’ people (teachers and their students) in ‘real’ contexts (classrooms), it became clear that we needed to better understand the complex ways in which systems cross institutional boundaries (Agre, 2003).

The Instructional Architect

The Instructional Architect (IA) is an end-user authoring service designed to support the instructional use of online resources in the National Science Digital Library and on the Web. The IA enables users (particularly teachers) to discover, select, sequence, annotate, and reuse online learning resources stored in digital libraries to create instruction (e.g., lesson plans, study aids, homework – collectively called IA projects). In this way, the IA is intended to increase the utility of online learning resources for classroom educators (Recker, 2006).

Two Examples

We begin the description of the Instructional Architect with two examples created by teachers using our tool (see Figures 1 and 2). The foreground of each figure shows one of the teacher’s selected online resources. The background shows the output of using IA: a web page containing the content created by the teacher, consisting of activities and annotations for online resources (referred to by links). Note how the level of detail in the projects varies; the project in Figure 1, intended for middle-school students, provides detailed activities for the students, whereas the project in Figure 2 (intended for kindergarten students) seems to be more of a lesson plan sketch.

Screenshot of an IA project page aimed for middle-school students

Screenshot of an IA project page aimed for kindergarten students

Figure 2: Screenshot of an IA project page aimed for kindergarten students

As is apparent from the figures above, teacher-created projects are fairly simple. Teachers are not web designers, nor should we expect them to be. Instead, they are professionals attempting to efficiently and effectively address classroom and learning issues.

Indeed, much of the functionality of IA could be recreated with blog software coupled with a social bookmarking system. However, as previously noted, by following a user-centered design process, we believe the system better meets the basic requirements of teachers who wish to use digital library technology to quickly and easily meet classroom demands.

System Description

From the home page of the Instructional Architect, users can 1) browse projects, 2) register as a new user, or 3) login as a registered user or guest (with reduced functionality).

  1. Browse.Users can access IA projects by performing keyword searches or by browsing these IA projects by subject area, grade level, author’s last name, or title (see Figure 3).

    Browse IA Projects

    Figure 3: Browse IA Projects

  2. Register. Users can create a free account, which provides them secure access to their saved resources and IA projects.
  3. Login. After the user logs in, the IA offers three major usage modes. First, with the ‘My Resources’tool, users can search for resources in the NSDL Data Repository. Queries are sent to the NSDL REST-based search interface (Lagoze et al., 2006). Metadata records for matching resources are displayed to users in an abbreviated form (including title, author, brand, description, and date). After browsing these results and viewing resources, users can select desired resources for further use. Users can also select any Web resource including interactive and Web 2.0 content (such as RSS feeds and podcasts), and add it to their list of saved resources. Users can also organize their selected resources into folders (see Figure 4).

    My Resources

    Figure 4: My Resources

Second, with the ‘My Projects’ tool, users can create web pages in which they they select a look and feel for their project, input selected online resources, and provide accompanying text in order to create learning activities (called ‘IA projects’).

Finally, users can share their IA projects by ‘Publishing’ them and setting permissions on them, such as a) user-only view, b) users and their students (student view), or c) public view (anyone browsing the IA site). Users can also add basic metadata about their IA projects, including subject area, grade level, and core curriculum standard. These are then used to support browse and search of existing IA projects, as described above.

Evaluation Strategies

Early design and evaluation efforts (2001-2002) focused on measuring usability and utility, referred to as ‘developmental evaluation.’ This included a needs assessment and interface design and development of the IA. Each design cycle was followed by an evaluation that helped inform the design of the subsequent phases. Participants included graduate students as early testers, pre-service teachers, and expert teachers.

Methods included literature reviews, focus group interviews, and expert review of prototype interfaces, early testing by members of the target audience, and analysis of code changes by constituents. Early recommendations included a search tool, and combination tool, and a reflection tool. In addition a more in-depth case study approach was conducted with 8 in-service teachers in Utah. They provided input on how Internet resources were currently used in their teaching practice, and contributed to the needs assessment.

At that time, the design, development, and evaluation of our project were hampered by the fact that the NSDL was co-evolving with our project. This meant that technical standards were in flux, resulting in system instability. In addition, the library collections were simultaneously being seeded and grown, resulting in uneven and sometimes sparse holdings. The latter caused no small amount of frustration among our classroom teachers as they attempted to search for interesting and relevant learning resources. To address this problem, we worked with other educational digital libraries, including SMETE.org, DLESE.org, and the National Library of Virtual Manipulatives (nlvm.usu.edu) to devise means to query their metadata. We were able to greatly benefit from the maturity of these projects.

In 2005-2006, as the NSDL gained maturity, our project worked on tighter technical integration with the NSDL Core Technical Integration. This included queries of the NSDL search service, pilot implementation of community sign-on (CSO) via Shibboleth, and co-branding. At the same, however, the NSDL as a whole, seems to suffer a bit from ‘wheel reinvention.’ For example, many projects are developing tools with similar functionality to the IA. Partly due to the ‘not invented here’ syndrome, projects wanted functionality that differs slightly from what the IA provides. Hence, they found it easier to simple built their own. In general, the NSDL as a whole needs to consider strategies that avoid ‘tool silos’.

Comments on this entry are closed.

  • Lois McLean March 8, 2009, 4:19 pm

    Mimi describes the problems of the “wheel reinvention” and “tool silos” that are created when projects with similar functionality and audiences develop in parallel rather than in a complementary fashion. Part of this may be due to the fact, which she points out, that technical standards for the NSDL were not in place when earlier projects such as Instructional Architect began creating tools. Are there some specific approaches that the NSDL should consider to avoid similar problems in the future? For example, could the annual RFP be more explicit in referencing what’s already been developed and suggesting directions for combining related efforts?

  • Lois McLean March 8, 2009, 4:20 pm

    Despite the often-expressed value of the NSDL Annual Meeting, Whiteboard Report, and other community activities, projects can still be unaware of very specific resources, tools, or evaluation approaches that could be adapted for their own purposes. Matchmaking efforts seemed more prominent several years ago, with events such as Tool Time. Should the NSDL again take a more active role in such activities, e.g. by sponsoring workshops for non-pathways groups, such service providers or tool builders?

  • Lois McLean March 8, 2009, 4:20 pm

    Mimi and others have pointed out the critical nature of the NSDL Annual meeting and the value of committees (such as the Evaluation and Impact Standing Committee) in fostering collaboration. In light of the planned shift from committees to work groups, how can collaboration be encouraged and sustained in practical and effective ways?

  • Lois McLean March 8, 2009, 4:21 pm

    Mimi comments that the NSDL continues to be primarily driven by technical concerns, leaving out the voices of the users, especially in the K-12 world. Do you agree? If so, how can the NSDL foster development that acknowledges and acts on the needs of that user base?

  • Kuko Ako March 24, 2009, 11:01 am

    I agree 100% with Mimi’s point (as pointed out by Lois) about the NSDL being driven by technical concerns. While technical elements are an essential component of the NSDL, attention has to shift to the needs of educators who are accessing the online resources and also on quality versus quantity. In these times of increased accountability and high stakes testing, we should try not to lead teachers astray by presenting them with tons of content that is only tangentially or topically related to the big ideas they have to teach. Otherwise they will promptly get out. So an important question is: What objective measures–preferably based on human rather than machine methods–can we present to the user to help them quickly decide whether any given resource is worth even checking out further–that is, likely to be useful to their teaching?