Interview by Catherine Eberbach :: May 2008
Eberbach: Was there anything discussed during this inquiry group that particularly excites you or may help you to think differently?
Falk: This is the first time that I saw the NSF Framework for Evaluating Impacts of Informal Science Education Projects. I found this very interesting, although it's unclear if the framework is intended to have broader impact beyond informal science education. The framework probably works well for thinking about learner impacts, but it seems to be a bit of a forced fit for professional development.
Eberbach: What seems particularly forced?
Falk: The impact categories awareness, knowledge, attitudes, and behavior seem all right, but other things that affect professional development are not included. Perhaps facilitating collaborations and promoting dissemination should be in the framework. It just doesn't feel like the right categories for evaluating online professional development. I imagine that the impact measures and their definitions may have been made with learners in mind and then retrofitted for professional development.
Eberbach: How would you describe your professional field? And what are some of the professional development challenges that your field faces?
Falk: I think that I traverse several fields, but that I am primarily a researcher and evaluator in education, and science education in particular. In the field of science education, the challenges include finding ways to learn from prior research, finding excellent ways to synthesize research that is conducted with different populations and with different methodologies, figuring out when those syntheses should not be relied upon because the populations and methodologies are too different, and finding what can be generalized — or at least replicated. We need to get better at that.
Eberbach: And in terms of educational research?
Falk: We need to look carefully at research settings and questions, and how they lend themselves — or not —to different research methodologies. We should not be overly enthusiastic about randomized controlled research designs when they don't fit with the settings that we're studying.
Eberbach: Are there special challenges for online professional development communities?
Falk: It used to be that websites served to only broadcast information, or to only serve a community, or to only provide digital library resources. There are very few websites that are this single minded anymore. Even the New York Times includes discussion areas for its article. Who would have thought of the New York Times as a community site?
Today, there are hybrid websites. We need to rethink those early definitions and create new concepts, especially in terms of community — a word we use too loosely. Just commenting on articles in the New York Times website doesn't make me a member of the New York Times community. Websites that invite interaction, discussion, posting, or dialogue should not be equated with community. Communities don't exist online. Communities exist and are then supported by online environments. Creating a website does not create a community.
Eberbach: Your point is intriguing. It does seem to be an unspoken assumption that you create a community when you design a website.
Falk: It's a false assumption to think that you create a website and then you create a community. Communities exist and are very busy accessing the resources and websites that are most meaningful for them as individuals, as groups, and as nested communities. Often, communities are loosely defined entities that have many crossovers. For instance, a person who is part of an informal science education community may be part of their university community, and their physics community, and their museum community. They may also be part of a community focused on cardiovascular disease because they suffer from heart disease. Yet we have this notion that creating a website means we will create a community.
Online sites that serve professional communities often benefit dramatically from face-to-face interactions. That might be a challenge for CAISE because it wants to serve the informal science education field at large, rather than only NSF-funded projects. This means there will be limited opportunities to integrate its website with face-to-face events.
Eberbach: You mention that people are very loose with the word community. How do you define community? How has your thinking about community changed?
Falk: My thinking about community has indeed changed. My initial definition of community came from the work of Lave and Wenger, but my notions have changed with reference to communities and online websites. Ten years ago, when websites were new, communities were hungry for websites where they could interact online. In a very short period of time, peoples' engagement and involvement on the Internet has grown so dramatically that now we're inundated with online this and online that.
We each belong to multiple communities so that no single online website can claim our complete loyalty. Whereas we once yearned for connections, we now need tools to filter too many communications or connections. The goal now should not be for websites to measure success by drawing people in, but by serving an individual's best use of the website. There needs to be more serious consideration to having RSS feeds and tools that can be exported to PDAs, to iCal, or to iGoogle so that individuals can harvest things from multiple sites in ways that aggregate their own data.
Eberbach: From the other side of the coin, however, it is also essential that community websites get people to be content contributors. What is your experience with creating content?
Falk: It's a real challenge to get people within a community to share material in addition to posting to discussions. Even in the most successful communities, the ratio of people who post to those who read is very small — something like 15% people posting is a lot. People just don't have the time, nor do they have the motivation for more. In order to be an active contributor, they have to have decided that contributing is deeply connected with their work and that it is already a part of their work.
This is one reason why MSPnet (Math Science Partnerships Network) is project-centered. In a positive way, project teams can build this within their work so that their collaborative work in the project, their collaborative writing of a report is done with collaborative tools on MSPnet and then exists as a file on MSPnet, and then when you're ready to share it, you could just share it with the public through MSPnet.
The real challenge is to design a website so that it becomes an integrated way of accomplishing people's work. People don't have time for add-ons, so embedding the work that someone is doing anyway is a real value.
Eberbach: At some level, this group work is actually the work of the field, is it not? Someone is going to a website so that they can find materials and possibly use these for their own purposes.
Falk: Exactly, but I'm talking about the production of knowledge and not the conception of knowledge. So, yes, the field will come and browse through a project's work. Nevertheless, you want them to browse through a rich set of resources. One of our biggest challenges is getting those resources developed so that they can be shared.
Eberbach: You've described MSPnet as using a project structure so that people contribute content that is already part of their workflow. How are you doing that?
Falk: Well, the goal is to embed MSPnet in your workload and we're working so that it becomes more so. For instance, if you have a Math Science Partnership that has a leadership team that has multiple institutions that are geographically dispersed and they need to share files. And sending them via e-mail and having them not compile is not convenient.
So they can create a working group for themselves and use a calendar to schedule their time, and they can use a survey tool to figure out when meetings are best within a working group, and they can use a threaded discussion, and they can use the webinars — a web conferencing tool — to conference with each other. All this could be totally private, so that not only can no one see these artifacts. People outside the group don't even know that the group exists.
The purpose of this group is not to serve the field, but to serve their project. However, as this group continues to evolve, they might begin to craft an article together. Once it's a working draft, they may decide, "Oh gee, we'd like to share that and get feedback". But they may not want to share it with the fields at large, but would feel comfortable sharing it with the rest of their project. So then they could limit the visibility of that item to their project only. And then they might get feedback. They might fix it in this working group and then decide it's good enough to share other projects and maybe the public at large, so the fields at large. Then they would post share it with the MSPnet Hub Library, it would then appear in the library, and it would get much more visibility. It would get announced in a newsletter that would go out to 4,500 people that week. And then some portion of those 4,500 people would forward that e-mail to other colleagues and friends.
So now you have a dissemination vehicle with a lot of people looking at your work. And because authors always maintain copyright and have full control of the work that they submit, they post it as a working draft and ultimately submit it for publication. So there is a workflow that starts within a project, creating a pathway for ideas and resources to be developed collaboratively and then shared more and more publicly.
Eberbach: This model something to strive for, and with some modification, might work really well for the informal science education community.
Falk: MSPnet is designed with a central hub, and then that central hub — with its interactive functionalities — is then replicated for every single project. Each project has its own interactive site, which that could use to build their own library, their own resource center, their own working groups, and their own database. Then those things can be leveraged or they can post them to share in the hub, and they can take things from the hub to their project space. The reason that we built these project spaces is that most projects are very large and may involve multiple universities and multiple school districts that may be geographically dispersed. Each represents a huge community with their own sub-communities and need their own community spaces. Many ISE projects are smaller so that I'm in not sure whether they would make use of a full interactive space.
Eberbach: How did MSPnet get started? What needs did it intend to fulfill? And how were those needs identified?
Falk: MSPnet started as a one-year technical assistance grant so that we could complete a needs assessment. During that year we contacted and spoke with members of twenty MSP projects. These were the first cadre of projects funded by NSF. We tried to find out about their online needs, what they would use, and why they would use it. This was a very complicated process because people, who haven't tried things online, don't realize that they could have a need that could be solved by things online. So sometimes they won't name it, or other times they'll name it and then they won't really use it. So, it's not like it's a direct, you find out these needs and then that perfectly works.
We did a needs assessment through interviews and surveys, and we used some prototypes. The needs assessment was informed by a lot of work that we had already done for some local system initiatives. We were able to build very quickly a prototype website that we were able to have them to react to. It's very hard to have people react to the abstract. We used that year to build a prototypal website and to get people's needs and reactions to it, and that gave us a good head start. When MSPnet was funded, we were already well prepared with a website, a large database of articles that we leveraged from the local system of change work, and a lot of functionality that we leveraged from earlier projects. Our biggest challenge was figuring out what the architecture for a hub with nested communities looked like and how to populate it.
Eberbach: More specifically, were these needs intending to fulfill were to support the projects or the professionals participating in the projects?
Falk: One goal was to broadly help the NSF-funded Math Science Partnership (MSP) program, which was experimental at its inception. They were building these very large projects that were going to use higher STEM faculty to enhance teachers in the elementary, middle, and high school levels. It was not at all clear under what would circumstances that works, when it works what the degree of higher ed involvement needs to be, what the cultural difference between higher education and K-12 are that make that difficult. So MSPnet was intended to collect and harness this experimentation and the knowledge this generated, and then to share its findings with the fields at large.
In that way, MSPnet doesn't serve a particular project but it serves the projects in the larger program to be visible to each other, to the NSF funders, and to the fields at large. Many times projects live in little silos with their own website, which makes it very difficult to identify the cumulative knowledge that a program has developed. One achievement of MSPnet is building a digital library. We now have over 130 papers authored by PIs, co-PIs, and evaluators about the MSPs. That's huge. Someone from the outside who wants to evaluate what this program achieved could start with this corpus of papers. What were they writing about? What were their problems? The MSP program also has projects that are funded to look across the projects at different aspects of evaluation and they depend on MSPnet. These are heavy users of the website because they want to see what's going on across projects. MSPnet helps to make the work of individual projects more transparent.
Defining what the field is at large is a real question because there are not that many people who have the kind of resources to go out and transform entire school systems or districts, with multimillion dollar grants.
Eberbach: This is the hard part. How do you know that a website is impacting the field? What are some of the ways that you've considered conceptualizing and measuring impact?
Falk: Great question. Our field — this field of creating massive educational change by building partnerships between universities and K-12 districts — isn't quite a field. It's more of an idea and MSPnet is a test of the idea. The actual fields might be math professional development or science professional development for educators.
Translating a professional development program to a field is not easy. One thing we ask is: Do we have rich content and resources for math and science professional development on the website? When we started, our users were primarily PIs and co-PIs that were interested in policy reports about issues such as the changing face of equity or how math and science might affect dropout rates. And our resources were weighted in that direction. But we've learned from surveys that a lot of teachers also use the site and they are more interested in resources that work in classrooms. We have very, very diverse audiences and we serve multiple fields. So, there is no "the field." Creating rich content and resources for all of the "fields" using the MSPnet is critical.
Eberbach: How do you even measure — or do you measure — change or impact within teacher practice?
Falk: You don't. We don't.
Eberbach: Some might view this as a limitation to evaluating the impact of online professional communities.
Falk: Right. It would be akin to saying, "I provide a rich resource to teachers, so one could explore what a teacher has learned from that resource, and then one could explore whether that changed teacher practice, and then one could explore whether that change in teacher practice effect a change in student learning, and then whether that change in student learning effected a change in students' scores."
You have to set some parameters and decide where you will start and where you will end. However, we do look at whether participation on MSPnet affects other changes. Does participation improve collaborative practice within projects? Are projects successful in posting the work and sharing what they're doing? Can projects tell their stories better to their stakeholders and to the collective field at large? Can the program capture more of the work and the effort of the project over time so that they could later reflect upon what they done?
Eberbach: What do you think online communities will look like in five to ten years?
Falk: The brilliance of Facebook is the recognition that if each person is central in their community and is never on the periphery. We've talked about people who are central, people who are peripheral; we've talked about producers and we've talked about consumers. The best way to get somebody active and being a producer in their community is if they feel it's their own and they've created it. People will be part of multiple communities and there will be the tools on the web for them to be front and central in this nested world of their communities, and to be able to contribute to them and to control them. It isn't going to live by the person putting himself or herself in this community or that community. This is one way in which we are thinking about the redesign of MSPnet. By making the site so that it is more individual-centric, individuals may control their communities and their activities on the site, who they speak to, and preferences about the type and frequency of getting notifications.
Eberbach: It seem likely that putting individuals at the center of a community — rather than putting the project at the center — will really challenge web designers and developers.
Falk: I suspect that many websites started as projects. The catch is that projects do not contribute. Projects do not communicate. Projects do not read. People contribute. People communicate. People read. So what does a project mean? A project is the collective talent of a multiplicity of people who are working and engaged in the project. We need to design for that.
"Websites that invite interaction, discussion, posting, or dialogue should not be equated with community. Communities don't exist online. Communities exist and are then supported by online environments. Creating a website does not create a community."
"The goal now should not be for websites to measure success by drawing people in, but by serving an individual's best use of the website. There needs to be more serious consideration to having RSS feeds and tools that can be exported to PDAs, to iCal, or to iGoogle so that individuals can harvest things from multiple sites in ways that aggregate their own data. "
"The real challenge is to design a website so that it becomes an integrated way of accomplishing people's work."
"Many times projects live in little silos with their own website, which makes it very difficult to identify the cumulative knowledge that a program has developed."
"Our field — this field of creating massive educational change by building partnerships between universities and K-12 districts — isn't quite a field. It's more of an idea and MSPnet is a test of the idea."
"The best way to get somebody active and being a producer in their community is if they feel it's their own and they've created it."