What NRC meant by "rules of the road" is a matter of conjecture, but the expected differentiation of rules according to collaboratory size is clear. Nautically speaking, "rules of the road" are

regulations concerned with safe handling of vessels under way with respect to one another, imposed by governments on ships in its own waters, or upon its own ships on the high seas. (Webster's 1989)

So defined, "rules of the road" for the collaboratory are inter- or intra-collaboratory protocols. While the functioning collaboratories identified in Phase Two of this study are, on some levels, interrelated (by common funding source, because some sites hosts several collaboratories, and because the population of Collaboratory Pioneers is very small), they have not yet evolved to inter-collaboratory sophistication. An example of intercollaboratory work might be an investigation of the properties of plasma between the DEE DIII-D Tokamak and the SPARC collaboratories. Nevertheless, as the focus of collaboratory research shifts from technological implementation to sociological and cultural aspects of the work, generic "rules of the road" that apply to all collaboratories, and perhaps between collaboratories, now seem discoverable.

 

"Rules of the road" may also be organizational and/or cultural. Culture is an overarching part of Nardi and O'Day's (1999) information ecology (comprising people, practices, values, and technology). Anthropology defines organizational culture as

patterns of shared values and beliefs that over time produce behavioral norms adopted in solving problems. Similarly, culture is a body of solutions to problems that have worked consistently and are taught to new members as the correct way to perceive, think about, and feel in relation to those problems. The sum of these shared philosophies, assumptions, values, expectations, attitudes, and norms bind the organization together. Organizational culture, therefore, may be thought of as the manner in which an organization solves problems to achieve its specific goals and to maintain itself over time. Moreover, it is holistic, historically determined, socially constructed and difficult to change. (Heck 1996)

 

To mine for the definition and attempt to discover the "rules of the road" for the collaboratory, and simultaneously identify skills collaboratory pioneers value in prospective participants, PhaseThree of this study turns to the Delphi Method.

 

The Delphi Method

Delphi is a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with complex problems (Linstone and Turoff 1975). It consists of a series of interrogations of a group of individuals whose opinions are of interest, with interrogations continuing in '"rounds" where the anonymous responses of participants are submitted to the group whole for comment until consensus, divergence, or stasis of opinion is reached. Delphi is also an interdisciplinary, intersubjective, futures research technique that allows translation of qualitative data for quantitative analysis, and is particularly useful when the field of interest is too new to have adequate historical data for the use of other methods (Lang n.d.).

The Delphi Method was developed by Kaplan, Skogstad, and Cirshick (1949) and refined by Helmer and Dalkey of the RAND Corporation in 1953 to answer the U.S. Air Force's question about the likely outcome of a Soviet nuclear attack on the United States (Linstone and Turoff 1975). Since then, the Delphi has been used many times, by most disciplines, for a variety of reasons, and under many permutations.

There are no hard and fast rules for Delphi implementation (Turoff and Hiltz, in press). Nevertheless, all Delphi Studies follow a broad procedural outline (Lang n.d.) In brief, the steps for a Delphi are:

1. The problem is identified
2. An expert panel is developed
3. The panel is presented the problem and asked to respond
4. Responses are synthesized into a series of statements
5. The synthesized statements are submitted to the panel
6. The panel responds
7. The process continues until convergence, divergence, or stasis is identified.

Delphi Studies share three distinctive features and common characteristics: anonymity of response, feedback of individual response to the group, and statistical analysis using median and dispersion. Statistical analysis is achieved by synthesizing individual participant responses and presenting them to the group in rounds as statements for individual Likert Scale (sliding scale) responses. In most Delphi Studies, participants are kept anonymous and never meet face-to-face. In all Delphi Studies, however, anonymity of responses is maintained to avert dominance of the group by influential or powerful participants, to avoid specious persuasion, and to avert unwillingness to abandon publicly expressed opinions and other social-psycho affects of face-to-face group decision situations (Welty 1971, Rosenbaum 1991, Turoff and Hiltz, in press).

There are seven basic types of Delphi: the conventional, numeric, policy, and historic Delphi (Strauss and Zeigler 1975), the derivative (when used with other methods), the pedagogical (Rosenbaum 1991), and the conference Delphi. The pedagogical and conference Delphi are executed in face-to-face situations with anonymity of responses (Rosenbaum 1991).

The Delphi Method has been used for futures forecasting, for prioritizing (Cline 1997), for community needs assessment and gathering initial or new information (Carter and Beauleiu 1992). It has been used for soliciting interpretations, predictions, or recommendations (Strauss and Zeigler 1975), for environmental scanning (Lang n.d.), for personnel and budget allocations (Kao 1997), and for participant education (Strauss and Zeigler 1975). The Delphi has been used for both normative (target oriented) and exploratory purposes (Acolyte 1995).

Delphi operates on the principle that several heads are better than one in making subjective conjectures about the future…and that experts will make conjectures based upon rational judgement rather than merely guessing (Weaver 1971 in Ludwig 1997).

The Delphi's group consensus principle rests on a 1936 study (Loye 1978) by Douglas MacGregor that formulated what came to be known as the "MacGregor Effect, " which refers to his findings that predictions made by a group of people are more likely to be right than predictions made by the same individuals working alone.

The pitfalls of the Delphi Method are well-documented (Linstone 1975, Fisher 1978, Cook 1987), and include concerns that the Delphi does not adhere to traditional tests for statistical significance, sampling errors, and randomness. Delphi has been criticized for its tendency to encourage researcher urges to discount the future, make unwarranted predictions, and oversimplify. Major pitfalls also include the illusion of expertise among the panelists, and sloppy execution (poor panel selection, superficial analysis of responses, and designer's lack of imagination.) It has been faulted for its optimism-pessimism bias, which lures the researcher to project selectively and build on experiential data overlooking entirely new approaches, and for its tendency to researcher overselling or overuse, demonstrated by assuming the Delphi is methodologically appropriate. Further faults include using too many panelists, and methodological mismatch with the goal of the research. The Delphi has been characterized as an elitist rather than democratic process and faulted because participants structure the questionnaire, and finally, because of the possibility of deception.

Delphic deception is often illustrated by reference to the Greek myth of Ino, wife of King Athamus of Orchomenus. In the story, when the king dispatched a messenger to the Oracle at Delphi, and the King's wife, Ino, bribed the messenger to return with a falsified story, in the second round of consultation at Delphi, the Oracle based her pronouncement on the false version of her first utterances.

Criticism notwithstanding, studies comparing the Delphi's results with other methods (Ulschack 1983) confirm the effectiveness of the method related to generating ideas and use of participants' time (Ludwig 1997). There is renewed interest in the Delphi Method in light of the recent evolution of Computer Mediated Conferencing Systems and the personal computer's ability to immediately scale qualitative responses and calculate and represent quantitative group responses in real time. So used, the Delphi has been put forth as an appropriate platform on which to develop long-term knowledge systems, or "Collaborative Expert Systems" (Turoff and Hiltz, in press) in which statistical scaling techniques convert data from nominal to ordinal or ratio format (making them useful for inferential analysis), and allow dynamic contributions to an evolving, and queriable real-time knowledge system. Several web-based Delphi software programs are being developed and tested (Chen, Chiu and Beiber 1998, Mortensen et al 1997), although there are no reports of implementation or execution.

Turoff and Hiltz (in press) contend that the most important design criteria for the electronic Delphi is participant choice. The effectively designed electronic Delphi would allow participants to choose the sequence in which to examine and contribute to the problem-solving process and to exercise personal judgement about what part of the problem to deal with at any time in the process. Work is only just beginning on the design of the ideal computerized Delphi: one that would allow such sequential, parallel, or asynchronous activity by participants, and there is a need for a model which integrates the individual problem solving process with the group process (Turoff and Hiltz, in press). But, when developed, the electronic Delphi holds great promise as an analytical tool for application to existing and evolving collaboratory archives such as those discussed in this study's Phase Two, as well as for expansive research into the collaboratory's information ecology. This study employs a permutation of the evolving electronic Delphi for the purpose of identifying new information and to probe the emerging collaboratory environment for evolving cultural practices, and, while doing so, to identify skills that might be incorporated in collaboratory training and education programs.

 

Collaboratory Pioneers

The careful selection of participants and participant group size is critical to the successful Delphi. Delphi participants must be purposively (rather than randomly) selected (Ludwig 1997) based on individual qualifications and characteristics. While Delphi Studies have been successfully conducted with hundreds of participants, the median group size is 15-20 participants (Ludwig 1997). Debecq, Van de Ven and Gustafson (1975) suggest using the minimally sufficient number of respondents. Successful Delphi Studies have been executed with as few as five (Telecat 1998) or six participants (Strauss and Zeigler 1975). Since the Delphi is characterized as more of a "search for public wisdom than it is a search for individual knowledge or deliberative judgment" (Lang n.d.), the careful selection of individual experts who have knowledge of the subject and a participatory temperament is more important than rigid group size requirements.

Participants solicited for this study's Delphi meet the following criteria:

1. Participant must be associated with a functioning collaboratory that meets the CIRAL criteria for inclusion developed in Phase Two.
2. Participant must have a "big picture" position in the collaboratory (i.e. not be simply an occasional collaboratory participant or involved with a single, isolated aspect or portion of the collaboratory such as software development or infrastructure maintenance).
3. Participant must be a practicing scientist who has actively participated in collaboratory experiments and activities for at least one year (i.e. not just be an administrator or manager, or new to the environment).

These criteria reduce considerably the already small number of available Collaboratory scientists. Extensive and broad involvement in, and affiliation with a collaboratory that meets the CIRAL criteria was deemed important so that responses would not reflect either a singular experience or a predominately theoretical or administrative perspective. Seven candidates were identified during Phase One and Two of this study and were invited via email to participate in the Delphi. The email invitation included a brief description of the dissertation, with focus on the subject and execution of the Phase Three Delphi. Each invitee was asked to complete the online consent form at http://www.intertwining.org/dissertation/consent/p3consen.htm (Appendix D). A positive consent generated an invitation to nominate others who might also be appropriate participants. Six of the seven invitees agreed to participate. None nominated other participants. The participants were given several months to consider the subject of the Delphi before the Delphi was actually executed, and asked not to discuss the study with anyone during the interim.

All of the participants are involved with the functioning collaboratories explored in Phase Two of this study. All are practicing scientists who hold top-level positions in their respective collaboratories. All have national reputations; several have international reputations. Four of the participants are male; two are female. The researcher did not know any of the participants before the study was executed. Several participants contacted the researcher, as invited, by return email or telephone to ask questions or discuss the technicalities of the Delphi. There was no discussion of the Delphi topic prior to execution of the study, however.

 

Round One

The six participants were sent the Round One questionnaire (Appendix E) as an email message and asked to respond within two weeks. The questionnaire gave a brief description of the Delphi technique along with the short excerpt from the "rules of the road" paragraph from NRC (1993, 3) cited earlier in this chapter. Participants were asked to respond to two questions: "What are the 'rules of the road' for the Collaboratory?" and "What skills do you value in prospective participants?" "Rules of the road" was not defined so that researcher bias could be avoided and because it was expected that the emergent definition would be an important finding of the study. Participants were not identified to each other.

Four of the six participants responded to the Round One questionnaire. In many Delphi Studies the unresponsive participants are sent gentle reminders and urged to complete and return the questionnaire. However, keeping with Turoff and Hiltz's (in press) criteria of protecting and preserving choice in level and sequence of participation in the Delphi process, this study did not follow that practice. It was assumed that unresponsive participants would contribute in later rounds and as they deemed appropriate. One of the participants expressed reluctance to respond to the Round One questionnaire because the questions were sociological rather than physical, and were outside the scope and discipline of expertise, but that person responded briefly nevertheless, expressing the expectation to participate more extensively as the study progressed.

Round One responses ranged from brief sentence fragments to fully developed paragraphs. Each participant's response was read and digested as a body with attention to clues about the respondent's interpretation of "rules of the road." The four responses were distinctly different. Two respondents took a philosophical route, one exploring and interpreting the need for "rules of the road" and the NRC's alleged difference between small and large collaboratories. The other focused on the underlying issue of trust-building in relation to size of collaboratory. Two participants took a more practical approach to the questions, identifying "dos and don'ts" and other functional, behavioral, and cultural matters, one taking a very instrumental approach, the other taking a more sociological approach.

After the general evaluation of individual approaches to the questions, the individual responses to the "rules" and the "skills" questions were chunked into thirty-two freestanding thoughts. Each individual thought was transcribed onto a 5x7" card. The cards were shuffled to achieve random order and transcribed to an interactive web page (the text of which is included as Appendix F). For the "rules" section, respondents were asked to respond to each thought by completing a five point Likert Scale representing their level of agreement with each thought. Possible responses were agree, somewhat agree, no comment, somewhat disagree, disagree. A comment box was provided after each thought to allow respondents to expand, explain, or develop any of the thoughts. An additional comments box invited participants to add new items to the Round One responses and also allowed participants who did not respond to Round One to add afterthoughts which could be incorporated into subsequent rounds.

Eight of the thoughts in the "skills" section was assigned a one-to-ten ranking scale (one being an unnecessary skill and ten being a mandatory skill). The last eight thoughts on the Round Two instrument were the quantitative and qualitative findings of Phase One of this study. Participants were asked to agree, disagree, make no comment, or indicate that the thought did not apply. A total of forty-one "thoughts" were submitted to the group as Round Two.

 

Round Two

The Delphi participants were emailed the URL, or online address, of the Round Two instrument and asked to respond within one week. The url is http://www.intertwining.org/dissertation/round2.htm. The email message also included the name, collaboratory affiliation, and email address of all the participants, along with a reminder not to communicate with each other about the study. Responses to the web page form were programmed to feed a concatenated hypertext file and a concatenated, comma delimited text file, and also generated an email message to the researcher with comma-delimited responses enclosed. The qualitative options on the instrument were automatically converted to quantitative data for analysis purposes. Responses were fed into a Microsoft Excel97® spreadsheet designed to allow side-by-side viewing of the Round One thoughts and the Round Two responses and comments of all the participants, as well as the statistical interpretation of the responses.

One of the participants who did not respond to the Round One questionnaire replied to the Round Two email message with concern about the number and inclusiveness of the participant pool, and their representation of the group "Collaboratory Pioneers." This respondent offered to "round up" more participants for the study. The researcher replied to the message, encouraging nominations of qualified participants and providing details about participant criteria. The participant did not nominate additional participants, however, nor respond to the Round Two instrument.

The Round Two email message sent to the other participant who did not respond to the Round One instrument generated a system email message indicating that the person's disk quota was exceeded and that the message could not be immediately delivered, but indicated that attempts to deliver would continue for five days. The system did not generate a final undeliverable message, but the participant did not respond to the Round Two instrument. The four participants who responded in Round One responded to Round Two. None of the respondents added additional comments or items to the Round One responses.

Round Two responses are insufficient in number for the Delphi's traditional interquartile range analysis in which respondents whose answers fall outside the center-most 50% are asked to justify or explain the deviant answer or response. Alternatively, a means-based analysis was executed in which the average answer for each question was calculated and qualitative cross-thought analysis for themes was undertaken.

Seven of Round Two's forty one thoughts generated unanimous agreement; two of the unanimous thoughts were from the "rules of the road" section of the study, three were from the "skills" section, and two were from the section testing the eight qualitative and quantitative findings from Phases Two of this study. Only one thought (from the testing section) generated unanimous disagreement among participants. One of the testing thoughts generated a different response from each of the four participants. Detailed analysis of each section of the Round Two responses follows.

"Rules of the Road" -- Discussion

All but one of the twenty "thoughts" about the "rules of the road" for the collaboratory were agreed to or somewhat agreed to by a majority of the respondents. While several of the thoughts generated individual disagreement, only one thought generated majority disagreement. The five point scale for possible responses was agree, somewhat agree, no comment, somewhat disagree, disagree. Respondents provided comments on seventeen of the twenty thoughts, for a total of thirty-two comments.

The two thoughts about "rules of the road" that generated unanimous agreement (with respondent comments) are:

1. Be direct. If you have an idea, complaint, or any comment, say it. If you need something, you must ask. Don't expect anyone to read your mind.

This is important -- the lack of body language, feedback, etc. makes it very hard to pick up subtle expressions of frustrations, etc.

2. You must get involved and get someone in the collaboratory interested in working with you on a problem.

Collaboratories should not be approached as a technology push -- deployment should be fueled by the needs of the scientists.

If you don't, it will slow down the pace of the research project!

One can only spend a limited amount of time playing with collaboration toys. Eventually you have to work on a real problem.

The only "rules of the road" thought that received majority disagreement concerned publications production. The original thought was:

When researchers visit a collaboratory facility it is probably to complete an experiment resulting in publications: if a researcher pops into a virtual room/session and discusses ideas with colleagues, there may be no direct publishable artifacts.

Three respondents somewhat disagreed with this thought, and two offered

comments:

Publications do not always come out of each experiment. My experience is that a group of experiments conducted over several sessions will usually get a publication.

Maybe true but this is "apples and oranges." While on a site visit, a researcher will have hallway conversations that do not result in a publication.

Although there was no strong overall disagreement with any of the remaining statements, communication issues related to a balance between "hallway conversations" (unscheduled, spontaneous meetings) and formal communications, and planned collaboratory meetings, and between experiment flexibility and rigid experiment planning. Both issues have an undercurrent about trust, and generated the strongest dialog among the remaining "thoughts." While planned, regular collaboratory sessions are considered important for stimulating frequent communication, they are seen as the equivalent of coffee breaks, and do not substitute for planned, scheduled project meetings:

'down the hall" (type meetings) imply spontaneity. Frequent communication is very important and scheduled meetings help do that.

It depends on the nature of the collaboratory. You cannot always structure the collaboration. It needs to adapt to the types of work. Some require regular participation, others don't.

One frustration I foresee with planned, regular sessions--as with in-person meetings--(is that) participants may not show up, whether due to conflicts or declining interest. To me, it seems the frustration associated with this would be greater for remote colleagues than for colleagues physically down the hall.

That's what coffee break style meetings, or environmental (always on) video are for.

Two of the participants somewhat agree that establishing at the onset who will do which part of the experiment and follow up analysis is important. The other two somewhat disagree, one commenting that such planning is no different than on a normal (non-collaboratory) project, and the other warning that maintaining flexibility and willingness to change is necessary since some experiments cannot be planned accurately.

It was generally agreed, and to one participant very important, that collaborators commit to making frequent deposits of data, notes, etc. to a shared electronic notebook or database so all collaborators can stay up to date and so the progress of the research can be as efficient as stopping by a colleague's office down the hall to take a look at data. Another participant commented that this takes work. The respondent who somewhat disagreed with this thought explained that while it is helpful, collaboratory technology should be able to automate these processes:

instruments sending data directly to notebooks or things like the Crosspad - you write on paper and an electronic copy is made that can automatically go to an electronic notebook.

It was agreed that electronic interaction shifts work between collaborators: that when a researcher physically goes to a facility to do an experiment they are available to help with instrument maintenance and configuration, to get supplies from the storeroom, etc., while the remote researcher is not. One respondent commented:

This is critical. I have heard many people express a lack of interest in collaboratories due to issues like this.

 

The differing communication needs as related to the trust-building process within large and small collaboratories were generally recognized. One respondent disagreed that size is an important consideration, however:

I don't see why (establishing trust in) large scale (collaboratories) should take any longer. If everyone is communicating electronically everyone is still on the same scale - i.e. everyone is "equal" electronically, there may be more people/groups to interact with, but I believe the trust level would develop at the same rate as a small collaboratory.

Another did not see size as a trust-building issue unique to collaboratories:

Isn't this just as for local small and large groups? My statement was that the unique aspect of collaboratories versus local interactions was not the size of the group, but the informal, cross-organizational aspect.

It was generally agreed that trust takes time to build and the time constant is much longer when contact is less frequent due to time and/or distance; however, this was seen as "very personality dependent -- 'trust at first sight' does exist in the world!"

The participants agreed that the balance of trade in informal interaction may favor one person/culture/organization over another with one participant pointing out that this is always true.

All participants but one (who disagreed but had no comment) agreed with the statement

Our inability to measure value of informal interactions is one reason we organize--we get common culture, i.e. people learn to provide similar amounts of informal help to each other; all the benefits of these interactions accrue to the organization; both these lessen the need to measure them.

It was generally agreed that many if not most research projects can be enhanced by using a collaboratory, but that collaborators must embrace that collaboratory work will take extra time to get up to speed and is subject to glitches in technology and the Internet, or collaboration will be slowed down such that it will not compete as an effective alternative to traditional methods.

Collaboratories can provide real benefits now. However, a user who thinks they will solve all their current collaboration problems without encountering some new (smaller) troubles is due for a reality check. Collaboratory developers/promoters need to make sure they don't oversell current capabilities or we'll have some backlash.

It was generally agreed that "rules of the road" are an attempt to find a balance between differing cultures, but such rules are still ad hoc:

Often, Collaboratories require that you formalize processes (perhaps by writing the procedure into collaboration software) that are currently informal or that differ between organizations.

All but one respondent (who disagreed), agreed with the statement

Because collaboratories are still new enough to be subsidized (we fund development of tools and creation of virtual facilities, accept papers on these topics, etc.) buys time to get the rules right. As collaboratories become standard practice, the subsidies decrease, and the need to equalize the benefits will increase.

 

"Rules of the Road" -- Analysis

Delphi participants represent all sizes and types of Collaboratory identified by early EMSL psychosocial studies as discussed in Phase Two of this study. Size of the collaboratory, and level of involvement in collaboratory activity might explain the subtle but remarkable differences in preference for balance between formal and informal communication, and planned and fluid experiment modes, as the NRC suspected.

Tuck and Earle (1996) explore the communication modes of different group sizes in the corporate environment, relying on classical cultural anthropology studies to conclude that group size is always a determining factor in group communication structures (Johnson and Earle 1987, Earle 1987, Schmookler 1984, Johnson, 1982). While each individual's role and behavior within the group is determined by inherited nature, upbringing, and training, two communication modes: the egalitarian and the hierarchical, and four universal subgroup structures: the working group, the camp, the tribe, and the state, naturally exist whenever people live or work together in groups.

The two communication modes and their size-based substructures are:

 

Egalitarian, arising from the belief in equality among all, includes

• The working group, which is a temporary association of two to six people, each with useful skills, who come together for a specific purpose, work democratically, organize quickly and informally, and disband when the problem is solved. Leadership of working groups arises from within the group by consensus. Members function at the innate behavior level, meaning that no new or dramatically different communication skills or rules are necessary; and

• The camp, which is made up of six to thirty people who have little specialization, and whose job is to make decisions. Although the camp itself may deny having a leader, the camp is lead by a facilitator who is respected for knowledge judgments and skills in organizing opinion. The facilitator leads rather than commands the camp by focusing the decision-making process. The camp does not solve problems, merely makes decisions about how and what problems will be solved (once a decision is made about a problem, the camp delegates the work and smaller working groups, which solve the problem.) When the decision making process of the camp is disrupted, camps often split, forming new camps. Individuals are frequently members of both a working group and a camp, as is the case with most small businesses or companies.

 

Hierarchical, a system of ranks, one above another, including

• The tribe, which is made up of fifty to one hundred people and usually encompasses several camp-size groups. The tribe has a clearly defined leader, many strata, and clear lines of authority. It has well defined rules and patterns of behaviors, and is lead by a chief, who makes decisions for and gives orders and edicts to the tribe. Tribes solve the problem of instability in large organizations, but at the cost of personal and group autonomy. The purpose of the camp is bureaucratic: to facilitate relations with other camps (including conducting commerce and war), for conducting religious observations, and to allow occupational specialization.
• The state, which is made up of one hundred or more people, and includes multiple tribes, camps, and working groups. Like the tribe, the purpose of the state is bureaucratic. A camp-like council (which, in the corporate structure, is the Board of Directors) leads the state.

The purpose of Tuck and Earle's research is to determine why Chief Executive Officers, or CEOs, fail. They find that CEOs frequently fail when a communication structure appropriate for one scale is used for groups of other sizes, as when, after promotion from the ranks, a CEO continues to use egalitarian communication modes when hierarchical modes are appropriate. They also find that that CEOs fail when they do not shift gracefully between modes, as when they move from communicating with their employees ( a hierarchical tribe or state) to communicating with their board of directors (an egalitarian working group).

The universal differences Tuck and Earle find in communication modes attributed to group size might also be reflected in the culture of the Collaboratory, and could explain the slight but remarkable difference in preference for either informal, unstructured communication and flexible experiment planning within a larger, structured environment, or more formal communication modes and well-planned experiments, and within them, informal and adaptive behaviors.

Conceivably, collaboratory participants might function in all four group sizes simultaneously: as a member of a working group (as on a specific experiment); as a member of a camp (in their role as a researcher associated with a specific collaboratory); as a member of a tribe (as in collaboratory management or administration within a larger laboratory environment); and as a member of a state (as in their place with larger U.S. National Laboratories or with government funding agencies.) Conceivably, each collaboratory implementation might need to support all four group sizes. Clearly, a keener understanding and mapping of the different communication modes and group sizes as they relate to the various collaboratory configurations, and on size-crossing participant roles, will shed light on the preference differences uncovered in the "rules of the road" portion of this Delphi. Such studies would also be useful for configuring automated collaboratory expert systems where the answer to a problem might be significantly different depending on which group size originates the problem, and from which size group a solution or opinion is offered. It would also be interesting to discover if having virtual presence as an individual functioning physically in one group size and virtually in another has impact on the way science is conducted, and the impacts of that simultaneity on the scientist.

"Skills Valued in Prospective Participants" - Discussion

In Round One of this Delphi, participants generated thirteen thoughts about skills they value in prospective collaboratory participants. Five of the thirteen thoughts were presented to the group using the same five-point Likert scale used in the "rules of the road" section. Participants were asked to indicate their level of agreement with each thought and were given the opportunity to make further comments and additions. Eight specific skills were also presented so participants could indicate on a scale of one to ten the level of value they had for each skill, with one being an unnecessary skill and ten being a mandatory skill.

Participants agreed unanimously and without further comment with two of the first eight thoughts generated in Round One. Those thoughts are:

1. Anyone who has a real project in mind (something they want to get done that is cumbersome using travel, email, fax) probably has the right mindset to go forward (trading difficulties of real-world interactions for the reduced difficulties of working via collaboratory).
2. Anyone looking for the perfect solution will probably be disappointed.

Three of the thoughts received substantial, although not unanimous agreement (although none of the statements was disagreed with). Those statements, with participant comments, are:

3. Know why the problem is important to study…enough so to get people interested in helping as well as justifying the time spent on the study.
4. Have some basic knowledge of the science. You don't have to be an expert, but you must be able to discuss it and provide appropriate support at your end to do what is necessary on your part.

The fact that collaboratories can allow anyone, e.g. grade school students to access the best instruments in the country should not lead to the expectation that they will start winning most of the peer-reviewed time allotments.

5. Be willing to participate/help with other problems of appropriate nature. Don't expect to be helped without returning the favor at some time in the future, for some arbitrary participant.

Eight specific skills identified by participants in Round One were presented for ranking in value on a one-to-ten scale (one being an unnecessary skill, ten being a mandatory skill). One of the eight received a unanimously low ranking. The remaining seven skills, on average, were ranked 5.5 and above. None of the skills were ranked mandatory, and none were ranked unnecessary. Following is the ranking of those skills, in descending order of value, with mean rank on the one-to-ten scale in parentheses:

6. Tolerance for evolving technology and practices (7.5)
7. Good communication skills (7.0)
8. Good to expert scientific knowledge (6.5)
9. Experience in the (scientific) techniques used (6.25)
10. General team skills (6.25)
11. Familiarity with Internet technology and software (not at a programmer level, but someone who uses a desktop PC on a daily basis and is familiar with spreadsheets, data processing software, etc.) (5.75)
12. Good computer skills/computer literacy (5.5)
13. No one is an expert at everything but everyone has some expertise in something. We expect you to offer to share it when the right time comes (1.25).

"Skills Valued in Prospective Participants" - Analysis

Collaboratory Pioneers value a balance of social, technological, and scientific skills in prospective participants over a superior expertise in any one of them. Social skills identified include tolerance for evolving technologies and practices, good communication skills, team skills, adaptability, and willingness to share and contribute equally. Scientific and technological skills valued include good to expert scientific knowledge, familiarity with scientific techniques, and personal computer applications and Internet fluency. In addition, collaboratory pioneers value in prospective participants the ability to identify and articulate the importance of a particular scientific problem, to engender interest and participation in that problem, and to recognize and be able to articulate the value of collaboratory work in excess of the impediments it might present without creating an inflated perception of collaboratory capabilities.

"Testing Perceptions" - Discussion

The final eight thoughts presented in the skills section were statements derived during Phase One of this research, and were not generated by the Delphi participants. These statements were included to test participant perceptions of these findings. Participants were asked to indicate whether they agree with, disagree with, have no comment on, or to indicate that the statement did not apply to the collaboratory.

Eight findings were generated during Phase One of this study. In Phase One, quantitative, taxono-bibliometric analysis of the library literature reveals a relatively equal contribution to collaboratory publications from the hard and soft sciences, and that the collaboratory is, by example, motivation, principles of interaction, and terminological hierarchy (Klein 1990, 56) an inherently interdisciplinary environment. Synoptic analysis of twenty-two Theory-Type Research publications produced four additional collaboratory principles:

• integration and adaptability are necessary and good
• change, choice, and personal power are requisite
• consensus, sharing and exchange are positive and practiced
• individuality and collectively are distinctly and respectfully maintained.

The absence of traditional male social and scientific behaviors, coupled with the traditionally female characteristics of these findings led to the emergent theory that the collaboratory is an ungendered environment, a theory that confirms, in part, the basic assumption that the collaboratory will fundamentally change the way science is done.

Two of these statements received unanimous agreement from Collaboratory Pioneers. Along with the comments, those statements are:

1. The collaboratory fundamentally changes the way science is done.

Yes and no--science is still science, but I think Collaboratories do represent a revolution in flexibility and the ability to solve complex problems rapidly.

2. Integration and adaptability are necessary and good.

Scientific experiments are by definition unique--collaborative problem solving software must be able to adapt to constantly changing procedures.

All but one participant (who had no response) disagreed with the statement "The collaboratory has been built from a relatively equal contribution from the hard and soft sciences." The statement generated only one comment:

I think the deployment of Collaboratories is being driven by technologists and physical/biological scientists with collaboration needs. The soft sciences need to contribute more -- real world insights and advice, predictive models that help abstract the results of psychology, cognitive, group dynamics and other studies in controlled setting to the real world.

All but one participant (who disagreed) agreed with the statement "Change, choice, and personal power are requisite":

Not requisite, but I think Collaboratories allow more change, choice, and personal power.

Two participants agreed with and two said the statement "consensus, sharing, and exchange are positive and practiced" did not apply.

I agree, but Collaboratories are not communes--people share just as they did without a Collaboratory, except that Collaboratories make sharing easier, so (it) more naturally occurs, as appropriate.

Two participants agreed with, one had no response to, and one responded that the statement "Individuality and collectivity are distinctly and respectfully maintained," did not apply:

Collaboratories do not change people's ideals about sharing, respect for individuality, etc., but by lowering barriers, they may bring practice closer to ideals.

All but one participant (who responded "does not apply') agree with the statement "The collaboratory is an interdisciplinary environment."

Agree, although I think it used to be mainly one discipline and is perhaps still somewhat skewed that way. It is easier to get going in it if one is collaborating with remote colleagues in the same discipline.

Collaboratories lower the barrier to interdisciplinary work, that's all. The drivers for interdisciplinary work come from outside (brought by researchers, sponsors, the public, etc.)

The statement, "The collaboratory is an ungendered environment" received the greatest variety of responses, a different response from each participant, including an agree, disagree, no response, and does not apply, but only one participant added a comment, "Agree, probably more so than a local environment."

"Testing Perceptions" - Analysis

The statements generated from Phase One of this study, and the reaction to them from collaboratory pioneers do not bear the weight of more than superficial analysis. Nevertheless, that collaboratory pioneers are unanimously unaware of the level and extent of social sciences' published contribution to the theoretical and practical research leading to collaboratory implementation warrants some comment and inference. First, the use of the word "built" in the question may have been misleading. Participants may have thought "built" meant actual, physical construction of the environment, and not theoretical or conceptual contributions to its development. Part of the knowledge gap these findings reveal is undoubtedly due in part to the library's failure to make collaboratory-related literature easily accessible (as discussed in Phase One) as well as to scientists' habit of reading and publishing in publications that are predominately within their own disciplines and specialties.

While it is not appropriate to project to the entire Collaboratory population from this thin study of collaboratory pioneers, it is appropriate to infer to the environment that an information opportunity exists within the Collaboratory for librarians and information specialists and researchers. Including librarians in collaboratory environments would benefit collaboratory participants and enhance online collaboratory resources; and a general understanding of collaboratory practices and culture will position librarians, as educators, to prepare prospective participants, and prepare for prospective participants' information needs.

That the "ungendered" statement received a different response from each Delphi participant is significant. The range of responses could be because the concept of "ungendered" is difficult, highly subjective, and perhaps cannot be disassociated from the multifaceted definitions and perceptions of feminism as discussed in Phase One. The range of responses may also be based on individual personal experience, situational perceptions, or lack of interest in the issues it raises. Certainly, the theoretical ungenderedness of the Collaboratory developed in Phase One warrants testing in the functioning work environment of the collaboratory and should be explored along with group size communication modes and experiment planning preferences.

 

Conclusion of Phase Three

Phase Three of this study constructs an intersubjective reality of the collaboratory by engaging Collaboratory Pioneers in a Delphi Study to determine the "rules of the road" for the collaboratory and identify skills they value in prospective participants. Six Collaboratory Pioneers associated with collaboratories meeting the CIRAL criteria for inclusion developed in Chapter Five, who have "big picture" positions within those collaboratories, and who are also practicing collaboratory scientists, where solicited for this Delphi and agreed to participate, although only four actually responded to the two rounds.

The first round of the Delphi asked two questions: "What are the 'rules of the road' for the collaboratory, a question raised in NRC's (1993) National Collaboratories: Applying Information Technology For Scientific Research, and "What skills do you value in prospective collaboratory participants. The Round One questionnaire generated twenty thoughts about "rules of the road" and twelve thoughts about "skills valued in prospective participants." Eight statements developed during the qualitative and quantitative analysis in Phase One were added, and a forty item web-based Round Two instrument was developed and resubmitted to participants. The instrument asked participants to indicate their level of agreement with thirty-two of the items, and to rank the value of eight items on a one-to-ten scale. A comment box allowed participants to expand, explain, or add new items to the study. Four participants responded to the Round Two instrument. No further reduction to consensus or stasis was necessary to preliminarily identify the "rules of the road" for the collaboratory and the skills Collaboratory Pioneers value in prospective participants, which are enumerated below.

Based on the four responses, following are the "rules of the road" for the Collaboratory:

1. Be direct in your communication. If you have an idea, complaint, or any comment, say it. If you need something you must ask. Don't expect anyone to read your mind.
2. Get involved and get someone in the collaboratory interested in working with you on a problem.
3. Have a real problem that the collaboratory can help solve. Be able to articulate the problem and accurately express to others how the collaboratory can be used to resolve it.
4. Understand the opportunities and limitations of collaboratory work.
5. Stay flexible within a formal framework of meetings and experiments. Collaboratories must include a balance of planned, regular collaboratory meetings, experiment sessions, and formal project meetings, and unscheduled, informal meetings and spontaneous experimentation.
6. Make frequent contributions to collaboratory data repositories.
7. Working in a collaboratory is not the same as being physically present in a laboratory. Remote collaborators need to find ways to contribute to and share the "chores" of collaboratory work to compensate for their physical absence.

Following are the skills Collaboratory Pioneers value in prospective participants, in descending order of agreement:

1. Tolerance for evolving technology and practices
2. Good communication skills
3. Experience in the scientific techniques used
4. Good to expert scientific knowledge
5. General team skills
6. Computer application and Internet competence.

It was tempting to continue the Delphi beyond two rounds just to probe more deeply into the individual perceptions and opinions revealed in the comments, but given the number of participants, such a probe would reveal more about the individual participants than it would about the cultural environment of the collaboratory generally.

Four themes that warrant further research surfaced during Round Two. The first, a statement derived from Phase One's findings that the collaboratory has been built from a relatively equal contribution from the hard and soft sciences, generated general disagreement from Delphi Pioneers. As discussed, this disagreement may stem from differing interpretations of the word "built." The respondents may consider "built" to mean physical construction and technologically implemented, rather than conceptual design or theoretic exploration, as Phase One's interpretation of the literature implies. It may also reflect the exact opposite attitude Wulf warned about in his White Paper : instead of theoretical researchers having problems accepting the value of practical, technological research, perhaps practical, technological researchers do not recognize the value of theoretical, conceptual research work.

The second and third themes that warrant additional research are closely related, if not intertwined, and concern the balance between informal communications and flexible experiment planning, and formal communication and rigid experiment planning. Group size is suggested as a possible reason for the differences of opinion and preferences. The fourth theme concerns finding a balance of contribution to or a new sharing ethic between on-site and remotely located collaborators.

That collaboratory pioneers underestimate the contribution that social sciences have made to collaboratory research is a knowledge gap that may also be attributed to shortcomings in the library's indexing and classification practices, and to scientist's tendency to maintain disciplinary focus in their reading. The issues of preferences in level and mix of communication modes, experiment flexibility, and the collaboratory sharing ethic may be attributed to group size. Expansive research in each of these four areas will be useful during continued design of collaboratory interfaces and will shed light on the information needs of collaboratory scientists.

The emergent population of scientists who practice in the collaboratory may also be fertile ground for future investigations. Since each of the pioneers is first a disciplinary scientist, and only secondarily an implementing technologist, this population may hold clues about the changing nature of the role of scientist, and the impact of technology not only on the way science is done, but the division of their labor between science and technology. It may also herald the emergence of a new discipline (Kuhn 1970), the collaboratory scientist.

 

Conclusion of the Study ->

Placed January 1999
Contact reseacher: twining@intertwining.org
Dissertation web: http://www.intertwining.org/dissertation

-30-