Miles Bryant
Friday, January 13, 2012
January Observation
In reviewing critical incident studies done in the past few years, I can observe that many do not follow the design structures that Flanagan outlined. For example, many gather what are described as perceptions or beliefs. Flanagan said the CIT wanted to record actual behaviors or remembered behaviors. I think this is a critical but common flaw in the way many use the CIT. A second observation is that many do studies setting out in advance to specify a number of research subjects they will interview. This again violates Flanagan's idea that one should begin gathering critical incidents in an iterative fashion until no new specific behaviors can be identified--thus achieving saturation. Many CIT researchers seem to have missed this important element.
Friday, November 4, 2011
Here is another proposal that gives a summary of the CIT
The Critical Incident Technique for the Social Science Researcher: Getting Skunked or Netting the Behemoth
Abstract
This paper outlines a research approach that has provided many social science researchers with valuable insights into human behaviors that contribute to improved practice. While the Critical Incident Technique (CIT) has been known to scholars since its development by John Flanagan in 1954, inclusion of the CIT in research methods texts is rare. The purpose of this paper is to describe the CIT, portray the technique as a means of examining human behavior in a physical context, provide some examples of its use, and offer some claims about its practical value to researchers. The paper does this by developing an extended metaphor or conceit using the unusual example of fly-fishing for trout.
Just as the fly-fisherperson wants know the preferred behaviors that lead to catching trout, the CIT researcher wants to identify behaviors that can be observed or recounted in a specific situation that clearly lead to success or failure. The CIT is a research design firmly motivated by a goal to improve practice. The CIT is, then, located at the nexus of research and practice. As such, it is a tool ideally suited to help social scientists “make things better”. As such, the proposed paper fits within the goal of the conference to explore various approaches to evaluation practices.
The CIT requires that we be curious about a common work experience or setting and that we seek to understand better the critical behaviors enacted in that setting. By critical, we need to know which behaviors clearly lead to success or failure. There are thousands of examples of workplace settings in which CIT researchers gathered data about critical behaviors. In this sense, the CIT can be a valuable methodological addition to the evaluator’s tool kit.
Miles Bryant
Mbryant1@unl.edu
Marilyn Grady
Mgrady1@unl.edu
http://www.ucpori.fi/eval2012cfp/
Abstract
This paper outlines a research approach that has provided many social science researchers with valuable insights into human behaviors that contribute to improved practice. While the Critical Incident Technique (CIT) has been known to scholars since its development by John Flanagan in 1954, inclusion of the CIT in research methods texts is rare. The purpose of this paper is to describe the CIT, portray the technique as a means of examining human behavior in a physical context, provide some examples of its use, and offer some claims about its practical value to researchers. The paper does this by developing an extended metaphor or conceit using the unusual example of fly-fishing for trout.
Just as the fly-fisherperson wants know the preferred behaviors that lead to catching trout, the CIT researcher wants to identify behaviors that can be observed or recounted in a specific situation that clearly lead to success or failure. The CIT is a research design firmly motivated by a goal to improve practice. The CIT is, then, located at the nexus of research and practice. As such, it is a tool ideally suited to help social scientists “make things better”. As such, the proposed paper fits within the goal of the conference to explore various approaches to evaluation practices.
The CIT requires that we be curious about a common work experience or setting and that we seek to understand better the critical behaviors enacted in that setting. By critical, we need to know which behaviors clearly lead to success or failure. There are thousands of examples of workplace settings in which CIT researchers gathered data about critical behaviors. In this sense, the CIT can be a valuable methodological addition to the evaluator’s tool kit.
Miles Bryant
Mbryant1@unl.edu
Marilyn Grady
Mgrady1@unl.edu
http://www.ucpori.fi/eval2012cfp/
Wednesday, November 2, 2011
Fly Fishing and the CIT
Critical Behaviors and Where They Take Us: A Research Design for the Doctoral Student and the Practicing Professor
OR
Fly Fishing: Getting Skunked or Netting the Behemoth
Miles Bryant
University of Nebraska-Lincoln
Lincoln, NE 68588-0360
Critical Behaviors and Where They Take Us: A Research Design for the Doctoral Student and the Student of Practice
OR
Fly Fishing: Getting Skunked or Netting the Behemoth
Introduction
Life is full of situations where specific behaviors can lead to successful or unsuccessful outcomes. Athletic teams win or lose based on behaviors; critical care nursing practices may yield a recovered patient or a deceased patient; a classroom teacher’s questioning techniques may result in learning success or learning failure; the physical and observable steps made by an artisan bread baker may produce a delicious loaf of bread or a lump of inedible dough. And the behavior of the fly fisherperson on a stream, river or lake can lead to a fish or not. It is this last example of the behaviors of the fly-fisherperson that I will use to outline a very practical research design: the critical incident technique. And I have a very particular kind of fish in mind—trout—for most often the trout is the object of the fly-fisherperson’s actions.
The behaviors associated with fly-fishing for trout are complex and allow for many comparisons to the behaviors of the Critical Incident Technique (CIT) researcher. To illustrate, here are some examples of fly-fishing behaviors. One has to be strategic in determining where to fish for trout. One has to make many practical decisions about the equipment that one uses to catch a trout. One has to be receptive to all the evidence that may reveal where a trout lies in wait. One has to employ skill in casting for a trout. One has to behave in certain ways once one has hooked a trout. And within each of these general categories of behavior are multiple smaller examples of actions by the fly-fisherperson that can lead to success or failure. In the matter of selecting equipment, one can chose a fly that attracts a fish, a fly attached to the appropriate type of fly-line, leader, and tippet so that the artificial fly looks to the trout like a real fly looks.
Just as the fly-fisherperson wants to catch trout, the CIT researcher wants to identify behaviors that can be observed or recounted in a specific situation that clearly lead to success or failure. Think about this. In these areas we are often urged to locate our scholarship at the nexus of research and practice. Here is a research design firmly motivated by a goal to improve practice. The CIT is, then, located at the nexus of research and practice. This is a perfect place for a student of educational leadership.
The Elements of the CIT
The rest of this paper reviews the basic design elements of the CIT, using metaphors from the activity of fly-fishing to help illustrate the research design. The paper ends with comments about the utility of the CIT for the doctoral student specifically and the researcher in general.
1.Catching Trout: Determining the Aim
Trout fishing takes place somewhere where trout live. This is the setting. The CIT researcher is first of all interested in the behaviors that occur in a setting, seeking to know what results from very specific behaviors that are realistically associated with positive or negative outcomes. To repeat, it is the behaviors in that setting that the CIT researcher seeks to examine. One could, for example, simply enjoy the river and never lift a hand to the butt of a fly rod.
One might be more attentive to the contents of the coolers than on casting for a fish. Were one to do so, that behavior would obviate the selection of this particular incident as informative. But, it might also signal a behavior that clearly leads to failure—if catching a trout is the goal.
The CIT requires that we be curious about a common work experience or setting and that we seek to understand better the critical behaviors enacted in that setting. By critical, we need to know which behaviors clearly lead to success or failure. There are thousands of examples of workplace settings where CIT researchers gathered data about critical behaviors. In each study, the first step was to clearly state the aim or goal of the activity in question. If this reminds you of the “time and motion studies” of Taylorism or scientific management, it should. In many ways, the CIT is a child of that early management theory. Just as did the proponents of scientific management, the CIT researcher wants to assess a human performance aimed at an objective or goal.
When it comes to determining the aim, nothing expresses this core element quite as clearly as the goal of the trout fly-fisherperson. Catching fish is what it is all about.
2. Behaviors
It may seem obvious. To carry out a Cit study One has to be curious about the outcomes of specific human behaviors in a setting. That is, one has to want to know more about the factors that lead to goals or aims of an activity be it fly fishing or be it the behaviors that lead to superior customer satisfaction interactions in a grocery store (Bengt and Nilsson, 1964). It was this key element of observed behaviors that Flanagan (1954) the founder of the CIT advocated as the central strength of the research design. When Lewis, Yarker, et al., 2010) set out to identify the management behaviors that helped workplace stress in nursing, their goal was to find practical ways to improve the working conditions for nurses. To do so they wanted to identify critically important behaviors. When Grady and Bryant (1991) set out to identify the factors behind serious conflicts between superintendents and school boards, they wanted to know what specific behaviors were believed to have led to conflict.
In fly-fishing for trout, a skilled fisherperson will not just cast her fly anywhere on the water. Rather, she will take time to read the river and use her knowledge to be selective in where she casts. And she will know that if she is nymph fishing or casting dry flies, it will be important to keep her fly line from creating an unnatural drag on her fly. These are specific behaviors associated with critical outcomes.
In the mind of the CIT researcher, it is human behavior that leads to success or failure. The research design is oriented toward the identification of those behaviors.
3. The Setting, Incident and the Respondents
Just as fly fishing for trout takes place in specific types of places at specific times, so too does the critical incident. The CIT researcher wants not just general human interactions but human interactions that take place in a setting and situation that she or he has pre-defined. Thus, care must be taken to be very deliberate in delineating the stage (or stream) on which one wishes to gather data.
With the setting well defined, the CIT researcher can then proceed to study multiple incidents, recording the behaviors from each that were associated with critical outcomes. Not every setting will yield critical incidents. But many will. And it is those settings where clearly good or clearly bad results occurred that will drive the study. This accumulation of critical incidents through successive rounds of interviewing and analysis eventually leads to what Flanagan (1954) called saturation. A great range of critical behaviors has been identified and new iterations of incidents and interview yield no new data. At this point the CIT researcher is in a position to start to name a set of behaviors that are critical to success or failure. This, of course, is where the CIT becomes a great benefit to practice.
It should also be obvious that in order to gather data about critical behaviors, one’s respondents or interviewees must be knowledgeable. Best are those who observed behaviors and who can recall those observations in detail. Hearsay from third party participants in the incident can sometimes be valuable but the most descriptive information about behaviors will come from those who were in a position to observe or experience the behaviors.
4. Recorded Experience through Interviews
One of the key elements of the CIT is problematic. Most of the data that the CIT researcher will gather comes from interviews. And these interviews will be conducted by asking the respondent to recall a specific incident and the behaviors that she or he observed at the time.
I could approach a fly-fisherperson on the North Platte River and watch them fish. In so doing I might well be able to identify success or failure (fish or not); I might well be able to identify productive fishing behaviors; I might be able to make judgments about the fisherperson’s skills. I could also do this by asking the fisherperson to tell me about the one that got away. What were the behaviors that may have led to that loss. And, I could also ask the person to recall the behaviors that she or he recalled as important when the big trout was caught. Human memory is fallible. The recollected set of behaviors will, when amalgamated over a number of interviews and critical incidents be reliable. For this reason, more interviews about more incidents is best.
The interview is the main method of gathering data in the great majority of CIT studies. But, the interview differs from the typical qualitative interview that is viewed by many researchers as epistemologically subjective. The data gathered in a CIT study is certainly normally filtered through the subjective lens of the person recalling the incident. But, if the questioning techniques are skillful, the data will approximate what really happened. The CIT researcher is after the reality of the incident with its accompanying behaviors, not the subjective reality within the subjects mind. Thus, the CIT interview will feature a persistent effort to ask the respondent to identify behaviors and not present an interpretation of what took place.
The CIT: A behaviorally grounded research design approach that seeks to interview knowledgeable individuals inviting these individuals to recall specific incidents in which critical behaviors led to success or failure.
5. Success/Failure in the Lives of Ordinary Folk
Stephen Gould, the famous scientist and author who worked at Harvard for much of his career once observed that if you want to understand what ordinary folks do, one thoughtful deviant will teach you more than ten thousand solid citizens (Gould, 1985, p. 101). Gould could have written the banner for the CIT researcher. To amplify the importance and magnitude of behavior and its relationship to outcomes, the CIT researcher seeks egregious or exemplary behaviors.
You are telling me that this incident you remember led to a
disaster. Would you tell me what John did that led to such a disasterous outcome? What exactly did he do?
Armed with this question, you can begin to probe for specific behaviors.
You are seeking behaviors at what could be called the tails of a normal distribution.
It is more common for the researcher to seek to gather data from points within two standard deviations from the mean. In contrast the CIT researcher seeks data that can be found at the tails of the normal curve—these are Gould’s deviants and they can yield a great deal of information about what works and doesn’t work for ordinary “folk”.
For the fly fisherperson who gets “skunked” (the term for catching no fish), her behaviors are perhaps egregious, assuming there were any fish where she fished. On the other hand, if she caught a behemoth, we should be able to identify behaviors that led to this success
6. Applications and Conclusion
This CIT provides students and scholars with an accessible research method that allows them to produce applied and practical research findings. The CIT requires the researcher to gather data from research subjects about behaviors in a bounded context or work setting, behaviors that result in critically beneficial or critically ruinous outcomes. Generally these data are gathered through a well-planned interview process that focuses on detailed descriptions of human behavior.
Is the technique timely? Academic Search Premier, a library database, lists many scholarly articles that have been published in a great variety of journals over the last few years. Here we cite but a few (Trepal, et al., 2010; DeBrew, 2010; Regel, 2010; Wong et al., 2009; Heinrichs, et al., 2009; Dharamsi, et al., 2010; Joyce & Rankin, 2010; Sousa & Costa, 2010; Blouin, 2010; Lee & Vennum, 2010; Emusu, et al., 2009; Sverker, A., et al., 2009; Brunton & Jeffrey, 2010; Holtz & Harold, 2008). Over the past two years, doctoral students at such universities as Virginia Commonwealth University, Stanford, University of Iowa, Michigan State University, University of North Carolina-Chapel Hill, Rutgers University, and Columbia University have listed the CIT as a methodological tool used in completing their dissertations (Meritt, 2011; Akers, 2010; Stinson, 2010; Meiners, 2010; Creed, 2010; Dunn, 2010; Boulanger, 2009). There is ample evidence that the research community continues to find and utilize the CIT in many different academic fields: human sexuality, counseling, nursing, social welfare, teaching, health informatics, ergonomics, management education, higher education, leadership, and organizational psychology. In the time period since WWII literally thousands of studies claim to have used this technique.
Remember, the aim is to identify the behaviors that lead to fish. We are more interested in reporting successes than failures.
Citations
(n.b. not all citations appear in the text)
Akers, D. L. (2010). Backtracking events as indicators of software usability problems. Ph.D. Dissertation. Stanford University.
Anderson, B. & Nilsson, S. (1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology. 48(6). 398-402.
Bickman, L. & Rog, D. (1998). The handbook of applied social research methods. Thousand Oaks, CA: Sage Publications
Blouin, D. (2010). Reliability of a structured interview for admission to an emergency medicine residency program. Teaching and Learning in Medicine. 22(4), p. 246-250.
Boulanger, J. (2009). The first generation adult community college student: A case study of persistence strategies. An Ed.D. Dissertation. Columbia University.
Bengt, E.A. and Nilsson, S. (1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology. 48 (6), 398-402.
Brunton, M. & Jeffrey L. (2010). Using the critical incident technique for triangulation and elaboration of communication management competencies. Journal of Vocational Education and Training. 62(3), p. 239-255.
Conrad, C. & Serlin, R. (2005). The Sage handbook for research in education: Engaging ideas and enriching inquiry. Thousand Oaks, CA: Sage Publications.
Creed, M. W. (2010). Planning, problem solving, and Kirton’s A-I theory within an organizational framework. A Ph.D. Dissertation. University of North Carolina-Chapel Hill.
Creswell, J. W. (2005). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, N.J.: Merrill.
Creswell, J.W., (2009). Research design: qualitative, quantitative and mixed methods approaches. Thousand Oaks, CA: Sage Publications.
Denzin, N.K. & Lincoln, Y. (2011). The handbook of qualitative research. Thousand Oaks, CA: Sage Publications.
De Vaus, D. (2001). Research design in social science. Thousand Oaks, CA: Sage Publications.
Dharamsi, S., Richard, M., Louie, D., Murray, D., Berland, A., Whitfield, M. & Scott, I. (2010). Enhancing medical students’ conceptions of the CanMEDS Health advocate role through international service-learning and critical reflection: A phenomenological study. Medical Teacher. 32(12) p. 977-982.
Dunn, K. (2010). Toward an understanding of the epistemic values of biological scientists as expressed in scholarly publication. A Ph. D. Dissertation.
Rutgers State University of New Jersey-New Brunswick.
Emusu, D., Ivankova, N., Jolly, P., Kirby, R., Foushee, H., Wabwire-Mangen, F., Katongole, D., & Ehri, J. (2009). AIDS Care. 21(11), p. 363-370.
Flanagan, J. (1954). The critical incident technique. Psychological Bulletin. 51(4), p. 347-358.
Gould, S. (1985). The Flamingo’s smile: Reflections in natural history. New York: W.W. Norton.
Grady, M. & Bryant, M. (1991). School board presidents describe critical incidents with superintendents. Journal of Research in Rural Education, 7(3).
Heinrichs, K., MacKnee, C., Auton-Cuff, F., Domene, J. (2009). Factors affecting sexual-self esteem among young adult women in long-term heterosexual relationships. Canadian Journal of Human Sexuality. 18(4), p. 183-199.
Holtz, B. & Harold, C. (2008). When your boss says no! The effects of leadership style and trust on employees reactions to managerial expectations. Journal of Occupational and Organizational Psychology. 81(4), p. 777-802.
LeCompte, M.L., Millroy, W., Preissle, J. (1992). The handbook of qualitative research in education. San Diego, CA: Academic Press.
Lee, M. & Vennum, A. (2010). Using critical incident journaling to encourage cultural awareness in doctoral marriage and family therapy students. Journal of Family Psychotherapy. 21(4)., p 238-252.
Lewis, R., Yarker, J., Donaldson-Feilder, E., Flaxman, P., Munir, F. (2010). Using a competency-based approach to identify the management behaviors required to manage workplace stress in nursing: A critical incident study. International Journal of Nursing Studies. 47(3), 307-313.
Joyce, R. & Rankin, T. (2010). The lessons of the development of the first APA ethics code: Blending science, practice, and politics. Ethics and Behavior 20(9), p. 466-481.
Marshall, C. & Rossman, G. (1989). Deigning qualitative research. Thousand Oaks, CA: Sage Publications.
Meiers, R. P. (2010). The impact of structural and relational factors on supervisor employee negotiations. A Ph.D. Dissertation. Michigan State University.
Merritt, R. P. (2011). The decision making process of informal caregivers of dementia family members regarding nursing home placement. A Ph.D. Dissertation. Virginia Commonwealth University.
Piantanida, M. & Garmen, N. (1999). The qualitative dissertation. Thousand Oaks, CA: Sage Publications.
Regel, S. (2010). Psychological debriefing—does it work? Healthcare Counselling & Psychotherapy Journal. 10(2), p 14-18.
Rossman, G. & Rallis, S. (2003). Learning in the field: An introduction to qualitative research. Thousand Oaks, CA: Sage Publications.
Sverker, A., Ostlund, G., Hallert, C., & Hensing, G. (2009). “I lose all these hours”-Exploring gender and the consequences of dilemmas experienced in everyday life with coeliac disease. Scandinavian Journal of Caring Sciences 23(2), p. 343-352.
Sousa, L. & Costa, T. (2010). The multi-professional approach: front-line professionals’ behaviors and interactions. International Journal of Social Welfare. 19(4), p. 444-454.
Stinson, R. F. (2010). Critical incidents that lead to homelessness: Recommendations for Counselors. A Ph.D. Dissertation. University of Iowa.
Thomas, R.M. (2003). Blending qualitative and quantitative research methods in theses and dissertations. Thousand Oaks, CA: Sage Publications.
Trepal, H, Bailie, J., & Leeth, C. (2010). Critical incidents in practicum supervisees’ perspectives. Journal of Professional Counseling: Practice, Theory & Research. 38 (1), p 28-38.
Wong, E., Scott, L., Briseno, J., Crawford, C., & Hsu, J.Y. (2009). Determining critical nursing interventions for the critical care setting: A pilot study. International Journal of Nursing Terminologies & Classifications. 20(3), p110-121.
OR
Fly Fishing: Getting Skunked or Netting the Behemoth
Miles Bryant
University of Nebraska-Lincoln
Lincoln, NE 68588-0360
Critical Behaviors and Where They Take Us: A Research Design for the Doctoral Student and the Student of Practice
OR
Fly Fishing: Getting Skunked or Netting the Behemoth
Introduction
Life is full of situations where specific behaviors can lead to successful or unsuccessful outcomes. Athletic teams win or lose based on behaviors; critical care nursing practices may yield a recovered patient or a deceased patient; a classroom teacher’s questioning techniques may result in learning success or learning failure; the physical and observable steps made by an artisan bread baker may produce a delicious loaf of bread or a lump of inedible dough. And the behavior of the fly fisherperson on a stream, river or lake can lead to a fish or not. It is this last example of the behaviors of the fly-fisherperson that I will use to outline a very practical research design: the critical incident technique. And I have a very particular kind of fish in mind—trout—for most often the trout is the object of the fly-fisherperson’s actions.
The behaviors associated with fly-fishing for trout are complex and allow for many comparisons to the behaviors of the Critical Incident Technique (CIT) researcher. To illustrate, here are some examples of fly-fishing behaviors. One has to be strategic in determining where to fish for trout. One has to make many practical decisions about the equipment that one uses to catch a trout. One has to be receptive to all the evidence that may reveal where a trout lies in wait. One has to employ skill in casting for a trout. One has to behave in certain ways once one has hooked a trout. And within each of these general categories of behavior are multiple smaller examples of actions by the fly-fisherperson that can lead to success or failure. In the matter of selecting equipment, one can chose a fly that attracts a fish, a fly attached to the appropriate type of fly-line, leader, and tippet so that the artificial fly looks to the trout like a real fly looks.
Just as the fly-fisherperson wants to catch trout, the CIT researcher wants to identify behaviors that can be observed or recounted in a specific situation that clearly lead to success or failure. Think about this. In these areas we are often urged to locate our scholarship at the nexus of research and practice. Here is a research design firmly motivated by a goal to improve practice. The CIT is, then, located at the nexus of research and practice. This is a perfect place for a student of educational leadership.
The Elements of the CIT
The rest of this paper reviews the basic design elements of the CIT, using metaphors from the activity of fly-fishing to help illustrate the research design. The paper ends with comments about the utility of the CIT for the doctoral student specifically and the researcher in general.
1.Catching Trout: Determining the Aim
Trout fishing takes place somewhere where trout live. This is the setting. The CIT researcher is first of all interested in the behaviors that occur in a setting, seeking to know what results from very specific behaviors that are realistically associated with positive or negative outcomes. To repeat, it is the behaviors in that setting that the CIT researcher seeks to examine. One could, for example, simply enjoy the river and never lift a hand to the butt of a fly rod.
One might be more attentive to the contents of the coolers than on casting for a fish. Were one to do so, that behavior would obviate the selection of this particular incident as informative. But, it might also signal a behavior that clearly leads to failure—if catching a trout is the goal.
The CIT requires that we be curious about a common work experience or setting and that we seek to understand better the critical behaviors enacted in that setting. By critical, we need to know which behaviors clearly lead to success or failure. There are thousands of examples of workplace settings where CIT researchers gathered data about critical behaviors. In each study, the first step was to clearly state the aim or goal of the activity in question. If this reminds you of the “time and motion studies” of Taylorism or scientific management, it should. In many ways, the CIT is a child of that early management theory. Just as did the proponents of scientific management, the CIT researcher wants to assess a human performance aimed at an objective or goal.
When it comes to determining the aim, nothing expresses this core element quite as clearly as the goal of the trout fly-fisherperson. Catching fish is what it is all about.
2. Behaviors
It may seem obvious. To carry out a Cit study One has to be curious about the outcomes of specific human behaviors in a setting. That is, one has to want to know more about the factors that lead to goals or aims of an activity be it fly fishing or be it the behaviors that lead to superior customer satisfaction interactions in a grocery store (Bengt and Nilsson, 1964). It was this key element of observed behaviors that Flanagan (1954) the founder of the CIT advocated as the central strength of the research design. When Lewis, Yarker, et al., 2010) set out to identify the management behaviors that helped workplace stress in nursing, their goal was to find practical ways to improve the working conditions for nurses. To do so they wanted to identify critically important behaviors. When Grady and Bryant (1991) set out to identify the factors behind serious conflicts between superintendents and school boards, they wanted to know what specific behaviors were believed to have led to conflict.
In fly-fishing for trout, a skilled fisherperson will not just cast her fly anywhere on the water. Rather, she will take time to read the river and use her knowledge to be selective in where she casts. And she will know that if she is nymph fishing or casting dry flies, it will be important to keep her fly line from creating an unnatural drag on her fly. These are specific behaviors associated with critical outcomes.
In the mind of the CIT researcher, it is human behavior that leads to success or failure. The research design is oriented toward the identification of those behaviors.
3. The Setting, Incident and the Respondents
Just as fly fishing for trout takes place in specific types of places at specific times, so too does the critical incident. The CIT researcher wants not just general human interactions but human interactions that take place in a setting and situation that she or he has pre-defined. Thus, care must be taken to be very deliberate in delineating the stage (or stream) on which one wishes to gather data.
With the setting well defined, the CIT researcher can then proceed to study multiple incidents, recording the behaviors from each that were associated with critical outcomes. Not every setting will yield critical incidents. But many will. And it is those settings where clearly good or clearly bad results occurred that will drive the study. This accumulation of critical incidents through successive rounds of interviewing and analysis eventually leads to what Flanagan (1954) called saturation. A great range of critical behaviors has been identified and new iterations of incidents and interview yield no new data. At this point the CIT researcher is in a position to start to name a set of behaviors that are critical to success or failure. This, of course, is where the CIT becomes a great benefit to practice.
It should also be obvious that in order to gather data about critical behaviors, one’s respondents or interviewees must be knowledgeable. Best are those who observed behaviors and who can recall those observations in detail. Hearsay from third party participants in the incident can sometimes be valuable but the most descriptive information about behaviors will come from those who were in a position to observe or experience the behaviors.
4. Recorded Experience through Interviews
One of the key elements of the CIT is problematic. Most of the data that the CIT researcher will gather comes from interviews. And these interviews will be conducted by asking the respondent to recall a specific incident and the behaviors that she or he observed at the time.
I could approach a fly-fisherperson on the North Platte River and watch them fish. In so doing I might well be able to identify success or failure (fish or not); I might well be able to identify productive fishing behaviors; I might be able to make judgments about the fisherperson’s skills. I could also do this by asking the fisherperson to tell me about the one that got away. What were the behaviors that may have led to that loss. And, I could also ask the person to recall the behaviors that she or he recalled as important when the big trout was caught. Human memory is fallible. The recollected set of behaviors will, when amalgamated over a number of interviews and critical incidents be reliable. For this reason, more interviews about more incidents is best.
The interview is the main method of gathering data in the great majority of CIT studies. But, the interview differs from the typical qualitative interview that is viewed by many researchers as epistemologically subjective. The data gathered in a CIT study is certainly normally filtered through the subjective lens of the person recalling the incident. But, if the questioning techniques are skillful, the data will approximate what really happened. The CIT researcher is after the reality of the incident with its accompanying behaviors, not the subjective reality within the subjects mind. Thus, the CIT interview will feature a persistent effort to ask the respondent to identify behaviors and not present an interpretation of what took place.
The CIT: A behaviorally grounded research design approach that seeks to interview knowledgeable individuals inviting these individuals to recall specific incidents in which critical behaviors led to success or failure.
5. Success/Failure in the Lives of Ordinary Folk
Stephen Gould, the famous scientist and author who worked at Harvard for much of his career once observed that if you want to understand what ordinary folks do, one thoughtful deviant will teach you more than ten thousand solid citizens (Gould, 1985, p. 101). Gould could have written the banner for the CIT researcher. To amplify the importance and magnitude of behavior and its relationship to outcomes, the CIT researcher seeks egregious or exemplary behaviors.
You are telling me that this incident you remember led to a
disaster. Would you tell me what John did that led to such a disasterous outcome? What exactly did he do?
Armed with this question, you can begin to probe for specific behaviors.
You are seeking behaviors at what could be called the tails of a normal distribution.
It is more common for the researcher to seek to gather data from points within two standard deviations from the mean. In contrast the CIT researcher seeks data that can be found at the tails of the normal curve—these are Gould’s deviants and they can yield a great deal of information about what works and doesn’t work for ordinary “folk”.
For the fly fisherperson who gets “skunked” (the term for catching no fish), her behaviors are perhaps egregious, assuming there were any fish where she fished. On the other hand, if she caught a behemoth, we should be able to identify behaviors that led to this success
6. Applications and Conclusion
This CIT provides students and scholars with an accessible research method that allows them to produce applied and practical research findings. The CIT requires the researcher to gather data from research subjects about behaviors in a bounded context or work setting, behaviors that result in critically beneficial or critically ruinous outcomes. Generally these data are gathered through a well-planned interview process that focuses on detailed descriptions of human behavior.
Is the technique timely? Academic Search Premier, a library database, lists many scholarly articles that have been published in a great variety of journals over the last few years. Here we cite but a few (Trepal, et al., 2010; DeBrew, 2010; Regel, 2010; Wong et al., 2009; Heinrichs, et al., 2009; Dharamsi, et al., 2010; Joyce & Rankin, 2010; Sousa & Costa, 2010; Blouin, 2010; Lee & Vennum, 2010; Emusu, et al., 2009; Sverker, A., et al., 2009; Brunton & Jeffrey, 2010; Holtz & Harold, 2008). Over the past two years, doctoral students at such universities as Virginia Commonwealth University, Stanford, University of Iowa, Michigan State University, University of North Carolina-Chapel Hill, Rutgers University, and Columbia University have listed the CIT as a methodological tool used in completing their dissertations (Meritt, 2011; Akers, 2010; Stinson, 2010; Meiners, 2010; Creed, 2010; Dunn, 2010; Boulanger, 2009). There is ample evidence that the research community continues to find and utilize the CIT in many different academic fields: human sexuality, counseling, nursing, social welfare, teaching, health informatics, ergonomics, management education, higher education, leadership, and organizational psychology. In the time period since WWII literally thousands of studies claim to have used this technique.
Remember, the aim is to identify the behaviors that lead to fish. We are more interested in reporting successes than failures.
Citations
(n.b. not all citations appear in the text)
Akers, D. L. (2010). Backtracking events as indicators of software usability problems. Ph.D. Dissertation. Stanford University.
Anderson, B. & Nilsson, S. (1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology. 48(6). 398-402.
Bickman, L. & Rog, D. (1998). The handbook of applied social research methods. Thousand Oaks, CA: Sage Publications
Blouin, D. (2010). Reliability of a structured interview for admission to an emergency medicine residency program. Teaching and Learning in Medicine. 22(4), p. 246-250.
Boulanger, J. (2009). The first generation adult community college student: A case study of persistence strategies. An Ed.D. Dissertation. Columbia University.
Bengt, E.A. and Nilsson, S. (1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology. 48 (6), 398-402.
Brunton, M. & Jeffrey L. (2010). Using the critical incident technique for triangulation and elaboration of communication management competencies. Journal of Vocational Education and Training. 62(3), p. 239-255.
Conrad, C. & Serlin, R. (2005). The Sage handbook for research in education: Engaging ideas and enriching inquiry. Thousand Oaks, CA: Sage Publications.
Creed, M. W. (2010). Planning, problem solving, and Kirton’s A-I theory within an organizational framework. A Ph.D. Dissertation. University of North Carolina-Chapel Hill.
Creswell, J. W. (2005). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, N.J.: Merrill.
Creswell, J.W., (2009). Research design: qualitative, quantitative and mixed methods approaches. Thousand Oaks, CA: Sage Publications.
Denzin, N.K. & Lincoln, Y. (2011). The handbook of qualitative research. Thousand Oaks, CA: Sage Publications.
De Vaus, D. (2001). Research design in social science. Thousand Oaks, CA: Sage Publications.
Dharamsi, S., Richard, M., Louie, D., Murray, D., Berland, A., Whitfield, M. & Scott, I. (2010). Enhancing medical students’ conceptions of the CanMEDS Health advocate role through international service-learning and critical reflection: A phenomenological study. Medical Teacher. 32(12) p. 977-982.
Dunn, K. (2010). Toward an understanding of the epistemic values of biological scientists as expressed in scholarly publication. A Ph. D. Dissertation.
Rutgers State University of New Jersey-New Brunswick.
Emusu, D., Ivankova, N., Jolly, P., Kirby, R., Foushee, H., Wabwire-Mangen, F., Katongole, D., & Ehri, J. (2009). AIDS Care. 21(11), p. 363-370.
Flanagan, J. (1954). The critical incident technique. Psychological Bulletin. 51(4), p. 347-358.
Gould, S. (1985). The Flamingo’s smile: Reflections in natural history. New York: W.W. Norton.
Grady, M. & Bryant, M. (1991). School board presidents describe critical incidents with superintendents. Journal of Research in Rural Education, 7(3).
Heinrichs, K., MacKnee, C., Auton-Cuff, F., Domene, J. (2009). Factors affecting sexual-self esteem among young adult women in long-term heterosexual relationships. Canadian Journal of Human Sexuality. 18(4), p. 183-199.
Holtz, B. & Harold, C. (2008). When your boss says no! The effects of leadership style and trust on employees reactions to managerial expectations. Journal of Occupational and Organizational Psychology. 81(4), p. 777-802.
LeCompte, M.L., Millroy, W., Preissle, J. (1992). The handbook of qualitative research in education. San Diego, CA: Academic Press.
Lee, M. & Vennum, A. (2010). Using critical incident journaling to encourage cultural awareness in doctoral marriage and family therapy students. Journal of Family Psychotherapy. 21(4)., p 238-252.
Lewis, R., Yarker, J., Donaldson-Feilder, E., Flaxman, P., Munir, F. (2010). Using a competency-based approach to identify the management behaviors required to manage workplace stress in nursing: A critical incident study. International Journal of Nursing Studies. 47(3), 307-313.
Joyce, R. & Rankin, T. (2010). The lessons of the development of the first APA ethics code: Blending science, practice, and politics. Ethics and Behavior 20(9), p. 466-481.
Marshall, C. & Rossman, G. (1989). Deigning qualitative research. Thousand Oaks, CA: Sage Publications.
Meiers, R. P. (2010). The impact of structural and relational factors on supervisor employee negotiations. A Ph.D. Dissertation. Michigan State University.
Merritt, R. P. (2011). The decision making process of informal caregivers of dementia family members regarding nursing home placement. A Ph.D. Dissertation. Virginia Commonwealth University.
Piantanida, M. & Garmen, N. (1999). The qualitative dissertation. Thousand Oaks, CA: Sage Publications.
Regel, S. (2010). Psychological debriefing—does it work? Healthcare Counselling & Psychotherapy Journal. 10(2), p 14-18.
Rossman, G. & Rallis, S. (2003). Learning in the field: An introduction to qualitative research. Thousand Oaks, CA: Sage Publications.
Sverker, A., Ostlund, G., Hallert, C., & Hensing, G. (2009). “I lose all these hours”-Exploring gender and the consequences of dilemmas experienced in everyday life with coeliac disease. Scandinavian Journal of Caring Sciences 23(2), p. 343-352.
Sousa, L. & Costa, T. (2010). The multi-professional approach: front-line professionals’ behaviors and interactions. International Journal of Social Welfare. 19(4), p. 444-454.
Stinson, R. F. (2010). Critical incidents that lead to homelessness: Recommendations for Counselors. A Ph.D. Dissertation. University of Iowa.
Thomas, R.M. (2003). Blending qualitative and quantitative research methods in theses and dissertations. Thousand Oaks, CA: Sage Publications.
Trepal, H, Bailie, J., & Leeth, C. (2010). Critical incidents in practicum supervisees’ perspectives. Journal of Professional Counseling: Practice, Theory & Research. 38 (1), p 28-38.
Wong, E., Scott, L., Briseno, J., Crawford, C., & Hsu, J.Y. (2009). Determining critical nursing interventions for the critical care setting: A pilot study. International Journal of Nursing Terminologies & Classifications. 20(3), p110-121.
Introduction
This is a new blog (new as of Nov of 2011) that seeks to link researchers in educational and social sciences who are interested in the critical incident technique. This research approach is widely used in applied research and offers some very distinctive features not common in more prevalent research methods.
Information from readers who have used this approach is welcome.
Miles Bryant
Information from readers who have used this approach is welcome.
Miles Bryant
Subscribe to:
Posts (Atom)