Clinical Evaluation of NOC Outcomes

12 International Journal of Nursing Terminologies and Classifications Volume 14, No. 4 Supplement, October-December, 2003

Presented Papers: Nursing Interventions

Presented Papers: Nursing Outcomes

Nursing-Sensitive Outcome Implementation and Reliability Testing in a Tertiary Care Setting

Julia G. Behrenbeck

PURPOSE. To describe the NOC outcomes most rele-vant for specialty nursing practice and in selected fieldsites representing the continuum of care; to assess theadequacy of measures (reliability, validity, sensitivity,specificity, practicality); and to describe the linkagesamong nursing diagnoses, interventions, and outcomesin clinical decision making.

METHODS. Data were collected on 434 patients dur-ing the 12-month data collection period at a tertiary carecenter: cardiac surgery intensive care (n = 76), cardiactransplant unit (n = 153), and medical unit (n = 205).Medical diagnoses of patients on the two cardiac unitswere related to cardiac disease. Medical diagnoses of pa-tients on the medical unit were extremely varied (rang-ing from e.g., gout to pneumonia). Data were collectedon 65 separate outcome labels for a total of 633 ratings.

FINDINGS. In the cardiac transplant ICU, data werecollected on 42 outcomes: 30 had an average interrater re-liability of ≥85%, and 16 had an absolute agreement inter-rater reliability of ≥85%. In the cardiac surgery ICU, data

were collected on 30 outcomes: 25 had an average inter-rater reliabilty of ≥85%, 6 had an absolute agreement in-terrater of ≥85%. In the medical unit, data were collectedon 45 outcomes: 41 had an average interrater reliability of≥85%, 14 had an absolute agreement interrater reliabilityof ≥85%. Four outcomes have been implemented into thedocumentation system for all patients: Tissue Integrity:Skin and Mucous Membranes, Mobility Level, Knowl-edge: Disease Process, and Coping.

CONCLUSIONS. Overall, nursing staff were very pos-itive about having the opportunity to participate innursing research. Staff were able to think about the rela-tive status of their patient and how nursing care con-tributes to the patient's recovery. They appreciated theopportunity to discuss this with a colleague during theinterrater exercise. Increased familiarity with NOC al-lows staff members to determine which outcomes com-prise core nursing-sensitive outcomes for their clinicalsetting.

Author contact: [email protected]

Personal Autonomy: Development of a NOC Label

Carol Caldwell, Dianne Wasson, Veronica Brighton, Lois Dixon, and Mary Ann Anderson

PURPOSE. To develop a NOC label for the concept ofpersonal autonomy.

METHODS. Using a focus group approach, a NOClabel, definition, and indicators were developed for per-sonal autonomy. Using a concept analysis method, theNOC Psychological/Cognitive Focus Group constructeda conceptual definition of personal autonomy and createda list of indicators. An extensive review of nursing, psy-chology, ethics, and philosophy literature from 1988 to2001 was conducted using approximately 30 empiricallybased and theoretical sources. The focus group identified

pertinent data from the literature and clustered the datainto the predetermined categories of conceptual definitionand indicators, giving careful attention to the rules of de-velopment for outcome definitions and indicators.

FINDINGS. The label, definitions, and list of indica-tors were conceptualized at several levels. The defini-tion and indicators were refined into a conceptually andclinically coherent outcome. Personal autonomy is de-fined as �the ability of a competent individual to governone's own life and make choices based on the informa-tion available.�

BACKGROUND. Nursing has increasingly found itselfin the position of having to demonstrate it makes a dif-ference in the outcome of a patient's recovery. How careis documented plays an important role in validating theeffect nursing interventions have on patient outcomes.Standardized nursing languages have been and arebeing developed to provide a common basis of docu-mentation, communication, and comparison of nurse-sensitive patient outcomes.

MAIN CONTENT POINTS. Two cardiac surgery ICUsparticipated as clinical test sites for The University ofIowa continuation grant to refine and validate measuresfor NOC outcomes. In the cardiac transplant ICU, datawere collected on 42 outcomes: 30 had an average inter-rater reliability of ≥85%; 16 had an absolute agreementinterrater reliability of ≥85%. The top 4 outcomes using80% of the indicators were Communication Ability, Neu-rological Status, Tissue Integrity: Skin and Mucous Mem-branes, and Vital Sign Status. The bottom 4 outcomesusing ≤51% of the indicators were Nutritional Status:

International Journal of Nursing Terminologies and Classifications Volume 14, No. 4 Supplement, October-December, 2003 13

Food and Fluid, Respiratory status: Ventilation, TissuePerfusion: Cardiac, and Safety Behavior: Fall Prevention.

In the cardiac surgery ICU, data were collected on 30outcomes: 25 had an average interrater reliability of≥85%; 6 had an absolute interrater reliability of >85%.The top 5 outcomes using ≥87% of the indicators wereBlood Transfusion Reaction Control, Knowledge: Medi-cation, Pain level, Vital Sign Status, and Nutritional Sta-tus: Nutrient Intake. The bottom 4 outcomes using ≤54%of indicators were Comfort Level, Infection Status, TissuePerfusion: Cardiac, and Safety Behavior: Fall Prevention.

The study implementation process included meetingsand inservice classes (many one on one) with nursingleadership, resource "super user" RNs, and other staffRNs. Several 1-hour educational inservice classes wereoffered. Once the study was progressing, charts, e-mailmessages, reminder notes, and attending and reportingstudy progress at staff meetings were used to sustainstaff interest and stimulate memory.

Staff anecdotal impressions of the study ranged


The indicators are■ Demonstrates cognitive ability to make an informed

decision.■ Is able to receive a thorough disclosure of pertinent in-

formation about alternative treatments for condition.■ Comprehends the information received in thorough

disclosure of pertinent information about alternativetreatments for condition.

■ Comprehends the risks versus benefits of alternativetreatments

■ Makes choice from alternatives available.■ Expresses satisfaction in choice made.■ Expresses noninterference with decision-making pro-

cess by family.■ Expresses noninterference with decision-making pro-

cess by friends.■ Expresses noninterference with decision-making pro-

cess by healthcare providers.■ Expresses noninterference with decision-making pro-

cess by (other).■ Makes decisions free from undue pressure.

DISCUSSION. Nurses are educated to value and pro-tect clients' rights to make their own decisions in health-care matters. Informed consent is an essential value ofthe profession, but the concept of clients' competence to

receive the information, weigh all options, and make achoice free from undue pressure is often minimized oroverlooked. Not only must nurses advocate for clients'right to make decisions, they must be vigilant in ensuringthat patients are cognitively able to make those decisions.

One school of thought argues that true autonomy isnot possible, given that individuals exist within relation-ships: People cannot make decisions without pressurefrom the people in those relationships. The NOC Psycho-logical/Cognitive Focus Group suggests that while indi-viduals may consider the wishes of friends, family, andhealthcare providers, the person retains the right to makethe decision that best suits him/herself. Nurses are in aunique position to offer support to individuals as theyattempt to balance the information about treatment op-tions and the suggestions/demands of others.

CONCLUSIONS. Autonomy is not a simple concept. Itchallenges nurses to view individuals within the contextof relationships and with consideration of their cognitiveabilities. Use of NOC outcome indicators can facilitate animproved awareness of the facets of autonomy. Testingthe indicators in clinical practice will be necessary to con-firm this conclusion.

Author contact [Caldwell]: [email protected]

NOC Implementation and Testing in a Cardiac Surgery ICU

Kathy Demmer


NOC, the staff is determining which outcomes comprisecore nursing-sensitive outcomes that reflect the type ofnursing care given in an ICU. Two new outcomes regardingthe ventilator weaning process were developed. These havebeen submitted to the University of Iowa for consideration.


widely from thinking that the process was too time-consuming and lacked meaning, to discovering that theoutcome label and indicators really helped describe whatthe nurse assesses when taking care of a cardiac inten-sive care patient.

CONCLUSIONS. Through being more familiar with


Clinical Evaluation of NOC Outcomes

Marion Johnson, David Reed, Meridean Maas, and Sue Moorhead

PURPOSE. To evaluate NOC outcomes in field sitesacross the care continuum; to identify core outcomes andlinkages among diagnoses, interventions, and outcomes.

METHODS. Researchers tested 190 patient outcomes,developed and tested for content validity and coded in ataxonomic structure in an earlier phase of the research,for reliability, validity, sensitivity, and usefulness in 10field sites: 2 academic, tertiary hospitals; 3 communityhospitals; 1 long-term care nursing home; 1 community-based parish nursing practice; 1 ambulatory, academicnursing center staffed by nurse practitioners; and 2 com-munity health nursing agencies providing home care(visiting nurses) in 4 midwestern states.

Reliability: Two types of interrater agreement wereassessed. Near agreement was defined as the numericalratings that do not differ by more than 1 value on the 5-point Likert-type scale and include both the label andits indicators. Absolute agreement is the percent of abso-lute or total agreement between 2 nurses' ratings for theoutcome, irrespective of the indicator ratings.

Validity: For most of the outcomes, standardized toolsthat measure the same or similar concept were located toestimate construct validity. No standardized tools werelocated for some NOC outcomes, especially those thatdescribe discrete physiologic outcomes. To estimate thevalidity of these outcomes, specified information was ex-tracted from patients' health records. Pearson correlationwas used to determine similarity between the NOCmeasure and the standardized tool.

Sensitivity: Sensitivity is the extent that a measure cap-tures change in a phenomenon. This sensitivity to changeis often called responsiveness to change in health out-comes research. The smaller the change the measure isable to gauge, the more responsive it is. The sensitivity ofNOC outcomes is examined by comparing two measures

of the outcome on the same patient at two different times.Usefulness: Because the outcome labels will be used

by nurses to monitor and evaluate the effects of inter-ventions for patients, they must be clear with unambigu-ous definitions and measurement scales that nurses canrate (score) with ease and confidence. Field-site nurseswere asked to note their comments about difficultiesusing the outcomes and measures, and to provide sug-gestions for revisions on the data collection forms. Theinvestigators also met with the nursing staff to obtainfeedback.

FINDINGS. Data from approximately 2,300 patientshave been collected. Based on the results to date, there iscautious optimism that most of the NOC outcomes andmeasures will demonstrate an acceptable degree of inter-rater reliability. The percent agreement on many of theoutcomes is ≥80%, and the majority of weighted Kappacoefficients are ≥.60. Validity testing of the outcomes,while encouraging, shows a greater variance in correla-tion scores. This variance is directly related to the com-patibility of the two measures.

CONCLUSIONS. The research team is continuing todevelop, validate, and classify outcomes and outcomemeasures for individuals, families, and communities.Several issues are being addressed by the research--forexample, the issue of whether the outcome indicatorsshould be rated, and if so, how the final rating for theoutcome should be determined. One site is rating theoutcome label only and not the indicators. If these label-only ratings are shown to be as reliable and valid aswhen indicators are rated, it may suggest that nursesneed only rate the outcome label. This is an importantissue, given that most nurses are highly concerned aboutthe burden of documentation.

Author contact [Johnson]: [email protected]

PURPOSE. To measure and provide evidence sup-porting the use of NOC as a nursing standardizedlanguage.

METHODS. Initially, the group used nursing resourcesand concept analyses to determine physiologic outcomesimportant to nursing care and demonstrate contributionsof nurses to patient care. After refinement by the NOCteam, outcome and indicators were approved and con-tent validity surveys sent to members of the AmericanAssociation of Nephrology Nurses. A modified Fehring'sDiagnostic Content Validity Model was used to deter-mine validity. Respondents used a 5-point Likert-typescale. Each indicator was rated and a weight assigned,

International Journal of Nursing Terminologies and Classifications Volume 14, No. 4 Supplement, October-December, 2003 15

and a mean for each indicator was determined. Resultsof the importance and contribution scores were inter-preted using ≥.80 to indicate a "critical" indicator, and≥.60 to indicate "important" indicators.

FINDINGS. There were 59 total responses with 45valid cases. The nurses had worked an average of 21.4years. Twenty-five of the nurses were involved in imple-menting standardized language, including care-planningsystems. The majority, 95%, held master's degrees innursing and 82.2% were certified. All results were ana-lyzed for both contribution and importance using the es-tablished method to determine critical and important in-dicators in both categories.


Nursing Outcomes Across a Surgical Care Episode

Aleta Porcella

PURPOSE. To test the validity and reliability of theNOC labels.

METHODS. 94 NOC labels were assigned to neuro-logic, orthopedic, and surgical patients. Because of anursing shortage and nurses' objections to the time-con-suming nature of the study procedures, changes weremade in the data-gathering process. Realizing that busyfloor nurses would complete ratings on labels that hadalready been started but were reluctant to search through94 different labels to find the ones most critical to theirparticular patient, the pool of selectable NOCs was short-ened by nurse experts from 94 to a handful deemed mostcritical to the patient population. Outpatient nurses se-lected NOCs during patients' presurgical work-up andcompleted a first rating. Clinic nurses were asked tocomplete a third rating at the patients' postoperativevisit. This left only a second rating and some interraterreliability ratings to be done by inpatient nurses.

FINDINGS. Results from the surgical patient populationinvolved 122 subjects with preoperative clinic visit ratingsand postoperative inpatient setting ratings. Postoperativeclinic visit ratings were obtained on 94 of the 122 subjectsas well. Twelve different NOCs were used for this popula-tion. Most frequently chosen were Comfort Level (n = 78),Pain Level (n = 26), and Ambulation: Walking (n = 24).

For Comfort Level the average change from first tosecond rating (mean difference) was 1.12. Change fromfirst to third rating was 1.53. Change from second to

third rating was 0.48. These mean differences are all sig-nificant at the p<.001 level. Subjects who were rated onComfort Level in the inpatient setting only (n = 14)demonstrated a change from first to second rating of0.77, from first to third rating 1.43, and from second tothird rating of 0.67. The n on other individual labels ofthe study subset was too small for meaningful tests ofstatistical significance. However, grouping all labels to-gether, all mean difference ratings for this population aresignificant at the p<.001 level.

DISCUSSION. NOC is conceptualized as a neutral pa-tient state that will vary across the life span of the indi-vidual as that person's health status changes. It seems ax-iomatic that ratings spaced widely across the span of asurgical episode of care would demonstrate greaterchange than those clustered during the inpatient portionof the episode of care. Although very small, this studylends credence to the validity of the language to capturepatient state changes.

CONCLUSIONS. The question of when to rate NOCs isof more than passing theoretical interest. It is critical tothe appropriate use of outcomes. Although this workneeds to be replicated with a larger sample size, differentdiagnosis-related groups, and at different institutions,this preliminary work seems to support the ability ofNOC to demonstrate significant changes in outcome rat-ings across an acute surgical episode of care.


Validity Surveys: Selected NOC Physiologic Outcomes

Cindy A. Scherb and Sandra L. Bellinger

PURPOSE. To develop a nursing outcome label for theconcept of hyperactivity.

METHODS. Using concept analysis of an exhaustive re-view of the nursing and related literatures � includingpsychology, pediatrics, and general medicine�the NOCPsychological/Cognitive Focus Group constructed a con-ceptual definition for Hyperactivity Level and delineatedthe indicators. The group identified pertinent data fromthe literature and clustered the data into the predeter-mined categories of conceptual definition and indicators.

FINDINGS. Results included a conceptual definition forHyperactivity Level label: the severity of patterns of inatten-tion or impulsivity in children, and a list of 23 indicators thatreflect all dimensions of the outcome Hyperactivity Level. ■ Inattention■ Lack of active listening■ Difficulty organizing tasks■ Inability to stay "on task" ■ Lack of follow-through or completion of activities■ Difficulty with tasks that require sustained mental effort■ Careless mistakes■ Frequently loses things, excessive distractibility, exces-

sive forgetfulness■ Impulsivity, excessive fidgeting, inability to remain seated


■ Excessive running and/or climbing■ Excessive motor behavior■ Difficulty playing quietly■ Excessive talking■ Blurts out answers before the question is completed■ Difficulty awaiting turn■ Excessive interrupting of others; intrusive, abrasive,

loud, interpersonal interactions■ Inappropriate aggressive behavior■ Difficulty in keeping hands to self

CONCLUSIONS. While most people do not outgrowADHD, people can learn to control hyperactivity urgesand live satisfying lives. Using combinations of medica-tion, new behavior skills, and emotional support, peoplewith ADHD can develop ways to control their attentionand minimize their disruptive behaviors. With the skillsof the nurse as well as support from parents, teachers,and other healthcare providers, children with ADHD arebetter prepared to manage their hyperactivity and chan-nel it into more socially acceptable behaviors. By usingselected nursing outcomes and interventions, the childcan improve his self-esteem through the development ofself-control.

Author contact [Wasson]: [email protected]


Hydration: 13 indicators analyzed. ■ Contribution: No critical indicators; 11 were rated im-

portant (range: 60�77). ■ Importance: 8 indicators rated critical. The 4 highest

were shortness of breath not present (.90), blood pres-sure within normal limits (.89), peripheral edema notpresent (.88), and adventitious breath sounds not pres-ent (.88). Five were rated important (range: 69�78).

Urinary Incontinence: 19 indicators analyzed. ■ Contribution: No critical indicators; 9 were rated im-

portant (range: 61�74). ■ Importance: 5 indicators were rated critical, the 3

highest being recognizes urge to void (.88), absence ofurinary tract infection (.86), and absence of postvoidresidual (.85) (range: 67�88).

Dialysis Access Integrity: 15 indicators analyzed. ■ Contribution: 3 were rated critical: drainage at site not

present (.83), hematoma at site not present (.82), andbleeding at site not present (.81). There were 11 impor-tant indicators (range: 60�78).

■ Importance: 7 were rated critical, the 3 highest being

bruit (.97), thrill (.95), and drainage at site not present(.95). Seven were rated as important (range: 64�77).

Systemic Toxin Clearance�Dialysis: 14 indicators analyzed. ■ Contribution: No critical indicators and 12 important

indicators (range: 60�78).■ Importance: 5 were rated critical, the top 3 being ad-

heres to dialysis schedule (.95), urea reduction ratio(.95), and blood potassium within normal limits (.88).Nine were rated important (range: 63�77). DISCUSSION. Based on these results, the indicators are

representative of the outcomes. Analyses of the indica-tors supported an overall level of agreement. The highestaverages were in two outcomes related to patients un-dergoing dialysis, which may be because the nursescared for a specific target group.

CONCLUSIONS. The importance of indicators wasrated more highly than the nursing contribution. Most ofthe indicators were judged quite or very important. Theoutcomes are nursing sensitive, and nursing is more in-fluential than other disciplines for most of the outcomes.

Author contact [Bellinger]: [email protected]

Hyperactivity Level: Development of a Nursing Outcome Label

Dianne Wasson, Lois Dixon, Veronica Brighton, Carol Caldwell, and Mary Ann Anderson

Clinical Evaluation of NOC Outcomes

Documents

Transcript of Clinical Evaluation of NOC Outcomes