DevOps Workshop - Addressing Quality Challenges of Highly Complex and Integrated Applications
Addressing Large, Complex, Unstructured Problems
-
date post
17-Oct-2014 -
Category
Education
-
view
325 -
download
2
description
Transcript of Addressing Large, Complex, Unstructured Problems
Addressing Large, Complex, Unstructured Problems
Written by Alexa DiBenedettoAdvised by Roger Hoerl
Senior ThesisMarch 2014
I. Introduction
Abstract
We are all faced with problems throughout our lives, not only in our educations and
careers, but also in our personal lives and relationships. Despite the universal need for
effective means of addressing problems, there is surprisingly little agreement in the
statistical or scientific literature as to how to approach problems in general. The problem
solving frameworks that do exist tend to be designed for narrow, well-defined problems.
Unfortunately, the most significant problems faced by modern society tend to be large,
complex, and unstructured, that is, not well defined. I have identified major themes in the
problem solving literature across disciplines, which I have divided into two macro-
themes: six key phases that any problem solver must pass through and consciously
consider during the problem solving process, and the key considerations that should be
kept in mind during every step of the problem solving process. In addition, I have made
some recommendations to problem solvers who face large, complex unstructured
problems, such as understanding that there is no one best method for solving every
problem, being able to use a variety of tools and techniques, even if we aren’t familiar
with them, and allowing yourself more time than you think is necessary to work through
the problem solving process.
Large, Complex, Unstructured Problems
The term “large, complex, and unstructured problem” itself can be a bit ambiguous. Thus,
I explain what I mean by large, complex, and unstructured in terms that are general
enough they can be applied in any field, not just statistics. One characteristic of these
types of problems is that they are large in scale. For instance, solving a textbook problem
2
usually involves analyzing manageable data sets that we can simply enter into Excel or
statistical software, and apply regression, hypothesis testing, and so on. On the other
hand, real-world problems usually are not neat and tidy in this way, as there are often
extremely large amounts of data and information being gathered over a large span of time
or from many sources.
These problems are also complex. While this term is somewhat ambiguous, in this
instance it can mean that the thing we are trying to “solve” is complex, or the tools
needed to solve it are complex; clearly, there is no “correct” or standard solution. In most
cases, there are multiple stakeholders who each have a different objective within the
problem. Usually, the relationship between these stakeholders is unclear or undefined,
which causes even more complexity. There also may be legal or ethical issues in these
kinds of problems, in addition to questions about who should be responsible for
conducting and funding the research necessary to solve them. By highlighting all of these
complexities, it becomes clear why we must treat large complex problems differently
than we do regular “textbook” problems.
Lastly, these problems are unstructured, meaning that a problem is not necessarily
precisely defined, or set up with a clearly defined set of data, background information,
problem, or solution. Sometimes, researchers do not agree on exactly what question they
are trying to answer, or what problem they are solving. In this case, it becomes
impossible to simply plug in equations and regression models etc. Many preliminary
steps must be taken to assess the root of the problem being posed, such as why it is being
asked, what is important about it, what is the significance of the results we are aiming to
obtain, and so on. This aspect of problems makes them most difficult to address in a
3
cookie cutter way. However we can hopefully establish a way to streamline this process
so that it becomes easier and more manageable to attack such problems.
Motivation
While there is not much research being done on methods of solving large,
complex, and unstructured problems, there has been some thought into the challenges
caused by not having a systematic way of attacking such problems. Hoerl and Snee
(Closing the Gap 52) highlighted one of the main issues regarding this type of problem
solving: often, most of the emphasis is placed on the tools and methods used to solve
problems, and there is little done to teach students and researchers to apply these tools in
a more broad setting where the problems are not neatly laid out like they would be in a
textbook. This process, sometimes referred to as “statistical engineering,” might prove to
be the most important aspect in solving large, complex, unstructured problems, as it
focuses on creatively applying the tools we learn from doing “cookie-cutter” examples
and problems in school. This article actually suggests that the statistical problem solving
process only relies 20% on studying the “pure” science of statistics, and that 80% is
dependent on statistical engineering, where we consider how to link and integrate these
tools. Thus, the need to establish a structured way of solving large, complex, and
unstructured problems becomes clear.
The Mathematics Department at Harvard University recognized the need to
develop courses to prepare students to address the kinds of problems they would face in
the “real world” as statisticians or researchers. Thus, a new course was created, titled
“Statistics 399: Problem Solving in Statistics,” with the goal of “helping Statistics PhD
students transition through the qualifying exams and into research”(Harvard Statistics
4
website). The professors who created this class point out that well written exams not only
benefit the students taking them, but also help the professors who are writing them
(Blitzstein, Meng 1-2). They note that there is often no textbook or library to simply copy
and paste exam questions that are unique and also stimulating. Thus, there is a need for
creative problem generation, and even more importantly, creative problem solving. None
of these problems have cookie-cutter solutions. The goal of a course like Stat 399 is to
allow students to develop skills that will not only help them to solve simple, straight-
forward problems like the ones they’ve seen in class, but also recognize the similarities
and differences between various problems in order to allow them to solve problems that
are not as straightforward. These problems, called “nano-research problems,” are meant
to mimic real world examples and research.
Example
For the purposes of this paper, we can study large, complex, and unstructured problems
through the analysis of a particular example, making it easier to understand key concepts
and pieces of advice. An example of a real-life problem that is large, complex, and
unstructured is determining how to provide an affordable yet comprehensive healthcare
system to Americans. The amount of data that is available to study is immense – patient
records, insurance claims, Medicare/Medicaid records, and more, gathered over the last
few decades. Additionally, the scope of the problem in itself is large – it involves all
Americans.
In addition, the problem is unstructured – no one solution is “correct” because
there are many varying viewpoints on what is considered a “good” healthcare system and
what we even mean by the word “good.” Who is the judge of the “goodness” of the
5
healthcare system? The government? The citizens of America? Insurance companies?
Doctors? The list is almost endless. Further, what exactly do we mean by “Americans”?
Do we mean people who were born in America? Anyone who lives in America, including
or not including illegal immigrants? Additionally, how do we define the problem? Are we
searching for a solution that is beneficial to the most people for the least amount of out-
of-pocket expense? Most profitable for the medical profession or private insurance
companies? Least expensive to the US Government? Has the highest rate of surgical
success? Least number of fatalities? Clearly, this list goes on indefinitely.
Our problem is also complex. We also have no metric to measure the success of the
healthcare model we come up with, because our original problem was not well defined.
There are multiple stakeholders, each with different objectives: doctors, patients, the US
government, insurance companies, pharmaceutical companies, hospitals, and more. There
are many legal and ethical issues regarding the healthcare system: is it ever appropriate to
refuse treatment to someone? Should cost be considered as a factor, or is age more
important? How important is the risk of failure or probability of success? Who should
pay for medical research to create this healthcare model? What is the role of individual
responsibility – if a person chooses to participate in inherently dangerous activities,
should they pay a higher premium? Who should pay for medical insurance – an
individual’s employer, the government, or the individual? This list is almost endless. It is
clear from this real life example that large, complex, unstructured problems could benefit
from a systematic approach, and that this would make solving problems in the future
much more consistent and simplified.
6
II. Literature Review and Interviews
Researchers in almost every discipline face large, complex, and unstructured problems.
Thus, an important step in creating guidelines for solving such problems that can be
applied to a variety of fields is researching and analyzing the available literature across
disciplines. While we have not interviewed professors in all disciplines or researched all
fields, we have taken a sampling of relevant disciplines that have something to say about
the process of problem solving, and acknowledge that our list may leave some input out,
and is certainly not exhaustive.
Input from Psychology
There has been some research on the problem solving process in Psychology, which
focuses on brain activity and the personality traits that contribute to the way we solve
problems. One finding has been about the “aha moment,” or the sudden moment of
clarity during which a person realizes the answer to the problem they have been trying to
solve. In their article, called “Gaining Insight into the Aha Experience,” Topolinski and
Reber define and interpret the “aha moment.” The four key components of the “aha”
experience are: suddenness, ease, positive affect, and the feeling of being right (403).
Suddenness refers to the idea that this “aha” moment comes unexpectedly and
immediately. The insight is gained and processed easily, and is gratifying. Lastly, the
problem-solvers have confidence in their solution, before they have formally tested the
“correctness” of it. In other words, “insight is an experience during or subsequent to
problem-solving attempts, in which problem-related content comes to mind with sudden
7
ease and provides a feeling of pleasure, the belief that the solution is true, and confidence
in this belief” (402-3).
Another interesting idea that this article proposes is the fact that when a general-
knowledge question is more easily processed, people feel more confident in their
answers, independent of their actual knowledge. Additionally, the sudden onset of a
solution is sometimes enough to make a person feel more confident in his or her answer
(403-4). In conclusion, the authors state, “when a solution to a problem pops into a
person’s mind, information that has been difficult to process can be processed more
fluently” (404). One negative aspect of this moment of insight is the fact that most of the
time it cannot be artificially induced, and its onset cannot be predicted. Thus, while it can
be a helpful aspect of solving a problem, we cannot rely on experiencing it when we are
solving problems.
In addition to exploring the “aha moment,” researchers in Psychology have
attempted to understand the way people make decisions based on different personality
characteristics. Professor Graham Wilson describes the way our personalities affect our
decision-making styles in a chapter of his manual (Decision-Making and Problem
Solving). The objectives of this chapter are: to identify your personal psychological type
and relate it to your personal preferences, to describe factors and personal styles that have
an impact on decision making, to distinguish between situations requiring individual
decisions and those requiring group decisions, and to identify the attributes of effective
decision makers. This unit is based on the Myers-Briggs Type Indicator (MBTI), and
highlights the different mental processes we use to think, as established by psychiatrist
Carl Jung: taking in information (perceiving), and organizing information and drawing
8
conclusions (judging) (3-1). Jung also differentiated between two different ways people
either perceive or judge: people take in information (perceive) by either using their sense
or their intuition, while people organize information by either thinking or feeling (3-2).
Combining these with one other set of preferences, we have the MBTI, which assesses
preferences on the following scale: extroversion vs. introversion (where energy is derived
and focused), sensing vs. intuition (how information is obtained), thinking vs. feeling
(how decisions are made), and judging vs. perceiving (how a person is oriented toward
the external world). Combining these four scales, there are a total of 16 psychological
types, none of which is “right” or “wrong,” “better” or “worse” (3-3-4).
This manual suggests that our preferences inherently affect the way we make
decisions. The results of sensing vs. intuition (S/N) and of thinking vs. feeling (T/F) have
a lot to do with our personal decision making styles. Sensing favors stability, basing
decisions on past experiences, while intuition favors innovation, basing decisions on
creativity and insight. On the other hand, thinking favors effectiveness, objectivity, and
logic, while feeling favors integrity, and considering people’s values and needs (3-4). The
critical thing to take away from this chapter is that a person cannot effectively make
decisions if they do not combine all of these aspects, using sense and intuition, thinking
and feeling (3-5).
Another important consideration is who should be making the decisions. Four
categories are presented: individual, consultation, group, and delegation (3-6). In
individual decision-making, the leader must make the decision alone, collecting input
from others only when it is relevant and necessary. When decision making through
consultation, the leader shares the issue with one or more people, who all provide input,
9
although the final decision may or may not take this input into consideration. Group
decision-making is similar, however all participants have an equal say in the final
decision. During delegation, the leader chooses someone to make the decision within
certain guidelines set forth by the leader (3-6). One thing to avoid during group decision-
making is a phenomenon called “groupthink,” where members “let their need to agree
with each other interfere with their ability to think about the decision critically” (3-8).
This is where the group leader plays a huge role in the decision making process, as he or
she has the authority to prevent “groupthink” from happening (3-9).
The last aspect of decision making that is addressed in this chapter are the
attributes that make an effective decision maker. These attributes are: knowledge,
initiative, advice seeking, selectivity, comprehensiveness, currency, flexibility, good
judgment, calculated risk-taking, and self-knowledge. Most of these are self-explanatory.
However, some of them need to be explained in greater detail. For instance, selectivity
means that an effective decision maker seeks only pertinent data, and avoids getting
bogged down by extraneous information. Currency implies considering current
conditions and taking advantage of current opportunities. Finally, self-knowledge means
knowing yourself and your strengths and weaknesses. (3-14-5). Two ways of judging
whether someone is a good decision maker are: whether they make competent and
confident decisions, and whether most of their decisions work out right (3-13).
The IESE School of Business’s six-step process to resolving unstructured
problems explains that the benefits of their process is its ability to bring together two
different types of people - those who see the world in terms of numbers, and those who
see the world in terms of words and emotions. Problem solving that uses only one of
10
these approaches is problematic because we may be missing key statistical or quantitative
analysis, or we may be missing key insight into why the problem exists, or how to solve
and implement the results of the problem. When both approaches are recognized and
utilized together, we have the highest chance of sustained success (4-5). Thus, we cannot
rely on one type of person to solve any given problem, and combining the strengths of
different kinds of people will help us achieve the greatest possible results.
Erika Wells, Visiting Assistant Professor of Psychology at Union College, gave
her perspective on the problem solving process in Psychology. She tried to give me an
explanation for why and how people make decisions when it comes to problem solving.
She believes that even when we think a problem is complex and unstructured, that it
actually is not. She explains that with enough work, any problem can be stated in more
simple and easy to understand terms.
Further, Professor Wells argues that most of the time, when we are solving a
complex problem, we reach an “Aha!” moment, or the moment of insight, during which
the solution suddenly becomes clear, which is supported by Topolinski and Reber. One of
the methods she suggests, when we do now know exactly what our problem is, is trial and
error. By trying simple analysis, even if we don’t know the exact problem we are trying
to solve, we can usually at least rule out the things that do not work or are not correct,
which brings us one step closer to solving the problem.
One of the important aspects of problem solving according to Professor Wells is
the size of the problem space. The larger the problem space, the longer it takes for us to
acknowledge all of our variables and alternative approaches and test them. Thus, we
11
should always take steps to try to reduce our problem space as much as we can, which
usually involves a subject matter expert.
Input from Economics
Lewis Davis, Associate Professor of Economics at Union College, suggests that the
easiest way to solve a large, complex, unstructured problem is to try to add structure to it.
He argued that oftentimes, what appears to lack structure actually does have structure, but
it is just hidden. Professor Davis’s focus is on Econometrics (applied statistics) and
Economic Development, which focuses on the policies that attempt to better the lives of
communities. In this field, many of the variables studied rely on another variable, which
relies on another variable, which leads to a never-ending chain of dependence. To avoid
this problem, he tries to find variables that are truly external and provide “exogenous
shocks to the system.” To do this, he notes that economists look for “natural
experiments” which provide us with limited knowledge that we can apply to a wider
question at least to gain a little insight, if nothing else.
Professor Davis admitted that most of the time, Economists and Economic
students do not actually systematize undefined/unstructured variables, but rather agree to
use metrics that have been developed by experts in other areas (i.e. using a political
scientist’s method for measuring how “democratic” a society is). He notes that the
hardest aspect of writing an Economics Senior Thesis is developing a problem that is
appropriate and researchable, and he often helps his students significantly with this step.
Another concern relating to problem solving in Economics is the fact that
creativity cannot be effectively “taught,” but it is often what separates the successful
12
students from those who are not successful. He questions whether this type of creativity
can be taught or if it is ingrained in us from a young age.
Input from Engineering
In addition, there is a lot to learn from the problem solving process in Engineering,
specifically Mechanical Engineering. Bradford Bruno, Associate Professor of Mechanical
Engineering at Union College, explained to me that the problem solving process in
engineering is similar to the process of solving problems in any field. The first step,
which is often overlooked, is defining the problem. The next step is generally
brainstorming and listing various alternative methods for solving the problem. Then,
testing is done to determine which alternative is most likely best. This step can be
difficult, however, because you must define your criteria for judging the “goodness” of
the model, which can be one of the harder steps of the process. For instance, in trying to
design a bridge, would the best bridge be the one that is cheapest to make and supports
the most weight? Or would it be made of the lightest materials and the longest lasting?
Clearly, there is a lot of freedom for the mechanical engineer to decide which criteria are
most important, and which ones can be compromised. The next steps are pretty standard:
building the model, testing the model, making changes to the model, and repeating the
process until an acceptable result has been achieved.
Professors of Mechanical Engineering at Union try to weave creative problem
solving into all of their courses, but it becomes more of a main focus during the senior
capstone project, where students must create and solve a problem all on their own (with
the help of an advisor). During this class, the engineering and mathematical tools are
emphasized along with the problem solving techniques mentioned above. This class is
13
meant to help prepare students to leave the college world where problems are laid out and
well defined, and enter the real world where this is usually not the case.
William Keat, Associate Professor of Mechanical Engineering at Union College,
focuses on engineering design, which is the process of studying a problem or task and
figuring out how to solve it, including the brainstorming, testing and evaluation stages.
The three main points in this process are design, building, and testing. However, within
the context of college-level courses in engineering, most problems students are asked to
solve are not large, complex, or unstructured.
In general, engineers follow a set of guidelines for solving problems. The six
main steps in this process are: recognizing the need to find a solution, defining the
problem, brainstorming possible solutions, evaluating these solutions in order to pick
what we think is best, developing a detailed solution, and testing and evaluating the
solution. For instance, Professor Keat is working with a senior thesis student to develop a
portable piece of equipment that helps senior citizens get out of chairs. The problem
needs to be solved because there is human need for it - there is no such device that we
know of that is affordable and easy to transport. The problem as they have defined it is
inventing a portable and affordable machine to help disabled people or senior citizens get
out of chairs more easily. Then, they had to come up with the requirements that they were
going to use to measure how well this piece of equipment works and how they will
measure these criteria.
Professor Keat says that brainstorming often entails breaking down the problem at
hand into the smallest possible pieces, so that it is manageable to work with. Then, if we
can individually solve each piece, we can attempt to put the pieces together to form a
14
solution of the whole problem. Next comes preliminary testing and evaluation, and then
testing of the final model to check if it passes our original criteria.
One unique difference between engineering and other fields is the emphasis on
teamwork. Most problems require a significant amount of background knowledge in a
variety of fields, which would be unreasonable for one person to know. So, combining
efforts allows us to come up with creative and helpful solutions. At the end of our
discussion, Professor Keat listed the two fundamental design principles for engineers, one
of which is the Information Axiom, which basically states to keep it simple. I think this
principle could also apply for solving large, complex, unstructured problems.
Professor Keat wrote “Engineering Design in General Education,” in which he
explains how he created a sophomore research seminar (SRS) that was targeted towards
engineers and non-engineers alike, which aimed at teaching students how to think as an
engineer. In this course, titled “Impossible Missions Design Teams,” Professor Keat
hoped to teach students with a variety of backgrounds about the fundamental principles
of engineering design. Professor Keat states, “The objective was for students to learn
basic research skills in the context of creative problem solving and engineering design.”
Professor Keat cites a book written by John Dewey, titled “How We Think,”
listing a five-step analysis of the thought process: felt difficulty, its location and
definition, suggestion of possible solutions, development of reasoning, and further
observation and experimentation leading to acceptance or rejection. He claims that these
steps can be directly compared to the steps an engineer takes when attempting to solve a
difficult or complex problem. The goal of this SRS was to use these steps, with the
15
“team” nature of these kinds of problems in mind, and develop a systematic approach that
everyone could use to solve the problems.
Input from Quality Management and Statistics
One field that commonly faces problem solving is quality management, where
researchers are constantly faced with creating solutions and analyzing them to determine
how to optimize their results. In “One Size Does Not Fit All,” Hoerl and Snee propose
attempting to break down large unstructured statistical problems into manageable
categories so that they can be analyzed and solved in a structured way. The first
characterization of a problem comes from two variables: is the solution known, and the
complexity of the problem. The answer to the first question is a simple “yes” or “no” (1).
However, simply knowing the solution to a problem does not make it easy to solve. In the
article, the example is given where the “solution” is that a particular person needs to lose
weight, but the “how,” i.e. ways to keep weight off, becomes the important factor, as
knowing a solution is meaningless if we cannot implement it. On the other hand, when
the solution is unknown, the way we approach the problem changes; instead of figuring
out how to implement the solution we already know, we must shift our attention to
figuring out what the solution even is (i.e. why is a person overweight, vs. how to
maintain weight loss).
We can also categorize a problem based on how complex it is, either low or high
based on the article. As discussed, low complexity problems involve isolated instances of
something going wrong, and they are usually easy to fix once the solution is known. High
complexity problems, on the other hand, do not necessarily have a particular isolated
problem, but rather the problem is on a much larger scale (2). When combined, these two
16
variables with two levels create a two by two matrix, which can be used to organize
problems that are seemingly very different into easy to understand categories. Once a
problem is categorized, it becomes much easier to start the problem solving process.
Hoerl and Snee study this process further in “Closing the Gap.” They state three
main problems in the field of statistics; that many times, statistical projects do not have a
high enough impact, that there is often a disconnect between statistical thinking and the
tools and methods used to implement such thinking, and that statisticians have the
potential to play a much more active role in whatever field they are in, instead of just
playing the passive role of a consultant (52). The term “statistical engineering” is
discussed, meaning in the literal sense the creative application of (statistical) knowledge.
For example, a chemist inventing a new substance is an example of science; while a
chemical engineer figuring out how to mass-produce it and use it in creative ways is the
goal of engineering. This idea can be applied to statistics just as easily. Thus, the role of a
statistical engineer is not to “advance fundamental laws of science,” but rather is to figure
out a way to use these laws of science in the most beneficial way to society as a whole.
The article suggests that the balance needed is actually 80% statistical engineering and
only 20% of statistics as a pure science (52-3).
The article suggests Lean Six Sigma (LSS) as an example of engineering: it did
not create any new tools as in science, but rather integrated existing tools in a more
efficient and beneficial way to generate enhanced results. LSS, as expressed in the article,
has not invented anything new, but rather provides a roadmap to address various types of
business problems (53). The article states that there are many people who are experts in
17
statistics and statistical tools, but lack the ability to apply these tools in a creative,
efficient, and meaningful way. This then, is the role of statistical engineering.
In “Tried and True,” Hoerl and Snee focus on the potential uses of statistical
engineering. The article opens with an example of using statistical engineering to
increase the efficiency of a transactional process. While many people can come up with
many correct methods for solving this problem, statistical engineering allows researchers
to streamline their results and create a methodical approach to problem solving (in this
case, it was designing an experiment). They realized that they could use methods they
already knew and used in other fields and apply them to their problem (in this case,
financial collections) (58).
Another example discussed is in the context of quality control. In this case, the
Product Quality Management System (PQM) was developed in the DuPont Company,
and over $30 million was gained in operating costs over two years. Many varying
statistical tools were woven into PQM, such as DoE, ANOVA, response surface methods,
and more. Again, no new tools were invented, but existing tools were linked and
sequenced in a novel way to achieve significant results (58). Together, statistical
engineers are able to create new and creative solutions utilizing existing methods to
confusing or unique problems.
Statistical engineering can be seen as a more basic example of simply
streamlining the way we approach any statistics problem. In the area of experimental
design, three steps – screening, characterization, and optimization – provide an overall
strategy of experimentation that goes beyond any one design. The screening phase
involves looking at large numbers of variables with the goal of narrowing them down and
18
figuring out which ones were most important to the problem at hand. During the
characterization phase, emphasis is put on measuring linear effects and interactions, and
finding good operating conditions. Lastly, in optimization, we use response surface
designs to make predictive models (59). However, it is important to note that no one step
can be used to solve a large, complex, and unstructured problem, but knowing the three
different stages of a problem may be a good place to start, and this methodology relates
to the generally sequential approach of solving problems.
In “Statistical Thinking: Improving Business Performance,” Hoerl and Snee focus
on adapting the Lean Six Sigma techniques to be more broad and applicable to more
types of problems. The overall strategy, called DMAIC, is an acronym for “Define,
Measure, Analyze, Improve, and Control.” An important aspect of Lean Six Sigma and
the DMAIC process is the fact that two entire phases are devoted to defining the problem
precisely, and quantifying it.
Picking a project and clearly defining the problem characterize the “define” stage.
In the case of large, complex, unstructured problems, this may prove to be the hardest
step, as once the problem is well defined it usually becomes easier to solve. The next
step, measure, is where we pick the output(s) that we believe to be important or need
improvement, and determine whether they are quantifiable. We gather as much data as
we can on these variables (130-32). Next comes the analyze step, which is where we use
our statistical tools to run various tests and create models to represent and explain our
data (132-34). The next step is the improvement step, which is when we propose a way or
ways to improve our problem based on the data we analyzed previously, and then test the
improvements to make sure they worked. This step and the one before it are generally
19
iterative, as one run of analysis and improvement usually do not solve a problem
completely, while they so usually point us in the right direction (135). Lastly, we control
our solution, so that we can sustain it once we are done working on the problem (135-36).
It is crucial to understand that this framework does not work on every kind of
problem, specifically where we don’t have a well-defined problem or we don’t have a
clear way of measuring our variables. This framework should not be used as strict
science, but rather as a suggested set of guidelines to help make the problem solving
process easier to attack.
A workshop presented by the National Science Foundation, titled “Discovery in
Complex or Massive Datasets: Common Statistical Themes,” focused on the application
of statistics in various fields, specifically in the case of large data sets. The areas
discussed were biology, finance, computer science, astrophysics, and more. Two key
points that were brought up in all cases were sparsity and machine learning. Sparsity
reflects the way we pare down information from extremely large data sets into a more
manageable size (5). In the case of the examples in this article, this usually meant
compressing data and information. Machine learning highlights the role of computers and
models in analyzing the data we have, which becomes a much more difficult and
important task when we have big data sets (6).
This workshop takes time to point out that statisticians work in all kinds of fields
and disciplines, and that even some of the examples given in this workshop come from
people who would not consider themselves “statisticians,” but use statistics in their
everyday life enough to understand its role and importance. The author(s) argues that
20
statistics can achieve the most success when it is applied to subjects outside of its origins
(7).
Statistics is meant to model situations that happen in everyday life, in all fields
such as the ones listed above. The contextual knowledge comes from all of these
disciplines, but the statistical tools used to solve these problems are all the same. Some
ways statisticians attack really large problems is by clustering, which is splitting the
problem up into smaller, more manageable groups. In this case, the way the data is split
up relies on the subject matter knowledge of the person running the experiment or
analysis, and the statistician’s role is to use this knowledge to make his job more
manageable.
One example, statistics used in biological data, shows that different kinds of data
are combined, resulting in large sets of often times confusing information. What makes
this type of problem possible to solve is the role of the scientist who has background
knowledge on the problem at hand (8-9). Without that, the statistician can only make
assumptions and guesses, and if he or she is not correct, then his outcome means nothing.
Statisticians also often need the help of a computational expert, and the teamwork among
these three groups of people is what makes large problems solvable.
The article concludes by addressing the fact that most fields deal with large and
complex data sets regularly, whereas they were not as major before the invention of
computers and the type of computing we do today. We also have to be aware of the effect
computers have on randomness in our experiments and data analysis. Some of the other
major questions and concerns brought up in the article are: how do we decide which
variables are important or not important? What role does traditional data analysis play in
21
analysis or large data sets? How can we systematically analyze the outputs of modern
statistical methods (20-1)?
Input from Science
Another field that applies problem-solving techniques in new and interesting ways is
Biology or Biochemistry. Brian Cohen, Lecturer of Biology and Biochemistry at Union
College, explains that in general, biologists have more data than they even know what to
do with (for example, DNA or brain activity data). Thus, he proposes trying to break the
problem down into the smallest possible pieces, if this is possible, i.e. “working from the
bottom up.” He suggests that this approach is more beneficial and efficient than trying to
take all of the available information and developing many pathways to the “solution,” and
then trying to test them all. Professor Cohen believes that one of the most important steps
is defining our target precisely, and then thinking about the ramifications of making the
changes we need.
Professor Cohen suggested that I look into systems biology, which is an emerging
approach to solving complex problems in biology, particularly biomedicine, which
focuses on interactions using a holistic approach. This approach is generally in opposition
to the scientific method, as it emphasizes the holistic approach as opposed to breaking the
problem down into manageable parts and solving them individually in a systematic way.
The student research that Professor Cohen advises is usually fairly well defined,
and the students generally each work on a little piece of the bigger problem that he is
trying to solve. Thus, the problem solving process in this case is defined but still requires
the creativity to come up with various ways of solving it, since the “answer” is not
always straightforward.
22
Another method for solving problems in any scientific field is the scientific
method, a process used in many analytical scientific disciplines to help acquire
knowledge or investigate phenomena. This method for problem solving was first seen in
Parmenides’s writings, where he uses deduction to solve problems. Later, the atomists
Leucippus and Democritus built upon this, and Aristotle formalized the scientific method.
The whole premise of the scientific method is to build upon previous advancements in
science by applying what we already know in order to learn something new. Importantly,
not all of the steps of the scientific method will always be used, and they don’t
necessarily need to be used in the order presented below (Edmund).
The scientific method consists of five steps, each of which requires significant
thought and creativity. The first step is formulating your question. As we have seen
numerous other times, this step generally seems trivial, but usually ends up being one of
the most important steps in the problem solving process. Sometimes, our questions are
straightforward (i.e. how can I bake a moist and delicious cake). Other times, the
questions are open-ended, and this is the case we are more interested in studying. This
stage also requires some research into similar studies or experiments done in the past.
The next step in the scientific method is creating a hypothesis. A hypothesis is a
conjecture about what we think will or will not happen. It is important that we set up our
hypotheses in a way such that we can draw meaningful conclusions from them. In
statistics, this means that our null hypothesis must be some statement of equality, which
is usually what we believe to be false. Thus, if we conclude that our null hypothesis fails,
we can accept our alternate hypothesis, which is what we believed would be true based
23
on our initial research and/or subject matter knowledge. Our hypotheses must also be
mutually exclusive and exhaustive of all the possible outcomes of our question (Harris).
Thirdly we make more predictions after we disprove or fail to disprove our
hypothesis. We want to continuously test our predictions until we have concluded
something worthwhile (or failed to do so…). Once we predict our outcome, we test it.
This is the part of the process where we see if what we predicted would be true is actually
what happens naturally. Our last step is analysis. Once we conduct our experiment or test,
we need to analyze the outcome and determine what our next steps should be. In general,
this process cannot be used only once to solve a problem, but rather will be needed in an
iterative way until we finally reach some sort of a conclusion. If we realize our
hypotheses were wrong, we should make new ones. If we notice flaws in our
experiments, we should design new ones. It is also important to note that whatever
conclusions we draw about our sample cannot be applied to anything outside of this
sample (Harris). We can use these results to help guide us but should not simple apply
our results to our target population if this is not the sample we originally tested. These
steps should be considered suggestions, which can help us in the problem solving
process, rather than strict rules that we must follow exactly.
Wolkenhauer et al. describe the role of systems biology in solving problems,
specifically in the field of medicine, in their article titled “The Road from Systems
Biology to Systems Medicine.” This article emphasizes the ways in which we can use
interdisciplinary techniques to solve biomedical problems, specifically in the context of
big data analytics where we are often trying to dig deeper into massive data sets. The
article claims that an interdisciplinary approach is necessary to “face the challenges of
24
integrating basic research and clinical practice” (502). Further, the article states:
“advances in measurement technologies can generate large-scale, multi level but also
heterogeneous datasets, requiring not only new computational platforms to manage data
but most importantly, requiring new ways of thinking, including the application and
development of methodologies from the mathematical sciences” (502). The importance
of systems biology is the multidisciplinary aspect, as it combines experimental work with
mathematical and computational analyses.
One of the motivations for studying systems biology was the time-sensitive nature
of the problems: cell processes happen quickly and standard statistical and bioinformatics
approaches do not work under these circumstances (503). Further, often times many
varying data sets are combined, which makes it more difficult to apply standard
techniques and necessary to apply new techniques that can handle heterogeneous data
sets.
This article also mentions the fact that mathematical modeling has not reached its
full potential within the field of biomedical research, but one of the positive aspects of
mathematical modeling is the fact that it can be done outside of the experiment being
conducted (504). There is a lot of emphasis placed on the relationship between the
“modeler” and the “experimenter,” as they must work together and utilize
interdisciplinary tools to solve these kinds of problems. Recently, there has been a focus
on “data-driven modeling,” and then “model driven experimentation,” which is a key
feature of big data analytics (504). Thus, not only is the actual experiment important, but
equally important is the step that comes before it, where data is analyzed without
performing actual experiments, to gather background knowledge on the problem at hand.
25
One of the common themes of the article is the need to have an interdisciplinary
approach to solving large and complex problems in biomedicine, as described: “many
scientific questions require a range of expertise from different fields. Therefore,
integration across disciplinary boundaries is crucial. The expertise for particular
technologies and experimental systems is rarely found in a single laboratory, institute, or
country, and this raises the need for standards and ontologies that support the sharing and
integration of data and models” (505).
Input from the Problem Solving Discipline
Graham Wilson, author of “Decision-Making and Problem Solving,” focuses on decision
making and problem solving in emergency situations. Thus, there is emphasis on
determining the problem (and how this is different from the symptoms of the problem),
and knowing how to react in a timely and appropriate manner. The book also has sections
that use personality traits to help determine what kind of decision maker you are and how
this affects the way we should proceed in these kinds of situations. The introduction
chapter highlights the fact that making poor decisions during the early stages of a crisis or
event can create more problems later on and make the decision-maker’s job more
difficult. The six units covered in this text are: introduction, the decision-making process,
identifying decision-making styles and attributes, ethical decision making and problem
solving, decision making in an emergency, and a course summary.
The second chapter focuses on the decision-making process. The objectives of
this chapter are to acknowledge the importance of making decisions before a crisis
happens as opposed to after, defining the steps of the decision-making process, and
distinguishing the causes from the symptoms of a real-life case study. The first thing
26
discussed is the difference between problem solving and decision-making, where
problem solving is “designed to analyze a situation systematically and generate,
implement, and evaluate solutions” (2-2). On the other hand, decision making is “a
mechanism for making choices at each step of the problem-solving process,” and it a
necessary part of the problem solving process (2-2). The five steps that are suggested to
solve problems in this book are: identifying the problem, exploring alternatives, selecting
an alternative, implementing the solution, and evaluating the solution. Wilson emphasizes
the importance of considering all of these steps, while it may not always be necessary to
write down or otherwise document them all.
Wilson breaks down each step into smaller pieces to make it more clear and easy
to understand. Wilson acknowledges, as we have seen many times before, that problem
identification is often the most important and most difficult step in the decision making
process. Every step we take after defining the problem will be based on this step. Wilson
defines a problem as “a situation of condition of people or the organization that will exist
and is considered undesirable” (2-6). Further, many people confuse the problem for its
solutions, by stating the problem in terms of its solution rather than the actual problem at
hand. Another important piece of this step is identifying the parameters of the problem,
which usually involves answering questions like, “what is happening?” “who is
involved?” or “what are the stakes?” (2-7). When considering complex and unstructured
problems, we also have to determine the best metric to define all of these parameters,
which can be surprisingly tricky.
The second step in the problem solving process is exploring alternatives. This step
actually consists of two parts: exploring alternatives, and evaluating alternatives (2-11).
27
Wilson suggests three methods for coming up with alternatives, which include
brainstorming, surveys, and discussion groups. Brainstorming consists of thinking out
loud either individually or in a group, while a survey is meant to poll a large number of
respondents. Discussion groups consist of those people who are directly involved in the
problem solving process and is similar to the brainstorming process but is more formal.
Once the alternatives are created, there must be some way to test them initially in order to
pick the “best” one to pursue. In order to evaluate alternatives, the manual suggests 6
approaches to consider: identifying the constraints, determining the appropriateness of
the solution, verifying the adequacy of the solution, evaluating the effectiveness,
evaluating the efficiency, and determining the side effects of the solution (2-13). One
other aspect of this part of the process, which hasn’t been mentioned in other literature, is
identifying contingencies, or trying to predict what might go wrong (2-12).
Next, we select an alternative. Whichever alternative we decided was the “best” in
the last step becomes the subject of our main attention during this step. There are many
factors that should be considered when selecting an alternative, such as political, safety,
financial, environmental, and ethical factors (2-15). Also, sometimes it is difficult to
implement this alternative once it is chosen, as there are often unforeseen repercussions
or consequences. This manual provides a worksheet that can help us determine the best
solution by listing the solution and its limiting factors, and considering combining
solutions if one does not stand out as a clear winner (2-16). After we pick the best
alternative, we must implement it, which generally consists of 5 steps: developing an
action plan, determining objectives, identifying needed resources, building a plan, and
finally implementing this plan. Developing the action plan entails creating a series of
28
steps to follow, as well as who will be responsible for each step. Then, we determine
objectives, which are “measurable targets that we use to monitor progress and establish
priorities” (2-17). We identify the resources we will need, considering cost, time, and any
special requirements. Then, we build our plan, which should state “who will do what by
when, where, and how,” (2-17). Finally, we put this plan into action. The manual includes
a checklist that can aid us in following through with the implementation and action plan
process.
The last step is evaluating the solution, which involves two parts: monitoring the
progress, and evaluating the results. To monitor the progress, we must ask whether the
situation has changed, are other resources required, and are there other alternative
solutions required (2-21)? This step is an ongoing process that does not end at the end of
our project, but must be continued to ensure our situation doesn’t change (2-21).
Chapter 4 focuses on the role of ethics in the decision making process,
specifically in emergency situations, and attempts to identify potential ethical issues that
may come up and ways which we can apply the problem-solving model to these ethical
issues. Ethics is defined as a “set of standards that guides our behavior, both as
individuals and as members of organizations; principles of right and wrong, such as being
honest, fair, and treating others with respect (4-2). It is important to note that ethics do
not imply legality. This manual offers a list of ethical “don’ts” that specifically apply to
emergency situations: don’t exceed your authority or make promises, and don’t use your
position to seek personal gain - it even goes as far as to make sure you not only act
ethically, but are aware of the appearance of being unethical as well (4-3). In addition,
some ethical “do’s” are listed: do place the law and ethical principles above personal
29
gain, do act impartially, so protect and conserve agency property, and do put forth honest
effort (4-4).
Next, the manual outlines four components of ethical decision-making:
commitment, consciousness, and competency. Commitment entails “demonstrating a
strong desire to act ethically and to do the right thing, especially when ethics impose
financial, social, or psychological costs” (4-6). Consciousness involves “seeing and
understanding the ethical implications of our behavior and applying our ethical values to
our daily lives.” Finally, competency can be broken down into three parts: evaluation
creativity, and prediction. Evaluation is the ability to “collect and evaluate relevant facts
and [to know] when to stop collecting facts,” creativity is the “capacity to develop
resourceful means of accomplishing goals in ways that avoid or minimize ethical
problems,” and prediction means the “ability to foresee the potential consequences of
conduct and assess the likelihood or risk that persons will be helped or harmed by an act”
(4-7).
The fifth unit focuses specifically on decision making in an emergency, which
changes the way we go about making decisions. Some impediments people face in these
situations are due to stress, such as time pressure, fatigue, lack of information, and
uncertainty. Often, people who must make decisions under stress experience conflict with
other key players, perceive selectively because of sensory overload, or experience
perception distortion and poor judgment (5-2). In addition, these people are usually less
tolerant of ambiguity and are susceptible to making premature decisions, have a
decreased ability to handle difficult tasks or work effectively, and experience a greater
tendency toward aggression and escape behaviors (5-2).
30
The last chapter summarizes the key concepts from the rest of the manual. It
defines problem solving versus decision-making, and outlines a five-step problem solving
model: identify the problem, explore alternatives, select an alternative, implement the
solution, and evaluate the solution (6-2). Next, it lists the factors that affect decision
making, which are political,, safety, financial, environmental, and ethical (6-3). The
manual also lists four decision-making styles, based on the MBTI test: sensing, intuition,
thinking, and feeling (6-4). The four ways to make a decision as outlined by Wilson are
individual, consultation, group, and delegation (6-5). The manual lists many attributes of
a good decision maker: knowledge, initiative, currency, flexibility, self-knowledge,
calculated risk-taking, and good judgment are just a few (6-6). Next, some ethical “do’s”
and “don’ts” are listed: don’t exceed your authority, or use your position to seek personal
gain, but do place the law above personal gain and act impartially, while putting forth an
honest effort (6-8). Finally, the three components of ethical decisions are commitment or
motivation, consciousness or awareness, and competency or skill (6-9).
In their article, titled “The Process of Solving Complex Problems,” Fischer et al.
discusses Complex Problem Solving (CPS) and how it applies to various fields. Five key
points of this process are highlighted: information generation, information reduction,
model building, dynamic decision-making, and evaluation. The article’s first sentence
does a good job of summing up the motivation behind this research: “In times of
increasing globalization and technological advances, many problems humans have to face
in everyday life are quite complex, involving multiple goals as well as many possible
actions that could be considered, each associated with several different and uncertain
consequences, in environments that may change dynamically and independent of the
31
problem solvers’ actions” (20). Thus, the aim of the article is to come up with a
systematic way of solving complex problems that can be applied in any field.
The article tries to more clearly define exactly what is meant by “complex
problem solving” by describing some of the important characteristics of such problems.
For instance, a problem is defined as complex when it has many elements or parts. A
problem exists, as described by the article, when we have goals but do not know how to
reach them (22). Thus, problem solving is trying to find the means of achieving our goals.
One of the most important aspects of problem solving in any field is that the
researcher should have knowledge on the tools needed to actually solve the problem (i.e.
applying equations, models, etc.), but also should have subject matter knowledge (or
should have an expert nearby).
The article continues, outlining the most important aspects of complex problem
solving (CPS), the first of which is human problem solving (23). The most important
aspects of this are creating an internal representation of the problem space. Additionally,
we must try to formulate a method for achieving our goal, given our internal
representation of it. This process is iterative and often requires us to go back to the
beginning if our methods do not work out. Frequently, the problem solver must define
parts of the problem him or herself, which is consistent with what we have to do to solve
unstructured problems.
Another aspect of CPS is expertise. Having expertise in a given field greatly
reduces the amount of things we must filter through to find the most important things. It
allows us to make educated decisions (even guesses sometimes) and helps guide our
technical analyses. In general, experts can solve problems faster than novices in their
32
given fields (25-8). Next, the article outlines decision-making strategies. The article
concluded that the important part of this discussion is not which one strategy is best, but
how to discern when to use which strategy depending on the circumstances. Problems
solvers base their decisions on the known solutions if they are available. If not, they focus
on solving the problem at hand given the circumstances, or trying to find new
information that would help them solve the problem. Information reduction is also
important in this process because it can simplify a problem that at first seems really
complex and large. The article acknowledges that a lot of research needs to be done in the
field of complex problem solving. However, the techniques listed above provide a good
start (37).
Similarly, TRIZ, also commonly referred to as the “theory of inventive problem
solving/TIPS,” is a problem solving technique, which was developed by Soviet inventor
Genrich Altshuller and his colleagues around 1946. The basis of this theory is that we can
use already established principles and insights from other fields that many not necessarily
be related to what we are studying, but if they are similar enough we can apply these
techniques to help us solve our new problems. Specifically, TRIZ is often used where we
have to somehow avoid an inherent conflict within our problem, such as creating a high-
power engine that is also light. These two qualifications seemingly contradict themselves
- the most high-power engine we could make is probably really large and heavy, while
the lightest engine we could make is not high-powered (Technical Innovation Center).
The idea behind TRIZ is to take our problem, find a general problem that has been
solved that can give us insight into our own problem, and apply the solution to this
general problem somehow to our specific problem. Technical contradictions are solved
33
using 39 elimination principles, while physical contradictions are solved using 4 basic
principles, which study super systems, subsystems, and separation of time and space. All
of this information was put together into a matrix, which attempts to systematically
approach the way we attack problems (Technical Innovation Center). Thus, we can use
TRIZ to solve large, complex unstructured problems because we can use previous results
from similar problems and apply them in new and creative ways.
III. Key Themes from Literature
I have found many recurring themes that appear throughout the literature for large,
complex, unstructured problems. I have organized them into two macro-themes – the first
contains six key phases that, regardless of the problem solving method used, a researcher
must pass through and consciously consider. I believe that these phases cannot be skipped
and must be taken into consideration regardless of the type of problem. The second
macro-theme contains the key considerations that a researcher must take into account
during all phases of the problem solving process. Thus, while there is no “best” or
“correct” method of problem solving, these key themes will always be encountered and
must always be taken into consideration.
Within each macro-theme, I have identified additional sub themes. In macro-
theme one, I suggest that there are six phases that a problem solver will necessarily
encounter (in order): identifying the motivation for solving the problem at hand, problem
definition, understanding the context, strategizing, brainstorming, testing, and evaluating
alternatives, and maintaining the solution. Within macro-theme two, I distinguish three
important things that all researchers much consider throughout the problem solving
34
process: gathering subject-matter knowledge, assembling a team of diverse perspectives,
and using diverse methods.
35
Figure 1: Outlining the six key phases of the problem solving process, and the three key considerations that should be applied during all six phases
36
I. While there is no “best” method that can be applied to every problem, there are
certain phases that appear in any problem solving process that cannot be skipped or
ignored.
1. Identifying the motivation for solving the problem at hand.
The first step of solving any problem, which cannot be skipped or ignored,
is understanding the need to solve the problem at hand. As Fischer et al.
explain, “in times of increasing globalization and technological advances,
many problems humans have to face in everyday life are quite complex,
involving multiple goals as well as many possible actions that could be
considered, each associated with several different and uncertain
consequences, in environments that may change dynamically and
independent of the problem solvers’ actions” (20). Clearly, whether we
realize it or not, we are faced with large, complex, and unstructured
problems in our daily lives. Whether it is in our professional lives, our
social lives, or our personal lives, these problems arise constantly.
However, there are some problems that need to be solved
immediately, and some that are not a high priority for us to solve. Further,
we must identify why this is a problem worth or not worth solving, by
considering the consequences of solving or not solving any given problem.
For instance, we might not have enough time to solve a given problem,
thus it is not worth solving. Or, we might only be able to slightly make a
situation better, like in the case of quality management for example. If we
can only reduce mistakes in a certain piece of machinery by such a
37
negligible amount that it wouldn’t make a noticeable difference, perhaps it
is not worth spending the time, money, and manpower to fix the problem.
In other words, given the many challenges of modern life, we must
“choose our battles” carefully.
2. Problem Definition
We cannot attempt to solve a problem that has yet to be defined. As in the
US Healthcare system example, we cannot begin to try to come up with a
“better” healthcare model if we do not know exactly what problem we are
trying to solve.
The first aspect of this stage is defining the actual problem
statement. As Hoerl and Snee state in “One Size Does Not Fit All,” this
step is often the hardest step especially in the case of large, complex, and
unstructured problems. In general, most people are comfortable solving
problems that are well defined, such as textbook problems in a college
course, because once the problem is defined they can easily identify the
appropriate methods to use. However, in the case of unstructured
problems, we have to define the problem before we can continue the
problem solving process. Wilson describes how often people fail to clearly
define the problem, and sometimes confuse the problem for its solutions,
which we should always avoid (2-6).
Another important aspect of this phase is defining the goals of our
research and problem solving process. In the case of the US Healthcare
system, do we want to cover the largest number of Americans? Increase
38
life expectancy? Or achieve some other goal? We must consider the
multiple stakeholders and ensure our problem is defined in terms of
everybody’s interests.
Similarly, we must make sure that we are determining the
necessary resources we will need to solve the problem, so that we can
define the problem and our goals keeping this in mind. As explained by
the IESE, if we define the problem in such a way that it becomes
impossible to solve, then our efforts will be futile.
3. Understanding the context
One important aspect of a large, complex, unstructured problem is that its
solution often requires deep understanding of its context. Big problems
have defied solution for a reason; to address them we need to understand
how they got there, and what sustains them. For instance, in the case of the
US Healthcare system, it is important to consider the economic status of
the US, the political situation, and the social condition of the country, as
well as the history of healthcare in the US. This could significantly
enhance the way we approach the problem and the solutions we come up
with.
In addition, we must know what has already been attempted or
solved in relation to our problem. For instance, using the methodology of
TRIZ, we should look for any information or advancements that relates to
our project, even if it does not directly answer the question we have posed.
Using TRIZ, we could apply similar techniques to a portion of our project
39
that is similar to a problem that has already been solved. In general, we
can apply strategies from various fields and problems that have been
solved in the past that relate to our field, regardless of whether or not it
directly answers our question. Any advancements we make simply by
researching what has been done already will help us achieve our results.
4. Strategizing
First, we must acknowledge the need for a strategy. Fischer et al. point out
that in order to create a strategy, we must first create the problem space
that we will be working within, which is defined as, “a set of possible
states of the problem, given the initial state, the applicable operators, and
certain goal states” (23). In small, structured, or simple problems, i.e.
textbook examples, this step is often assumed and not emphasized.
However, especially for complex problems, this step is crucial, as we
cannot continue with the problem solving process unless we
systematically classify the problem. Frequently, we realize that the current
format of the problem and the problem space do not promote our creativity
and ability to solve the problem, so we must go back to the problem
definition and restructure the problem space.
It is in this way that we utilize a sequential approach to problem
solving. Once we lay out the problem space, we can try to break up the
problem into smaller, sequential “steps” that make resolving the problem
easier. For example, in “Tried and True,” Hoerl and Snee argue that we
can use three steps in a sequential approach to solve experimental design
40
problems: screening, characterization, and optimization. During the
screening phase, our goal is to identify the most important variables, so
that in the characterization phase we can measure linear effects and
interactions. Finally, in the optimization phase, we make predictive
models from the data analysis in the first two steps. Approaching a
problem this way can also be helpful on a larger scale: first we want to
narrow down the factors we are studying, and then measure the effects
they have on the given question. Finally, we want to create some sort of
final model or solution based on this process. Importantly, we cannot just
use one step of this approach to solve a large, complex, unstructured
problem, and they must be used in this arrangement to achieve the desired
results.
5. Brainstorming, testing, and evaluating alternatives
One crucial step in the problem solving process is defining what our
criteria will be for determining how “good” our solution will be.
According to the IESE’s six-step problem solving process, “the key to
successfully solving a problem is choosing which if all of the possibly
important issues will really be the most critical in making a decision” (2).
In our example of the US Healthcare system, this means determining
which criteria will be used to judge the “goodness” of our healthcare
model – will we be most interested in reducing citizen’s out-of-pocket
payments, decreasing overall fatalities, least amount of government
spending, or some combination of these and many other criteria? We must
41
carefully consider this step before we move on, as it impacts the way we
judge the “goodness” of our solutions, which ultimately is the end result of
our problem solving process.
The next logical step in this process is identifying alternatives. The
IESE states that this step is more difficult in a large, complex, unstructured
problem, but it is critical nonetheless (3). This step is rarely mentioned in
the quality management and statistics literature, but is mentioned in the
engineering, biomedical, and psychological literature. I also think it is
important to note, as the IESE article explains, researchers must choose
the criteria before selecting their alternatives, as bias is introduced when
this process works in reverse (3). We want to avoid bias at every stage of
the problem solving process, and it is a quality found in human nature that
we will bias our criteria if we already have the alternatives laid out.
Wilson suggests various methods we can use to produce alternatives –
brainstorming, surveying, and utilizing discussion groups – each of which
having its benefits and unfavorable effects (2-13).
Another important aspect of this phase is the role of creativity. We
cannot teach people how to be creative, yet it is crucial that the people
who are working on the problem at hand are creative enough to think
outside of the box when it comes to alternative development. If we can
only come up with solutions that have been used in the past or that are
similar to things we have seen or done before, we are severely restricting
our ability to solve a large, complex, unstructured problem.
42
Once we pick our alternatives, we must test them to determine
which one or ones are the most likely to be successful, by using our
criteria. This is where researchers have the most freedom to use whatever
tools and methods are appropriate for their particular project, but this step
can also be confusing for those who are studying large, complex
unstructured problems because the correct method might not be apparent.
We continue to narrow down our alternatives by analyzing them and
comparing them to the criteria we selected until we are happy with our
results and our final solution.
Wilson suggests various ways of determining which alternate
solution is “best,” in addition to measuring the criteria we set forth. The
six things he recommends considering are: identifying the constraints,
determining the appropriateness of the solution, verifying the adequacy of
the solution, evaluating the effectiveness, evaluating the efficiency, and
determining the side effects of the solution (2-13). In addition to our
criteria, these guidelines can be a good way for us to determine which
alternative is the best and are a good place to start especially in the case of
large, complex, unstructured problems.
One last step we should always consider during this phase is
whether or not it would be feasible to actually implement our solution.
Wilson constructs a five-step method for implementing a solution once we
have picked the best alternative: developing an action plan, determining
objectives, identifying needed resources, building a plan, and finally
43
implementing this plan (2-17). We should always consider these steps, in
order, to ensure that since we have gone through the trouble of solving our
problem, we can successfully implement it.
6. Maintaining the solution
I believe that ensuring we can maintain whatever solution we decide to
implement is just as important as being able to come up with the solution
in the first place. Hoerl and Snee, Wilson, and the IESE all emphasize the
importance of sustaining our results, stating that we often need to come
back to our problem and reevaluate the solution to ensure it is still
successful after some time has passed and the problem has been closed.
Similar to the problem solving process as a whole, sustaining the
successful result is usually iterative and requires constant maintenance, but
is necessary to guarantee that all of the hard work it took to create the
solution was not in vain.
II. As we saw in Figure 1, the following key considerations should be kept in mind
during all of the stages of the problem solving process.
1. Gathering subject matter knowledge
There are many times during the problem solving process in which it
would be beneficial to have subject matter knowledge. For instance, when
defining the problem and the criteria, it would be helpful to have a subject
matter expert there to advise the researcher on how to appropriately define
the problem. For example, in the US Healthcare problem, it would be
appropriate to gather doctors, insurance companies, and government
44
officials to help the researcher understand exactly how the current
healthcare system works, so that he or she can correctly define the
problem. The NSF workshop mirrors this idea, highlighting the role of the
subject matter expert. Without the expert’s advise, a statistician cannot
make informed decisions. Specifically in the case of large, complex,
unstructured problems, data is often combined from multiple sources and
it is up to the subject matter expert to make sense of it all so the statistician
or researcher can perform his or her analysis.
Fischer et al. highlight another important aspect of consulting
experts during the problem solving process: it allows us to filter through
large amounts of information more quickly with less wasted time (26).
They explain that problems become easier to solve when we have a wealth
of subject matter knowledge, and something that seems complex may
actually be simple when we consult an expert. Further, consulting experts
allows us to make educated guesses instead of random guesses, and can
usually help guide our analysis. Before we even analyze any data, a
subject matter expert can help us make decisions that will directly impact
our analysis. Such expertise is critical at each subsequent phase of the
problem solving process.
2. Assembling a team of diverse perspectives
A significant amount of time should be spent constructing a team of
diverse individuals who will all bring something meaningful to the
problem solving process. Diversity can refer to different personalities as
45
well as disciplines and levels of experience. As the IESE describes, there
are two types of people – those who see the world in terms of numbers and
those who see the world in terms of emotions – and both types of people
should be involved in the problem solving process together (4-5). Further,
consideration should be given towards who should be making the
decisions and solving the problem.
Wilson presents four categories of decision-making: individual,
consultation, group, and delegation (3-6). Based on the context of the
problem, an appropriate method should be used, and this step should not
be overlooked. We should take extra care to ensure we avoid the
“groupthink,” where the desire to conform to the group drives people to
make biased decisions. Groupthink can be extremely detrimental to the
problem solving process, as creative solutions can be stifled by leaders
who are not open to new ideas. Leaders must be careful to avoid only
working with people who think like them – something that unfortunately
happens all too often without leaders and bosses even being aware of it.
By intentionally picking individuals who bring different perspectives to
the process, problem solvers can increase the chances of coming up with a
successful result.
3. Using diverse methods
I believe that one of the most important themes in the literature has been
that regardless of the problem solving technique, researchers must be
willing to be creative and use methods and techniques that are not familiar
46
to them. Sticking to what is comfortable and familiar will not foster
“good” problem solving. Rather, it prevents us from solving problems,
specifically large, complex, unstructured problems.
In addition, problem solvers must understand that only rarely will
one single method or technique solve any problem, regardless of the size,
complexity, or structure or lack thereof. So, it is important to keep a large
toolbox of methods and techniques available to use whenever necessary,
as well as the ability to be flexible and creative. A combination of
technique and creativity is important in problem solving, and no one
method or technique, when used alone, is universally “best,” which is
controversial to much of the literature that attempts to compare various
methods and pick the “best” one.
IV. Recommendations for Problem Solvers
After compiling as many of the problem solving techniques from various disciplines, I
have created my own list of recommendations that I would suggest to people trying to
solve large, complex, unstructured problems. This list is not exhaustive, and is meant to
be supplemental and additive to the key themes I outlined in the previous section.
My first recommendation for problem solvers is also the most important in my
opinion – it is crucial to understand that while I have provided a list of phases and
techniques that should be applied, this methodology is not meant in any way to be the
“correct” or “only” way of solving a problem. It is meant, however, to be a good starting
point for people who do not know what the steps are for solving a large, complex,
47
unstructured problem. One issue too often seen in the problem solving literature is that it
spends too much time defending different methods and techniques as the “best” ones.
However, I believe that this is a waste of time, as no single method can solve every
problem. This brings back the role of statistical engineering, where researchers and
problem solvers should focus on utilizing tools they have in inventive and creative ways,
rather than attempting to figure out which tool or tools are the “best” and work for every
problem.
In addition, I recommend that those responsible for addressing big problems
spend a significant amount of time gathering the right group of people to assist them
during the problem solving process. This step is often overlooked because natural
“teams” can form through existing relationships and problem solving teams in the past.
However, I advise that each problem be thought about individually, so that various
personalities and experts can be assembled to form a cohesive and productive problem
solving team. As I mentioned earlier, the phenomena of “groupthink” should be avoided
at all costs, so that the same mistakes are not made consistently and new approaches can
be utilized. No one person, regardless of their experience or expertise, can solve every
problem, and it is always beneficial to have diverse people on the problem solving team.
This is especially true for large, complex, unstructured problems, where there is often a
diverse amount of information necessary to make informed decisions, and it would be
inefficient to try to learn all of the required background information. In particular, we
need to include those with subject matter knowledge, those with data analysis skills, and
those who can creatively apply the tools and techniques they know.
48
Another important recommendation is being comfortable with using tools and
techniques that you are not familiar with. It is easy to rely on the tools that you know and
have worked in the past, but when dealing with large, complex, unstructured problems, it
is important to look outside of your “toolbox” when thinking of ways to solve the
problem at hand. This is another reason why it is so important to gather a diverse team,
each of whom brings a different viewpoint and skill set to the table. Being able to adapt
and adjust your skill set and use unfamiliar tools will make you more likely to succeed.
Having a diverse toolkit will enable problem solvers to select the method based on the
unique nature of the problem, as opposed to tailoring the problem to fit within a certain
framework so they can use the tools they are familiar with. In other words, it will help
problem solvers avoid viewing every problem as a nail because the only tool they have is
a hammer. Often, this will require researchers to read diverse literature outside of their
own fields.
I suggest reading the manual by Graham Wilson, referenced several times in this
thesis (“Problem Solving and Decision Making”). While this manual is meant to help
students teach themselves the methods of problem solving, it focuses on emergency
situations. It may not seem relevant to the process of solving large, complex, unstructured
problems, but it actually does a great job of breaking down this process into its most
basic definitions (i.e. what do we define as a “problem”?), and then providing various
steps and methods for working our way through the problem solving process. If you are
looking for a comprehensive guide to the basics of problem solving, especially in an
emergency situation, this is a great place to start, and it provides worksheets that a
49
problem solver could easily fill out with the necessary information to be even simpler to
use and understand.
Another recommendation is to give yourself more time than you think you might
need to solve the problem. Problem solvers should take the time to carefully define the
problem, as this is the “make or break” aspect of the problem solving process.
Additionally, researchers and problem solvers could benefit from having time to create
alternatives and analyze them, and then have more time to contemplate these solutions
and reflect on what they have accomplished. As mentioned in the article on the “Aha
moment,” you cannot force or expedite these moments of clarity; often just having more
time to think about the problem will prove beneficial (Topolinski and Reber). Going
along with this, do not get frustrated if your first few solutions do not work out. If
problem solvers could solve any problem in a few simple steps, large, complex,
unstructured problems would not even exist. So, do not let a temporary failure or minor
setback cause you give up on the problem solving process as a whole.
One final recommendation I have for problem solvers is to make sure that once
you have a solution, you can successfully implement and sustain the results. Our efforts
would be futile if we spent our resources on solving a problem, and could not implement
and sustain the results. This implies that throughout the problem solving process,
including deciding which problems to work on, we consciously consider the long term
view of sustainability.
V. Conclusion
In this thesis, I researched available literature and interviewed experts about the problem
solving process in various disciplines, such as psychology, economics, biology,
50
engineering, quality management and statistics, and the problem solving discipline. I
organized the wealth of information into two macro-themes: six key phases that a
problem solver will necessarily encounter during the problem solving process, and key
considerations that must be consciously applied during all stages of the problem solving
process.
The six phases of the problem solving that I have identified are: identifying the
motivation for solving the problem at hand, problem definition, understanding the
context, strategizing, brainstorming, testing, and evaluating alternatives, and maintaining
the solution. Within each of these phases there are other important things to consider.
These steps are not meant to be definitive and I do not claim that this is the “best” process
of problem solving. Rather, I believe that these steps are necessary regardless of the kind
of problem, and that problem solvers should seriously reflect on all of them.
The three key considerations that I believe should be employed during all stages
of the problem solving process are: gathering subject matter knowledge, assembling a
team of diverse perspectives, and using diverse methods. These themes should not be
overlooked, and researchers should make a conscious effort to utilize them during every
stage of the problem solving process.
In addition to the key themes above, I have also come up with a list of
recommendations to problem solvers, especially when attempting to solve large,
complex, unstructured problems. The first one, which I believe to be the most important,
is to take everything I have gathered and organized lightly. I do not intend this to be a
comprehensive method for solving large, complex, unstructured problems. Rather, I
suggest this paper as a good starting place for those who are unsure of how to begin
51
solving a large, complex, unstructured problem. While much of the literature on problem
solving in various disciplines spends time debating which methods and techniques are
best, I argue that there is no one best method. However, I believe that the phases I have
outlined will be present regardless of the type of problem being solved.
In addition, problem solvers must be flexible in all aspects of the problem solving
process. Whether this means working with people who have a variety of subject matter
backgrounds, people with different personalities, or being able to utilize tools and
techniques that they are not comfortable with or used to using, researchers and problem
solvers must be able to step out of the box if they want to solve a large, complex,
unstructured problem.
Finally, researchers should ensure that they have the necessary resources
available. These resources come in many forms, for instance time, money, and
manpower. Problem solvers should always give themselves more time than they
anticipate needing to solve a big problem, to allow them to take the problem definition
stage seriously, as well as all of the subsequent stages. Further, researchers should make
sure that they have access to all of the necessary resources they will need to not only
solve the problem, but also implement and sustain their results. If results cannot be
implemented or sustained, then the problem solving process was ineffective.
While there are many problem solving techniques presented within various
disciplines, my goal was to condense these methods to identify important themes that
could help people solve big problems across disciplines. By using the guidelines and
consciously considering the key themes I have proposed, I believe we can successfully
solve large, complex, unstructured problems.
52
Works Cited
Bruno, Brad. Personal interview. 23 Jan. 2014.
Cohen, Brian. Personal interview. 12 Feb. 2014
Davis, Lewis. Personal interview. 27 Jan. 2014.
Edmund, Norman W. "The Scientific Method Today." The Scientific Method Today.
Web. 16 Mar. 2014.
Fischer, Andreas, Samuel Greiff, and Joachim Funke. "The Process of Solving Complex
Problems." The Journal of Problem Solving 4.1 (2012): n. pag. Print.
Harris, William. "How the Scientific Method Works." HowStuffWorks.
HowStuffWorks.com, 14 Jan. 2008. Web. 16 Mar. 2014.
Hoerl, Roger W., and Ronald D. Snee. "Closing the Gap." Quality Progress (2010): 52-
53. Web.
Hoerl, Roger W., and Ronald D. Snee. "One Size Does Not Fit All." Quality Progress
46.5 (2013): 48. Web.
Hoerl, Roger W., and Ronald D. Snee. "Tried and True." Quality Progress (2010): 58-60.
Web.
Hoerl, Roger, and Ronald Snee. "Statistical Engineering: Tactics to Deploy Statistical
Thinking." Statistical Thinking: Improving Business Performance. 2nd ed.
Hoboken, NJ: Wiley & Sons, 2012. 93-138. Print.
Keat, Bill. Engineering Design in General Education. Symposium on Engineering and
Liberal Education. Web.
Keat, Bill. Personal interview. 30 Jan. 2014.
53
Meng, Xiao-Li, and Joseph Blitzstein. Nano-Project Qualifying Exam Process: An
Intensified Dialogue Between Students and Faculty. Department of Statistics,
Harvard University. Web.
National Science Foundation. Discovery in Complex or Massive Datasets: Common
Statistical Themes. Proc. of Mathematical Association of America (MAA),
Washington, DC. 2007. Print.
Rosenberg, Mike. "IESE’s Six Step Process for Resolving Unstructured Problems." IESE
Business School. Web.
"Technical Innovation Center, Inc." Technical Innovation Center, Inc., Oct. 2012. Web.
16 Feb. 2014.
Topolinski, S., and R. Reber. "Gaining Insight Into the "Aha" Experience." Current
Directions in Psychological Science 19.6 (2010): 402-05. Print.
Wells, Erika. Personal interview. 29 Jan. 2014.
Wilson, Graham. Decision Making and Problem Solving: Student Manual. 2002. Print.
Emergency Management Institute.
Wolkenhauer, Olaf, Charles Auffray, Robert Jaster, Gustav Steinhoff, and Olaf
Dammann. "The Road from Systems Biology to Systems Medicine." Pediatric
Research 73.4-2 (2013): 502-07. Print.
54