White Paper - · PDF fileUse Metrics to Reduce Time-to-market and ... Bob Buxton —...

Use Metrics to Reduce Time-to-market and Accelerate Time-to-profit*

Manage metrics in product development to ensure profitable growth and avoid metrics maladies

By: Bob Buxton — Product Management and Marketing Professional

Jose Campos — Founder, Rapidinnovation

David Worsley — Operations Effectiveness Consultant

White Paper

*This article is the result of interviews with managers and product development professionals from 10 different technology companies. Plus, the combined product development experience of the authors

Metrics to Reduce Time-to-Market & Accelerate Time-to-Profit

White Paper 2

White Paper

ContaCt us: [email protected]

The Research Project

The need to resolve the Time-to-Profit secrets, prompted the start of a research project. We conducted research on product development and sought to bring clarity to the role of metrics and their impact on development speed and more importantly on profitability. We asked executives and managers in marketing and R&D at 10 technology companies to participate in a set of interviews. In addition, we performed a literature search to understand the current state of product-development metrics.

Introduction

This article summarizes the findings from a survey to indentify the obstacles that keep development projects from gaining velocity without compromising value creation and the imperative of profitable growth, and the role that metrics play in the overall system.

Technology companies are driven by “time-to-market” (TTM) and for a good reason, first-to-market provides a competitive advantage — albeit temporary. However, TTM is a “process metric” which measures the time elapsed during a new

product development program (NPD) and says little about the ability of the new product to generate the targeted margin and volumes to ensure profitable growth. In contrast, “time-to-profit” (TTP) measures the time to achieve the forecasted profits of a new product.

To avoid too much focus on a single process metric, which may misdirect the development team, a companion metric such as “time-to-profit” should be added in order to balance resources and create focus on the ultimate goal: profitability, and this brings us to one of the secrets of the proper use of metrics; always balance a metric to avoid unintended results and provide the right balance to all stakeholders.

Stating the obvious

Metrics are solely a means to an end. Further, every metric must have clarity of its intended use and the expectation that measurable action must be taken. We found that some organizations do not take the time to explain the metric to all stakeholders on the assumption that clarity is obvious to all, which is not necessarily so. Also, there is the need to communicate to all, the expectation that without corrective action, the metric is utterly useless. We also found that frequently, there is no “metric owner”, the one person who is responsible and accountable that the metric is used properly, consistently and everyone is aware and ready to take action.

We also prioritize good practices that can help improve the productivity of product development. Finally, we offer a set of recommendations for action that should enable organizations to improve their product development models. All of this is seen through the prism of metrics.

Project Findings Obstacles and Barriers

Our findings indicate that there are four areas that are most likely to create obstacles to speed (time-to-market) — and thus worthy of measurement and corrective action. The top four categories are listed below in the order of the time at which they impact the NPD process.

1. Lack of clear strategy that is understood by all stakehold-ers

Table of Contents

Introduction 2

Specific Research Findings 3

Good Practices from the Research Study 5

Conclusions 7

Stage-gate is dead — Not! 3

Appendix A: Product Development Metrics 8

Appendix B: Metrics Maladies 11

Appendix C: The System of Metrics 14

Appendix D: One Example 15

mailto:info%40rapidinnovation.com?subject=


White Paper 3

White Paper


2. Lack of prioritization issues impact the period prior to the commencement of product development.

3. Execution issues such as a broken NPD process or lack of project management

4. Distractions — particularly of the engineering team — impact the active development program

Specific Research Findings

1. Lack of clear strategy

Strategic Planning Process issues include:

● Lack of a planning process with no formality and no metrics

● Planning process that does not align with the business model for a particular industry

● Strategic Planning which was not clearly communicated to all stakeholders in a way that was relevant to them

Some roadmaps were a list of projects without linkage to timed technology investments. In other cases the portfolio of planned product developments showed no connection to revenue or profit growth and were linked solely to replacement of current products.

Lengthy decision-making slows NPD execution. This includes, not only speed of decision making, but also the ability to rapidly communicate decisions with absolute clarity to all stakeholders.

Concerning Product Definition, the bigger issues included a lack of clarity and potential moving targets. These lead to spending excessive time during accrual development to clarify market requirements, fighting feature creep or the realization that the original product definition was woefully lacking. These issues lead to lengthened product development programs, the potential to miss market windows, unsuccessful products or a combination of all three.

click here for a list of metrics maladies, Appendix B

Critical Questions1

● What metrics are in place and reviewed regularly to assess the quality of your planning process?

1 Critical Questions are included by the authors to help the reader initiate the proper dialog within their organizations such that this article leads to visible improvement

Stage-gate is dead — Not!

Due to the emergence of alternative programming methods, the relentless pressure to reach time-to-market and emphasis on overall speed, a trend has emerged to see any “stage-gate” approach as out of date and irrelevant — a metric no longer useful

Consider that there is an unavoidable reality: the calendar. Regardless of the methodology or process you use to develop your products, in a competitive situation, the calendar is always there and everything your team does is always measured against time. That is, pretending that time is not a critical element of your development is a luxury that perhaps a few in pure science research can afford, if you are developing products it is inevitable that the calendar will always be there and your executives will use it to drive the company.

Your company is in the business of creating a profit and every activity is measured against this reality. Moving your development along without assessing the progress towards a profit is not acceptable. Your ability to create value for your customers in a timely fashion is an unavoidable equation.

While the phrase “stage-gate” may be out of fashion, the reality that you must measure progress against time and profits is non-negotiable. A loose treatment of the calendar and lack of focus on value creation are imperatives that cannot be avoided.

0

5

10

15

20

25

30

35

Distractions NPD Execution Strategic planning

Prioritization Schedule Technology Other

Num

ber o

f Res

pons

es



White Paper 4

White Paper


● What do you measure that tells you that your Product Roadmap is accurate, up to date and understood by all stakeholders?

● What do you measure to ensure that your customer requirements are accurate, clear and well-prioritized

2. Prioritization issues:

Difficulty in appropriately prioritizing development programs and to clearly communicate it to all stakeholders constitutes the second main category of obstacles to NPD velocity. Factors that contribute to faulty prioritization are:

Competition across divisions or product groups may be driven by competing objectives and metrics, where various groups fight for resources and projects, eventually starving the company of resources.

“Squeaky Wheel” syndrome jeopardizes carefully planned product development roadmaps because what is perceived as a pressing customer issue consume resources regardless of impact on long-term projects.

Lack of foresight means that additional programs are inserted into the roadmap, causing other programs to be temporarily halted or significantly deprived of resources.

Strategy and prioritization issues reduce to one thing: lack of a roadmap and portfolio management process, which can achieve two major benefits. This process can be used to gain consensus across the organization on the project prioritization and be used as a tool to re-prioritize when a new opportunity arises such as a when a project fails a major milestone, or a customer action or new competitive issue emerges.

Critical Questions

● Do you measure the number of changes to the Product Roadmap and later analyze the results to determine justified and non-justified changes?

● Do you have a definition of “defect” for your Product

Roadmap — and then measure the number of defects per year to improve quality?

● Do you measure how well all stakeholders understand the Product Roadmap to ensure clarity and collaboration?

● 3. Execution issues:

Range from the overall effectiveness of the management (leadership) process to the availability of “real” resources, leading to the specific issues listed. A lack of discipline can include not adhering to the stage-gate process, a failure to track progress, or a late escalation of problems. Unclear accountability includes such things as not holding relevant team members accountable for meeting deadlines, incomplete or missing deliverables or sales estimates. Cultural issues range from insufficient or unclear cross-function communication to lack of cross-functional collaboration. All the companies interviewed had some form of NPD process; however, some said that gaps in their process led to poor synchronization across functions, and others lacked a process for identifying risks and mitigating them. Many companies suffered from a lack of resources, in many cases trying to achieve more results with fewer resources. When accompanied by multi-tasking of engineers, this led to inefficiencies due to the “switch-over cost” between projects (Click here for an article on engineering productivity and the negative impact of “switch-over cost”.)

Critical Questions

● Do you regularly measure the number of defects in your NPD? Then take decisive action to implement corrective action? Click here for additional information on NPD defects Appendix A: NPD Metrics

● How do you know that your development has the right “pace” to achieve time-to-market and also time-to-profit?

● Do you have a metric that tells you how well the interaction between the head office and the “distant outposts” is working? Many companies outsource parts or

The MRD (Market Requirements Document) and Agile

Many companies are embracing AGILE as a product development framework due to its advantages on speed and response to changes.

We found that some companies are returning to using MRD’s because the Backlog is not sufficient to define all the tangible and intangible attributes of a new product. Further, the Backlog — at times — becomes too tactical; thus, losing sight of the ultimate business objectives that the customers are striving to achieve.

Agile is a good methodology and beneficial to many organization, but ultimately, the end-product must provide a full solution to the tangible, intangible requirements as well as those that are un-stated along with emerging needs. An MRD seems to be better equipped to handle the load.

Continue


mailto:info%40rapidinnovation.com?subject=Send%20me%20the%20article%20on%20Engineering%20productivity


White Paper 5

White Paper


all of the development to third parties or to branch offices in other countries — a difficult relationship to manage given the differences in time, geography and culture.

● Do you measure the productivity of your R&D organization and then take decisive corrective action

4. Distractions

This was the category with the largest number of responses across the companies, and therefore we looked at in more detail.

Dealing with Distractions required caution; some, such as helping manufacturing bring up a production line for the first time or attending a technical conference, are essential parts of the role of engineering. However, others can be unjustified.

Unjustified Distractions

Some distractions were self-inflicted. For example, quality issues can arise from an inadequate design resulting from rushed R&D. Others such as Sustaining can be mitigated by choosing an alternative solution to redesign when facing component obsolescence; perhaps a “last time buy” might be a more cost-effective solution when the opportunity cost of diverting engineering effort is taken into account.

Others were unjustified, for example sending a Senior Design Engineer to address a customer issue that a technician could have handled, or assigning an engineer to a sales call when it was too late to make a difference.

Critical Questions

● Do you measure the “up-time” of your R&D organization? Unjustified distractions can take up to 50% of the available time if not controlled. The average “up-time” for R&D is approximately 60 percent (40% down time)

Good Practices from the Research Study

Let’s look at the top five good practices that were identifiedacross the 10 participating companies.

#1 Measure the quality of the MRD

The Market Requirements Document (MRD)2 describes the prioritized customer requirements for satisfying customer needs. It should not define what could be done with technologies or how the requirement will be satisfied, as these may involve research and development, unknown timeliness and increased budget spending. These factors are managed by the project. The project output is a risk-reduced design that performs to the MRD and subsequently produces an innovative new product that is differentiated from competition. While the ultimate measure of an MRD quality can only be taken from the market response when the new product is introduced. It can be measured internally, for example, do engineers understand with much clarity the MRD? Do engineers understand the priority of the requirements? If the MRD is insufficient, the product will not sell to expectations, may need redesign or need to be aborted — click here to receive an article on the definition, roles and advantages of the MRD

#2. Track and improve the accuracy of engineering

drawings3

Every error in a drawing adds delay for a vendor, contractor or production team, through rework and material waste, often driving mistrust with the design team. Best in class is less than 3 percent of drawings having any errors with parts-list count of 2,000-plus (typically 100 drawings). Measure

2 With the advent of Agile and other alternative program-ming methods some software companies abandoned the MRD as a way to capture and prioritize requirements, replacing it with the Backlog. We find that some software companies are moving back to having an MRD, particularly hardware companies that generate applications or create embedded software.

3 One can view the output of an engineering organiza-tion as a set of drawings, which include schematics, block diagrams, bill of materials, theory of operation, reliability tables, etc. We use the term “drawings” to characterize all the documentation that emerges from R&D related to a development project


mailto:info%40rapidinnovation.com?subject=Article%20on%20the%20definition%20and%20role%20of%20the%20MRD


White Paper 6

White Paper


the number of drawing errors per release to manufacturing. Identify the root cause of the errors and drive to zero.

#4. Measure the percentage of outsourced R&D to improve team productivity

The outsource metric can drive more effective and efficient teams. Some companies try to keep all development in-house, forcing key engineers and personnel into areas of work that do not utilize their skills and training effectively. Other companies outsource too much, leaving little core competency in-house. Find the balance for your company. Best-in-class outsource proportions vary across industry segments, from 15 to 35 percent.

#5. Don’t neglect time-to-profit

Why measure only part of the process? It starts with the voice of the customer, as represented by the MRD, and ends with sales – not manufacturing. Measuring time-to-profit demonstrates the effectiveness of the MRD, and how quickly the new product can ramp-up in sales.

Use metrics correctly:

● Fewer metrics is better than many.

● Every metric must have an owner, frequency and action plan to implement corrective action.

● Metrics without action are useless.

● Make metrics abundantly visible.

Click here for examples of product development metrics

A call to Action

While our research identified many areas of improvement, we have selected four that, if implemented, will improve the performance of your product development efforts.

#1: Establish and sustain a system of metrics

Establishing a rational system of metrics. This is a set of fully coordinated and carefully planned metrics. Every functional group always has a critical financial metric – for example, in marketing it’s Product Margin; in R&D it’s COGS (Cost of Goods Sold); etc.

A system of metrics is illustrated in Figure 2. The top of the pyramid must always be the key financial metric for the

organization (Outcome Metric). In this hypothetical example it’s time-to-profit. Lower in the pyramid are the critical process metrics that contribute the most to the achievement of the financial metric.

#2. Develop and consistently use a product roadmap and consistently measure its effectiveness

Prioritizing and scheduling new development projects

COGS Gross MarginLevel 3

Level 1Time to Profit

Level 4

• Quality of the MRD

• Number of Spins

• Time to Market• Accuracy of the Sales Forecast

Accuracy of Resource

Allocation

Reduction of employee

turnoverR&D Up-time

Level 2

Execution of the Strategic Plan

Figure 2: System of Metrics Illustration

The above example illustrate a System of Metrics, in this case for a specific development program

Notice that level two has the two critical metrics (COGS and Margin) that R&D and Marketing impact with every trade-off during the life of the design process. This promotes collaboration

The GM has chosen Time-to-Profit (TTP) as the King Metric (Level 1) — due to space limitations the metrics that make up TTP are not shown. TTP is an “outcome, financial” metric, which is appropriate for Level 1

The above is an example simply to illustrate an application of a system of metrics. The selection of the metrics, levels and amount of detail is up to your team.

click here for additional information on Systems of Metrics



White Paper 7

White Paper


figures high in our set of recommendations. Our research shows that much improvement is needed in this area. Incidentally, a spreadsheet with a list of all R&D projects does not constitute a product roadmap, although it’s a good start.

Marketing and R&D must collaborate closely in using a roadmap, which itself is driven by the strategic direction of the company established by the executive team. It’s a year-around effort; we suggest a review four times per year. This review is not a trivial effort; many times it may take a full-day meeting to update and make the necessary changes. We suggest it as a priority activity for marketing and R&D, and as such it deserves focus and thoroughness — and a small price to pay for improve velocity and higher quality.

Our research indicates that many of the problems during product development can be traced to loose or nonexistent product roadmaps. Consequently, it makes sense to take a close look at your existing process and raise it to its proper level of importance. Remember, the appetite for new products is high; your responsibility is to select the vital few that will make your company successful.

#3. Measure and improve R&D up-time

Every R&D organization we interviewed is staffed by carefully selected design engineers who are expected to develop innovative new products. Yet our research showed that there are many distractions that keep R&D personnel from doing what is expected of them. We identified 60 % as the up-time benchmark; that is, the better R&D organizations spend approximately 40 percent of their time doing things that can be classified as distractions — not a pretty picture. Up-time is defined as “the time spent by design engineers developing new products – without unjustified distractions – while using their personal core competencies to maximize their contribution to the design”.

Distractions are interruptions that take time away from designing innovative new products. For example, phone calls from the sales force for information that is already available from other sources, or meetings that are poorly planned and

perhaps should have never involved R&D

We recommend you start measuring up-time in R&D. If the current benchmark is just 60 percent, it seems reasonable that every product development organization has the opportunity to make their R&D team more productive. — Click here for an article on measuring R&D productivity and ways to improve it

#4. Measure time-to-profit to improve financial performance

Time-to-profit measurement identifies a date and financial goals for an early post-launch review. For example, during the start of product development, the marketing department sets a forecast for the new product; this guides investment and other activities. We recommend formal reviews six and twelve months after public announcement. A formal review should accompany the metric, to ensure understanding and, most important, clear action toward improvement

During reviews, marketing, R&D, manufacturing and other key stakeholders meet to compare progress to forecasted results. We suggest you review revenue, profit, COGS, margin, delivery accuracy and other critical outcome metrics. This focus provides an early indication of the success of a new product, with ample time for corrective action. For example, it allows evaluation of the various forecasts. Some companies also use this review to evaluate how well the new product introduction was handled, as well as early manufacturing ramp-up, reliability and other critical indicators. — Click here to receive a model of the Comprehensive Product Life Cycle to help you establish the right set of metrics.

Conclusions

Our research showed there are numerous obstacles to improving NPD velocity. Our sample represented several industries in the high-tech sector, and it’s reasonable to suggest that these same issues are relevant across that sector. Further research would be needed to assess if the findings can be extended to other business sectors.

Continue


mailto:info%40rapidinnovation.com?subject=Send%20me%20article%20on%20Engineering%20productivity

mailto:info%40rapidinnovation.com?subject=Model%20of%20Lifecycle


White Paper 8

White Paper


The good news is that the sample companies also revealed a rich vein of good practices and metrics that can be employed to overcome obstacles to NPD velocity. We have identified five Best Practices and four Recommendations for Actions, and they touch on all phases of NPD, ranging from early strategic and roadmap planning through market needs assessment and product definition, and on to final product development and commercialization. However, it is also apparent that the use of metrics themselves must

be carefully implemented – they should not be allowed to proliferate, there should be an owner for each metric, and the frequency of measurement should be carefully chosen. Most importantly, metrics must be backed up by clear and verifiable action, and be abundantly visible so they become second nature within the organization.

By following the recommendations above, we believe any technology company can increase its new product development velocity and accelerate time-to-profit.

Participate in the DialogClick here if you have questions, comments or suggestions. The authors welcomes your participation in this important topic.

● What one piece of advice do you have concerning the use of metrics in product development?

● What is the one metric you could not do without?

● Comments? Suggestions?

Continue


mailto:info%40rapidinnovation.com?subject=My%20comments%20on%20metrics


White Paper 9

White Paper


Appendix A: Product Development Metrics

Examples of Velocity Metrics in Product Development — Process Metrics

Rate of Reduction in time-to-market — expressed in percent improvement over time.

You can use this metric as part of an initiative to reduce the time to market for new products.

Use it to measure the consistency (or lack) of cycle time of development programs

Use to gain competitive advantage when first-to-market is important to your customers or to your company.

● A good way to track improvement in your ability to reduce time to market.

● Requires further analysis to identify the root causes of the delays and then corrective action to mitigate the reoccurrence of the problem; that is, why does time-to-market takes too long?

● Requires tenacity and several years of tracking to get the benefits.

● Requires discipline in selecting projects to compare. Comparing short s/w project with a major hardware platform would give false results.

Percent of Projects that Met Time-to-Market target

A good practice is to set a goal of reduction of the time-to-market, i.e., a percent increase of development projects that meet time-to-market

You can use this metric to identify areas of improvement through Retrospects. As with all metrics, you must perform root-cause analysis in order to identify the needed corrective action; then proceed to take visible action and track the improvements.

● Good metric to track improvement of time-to-market. .

● Can be difficult to compare projects (apples to apples).

● Similar to time-to-market, but focused on the project.

Percentage of projects that achieve break-even before original estimate

Breakeven is a critical milestone for every development project. It marks the date when the investment is paid back. Obviously, achieving breakeven prior to the forecasted date is a welcome improvement.

Comparing the percentage of projects, over time, that achieve breakeven before the originally estimated time is a composite measure of the overall organization’s improving or degrading ability to correctly estimate length of development and the sales ramp following launch.

● Easy to calculate, depending on information systems available

● Independent of project size and complexity

● Beware of padding of original estimates

● Takes time to collect data for long term comparison, but assuming data available, could be done retrospectively to assess how organization’s ability has changed over recent years.

● Must ensure consistency across projects in defining original development start date

click here to return to article Continue



White Paper 10

White Paper



Examples of Process Metrics for New Product Development

Consistency of Processes Across All Development Projects Over Time

This is an issue that impacts “concurrency”, the ability to develop several new products at the same time. Lack of consistency of processes across development projects creates defects which reduces productivity. It also creates confusion when team members move from one project to another, which causes delays due to the learning and “unlearning” curves.

Differences in tools and methods may be needed, but they must have a rational and defensible reasons for existing.

● A useful metric when there are several development teams working on new product development. Consistency of processes across teams improves productivity

● Can be difficult to track due to the barriers that may exist amongst various development teams. These can be political, or structural.

Defects per drawing

The objective is always zero defects — that is “do it right the first time”. Regrettably this is not always possible when inventing solutions and creating innovation. None the less, the need to measure the ability of a development organization to output drawings with the fewest defects is imperative.

There are many opportunities for error such as:

Incorrect parent part number

Incorrect component part number

Errors in schematics

Wrong or flawed documentation

Etc.

● Top metric for validating design and validating market requirements document.

● Essential to continuous improvement in R&D organizations.

● Can be used to measure at the assembly, sub-assembly or drawing levels.

● Fixing drawing errors can save time and material in Manufacturing Assembly, Stock Room, Quality Assurance, Fabrication Shop and Customer Service.

● Best in class Accuracy of Drawings > 98%

Total cost of development projects completed in current fiscal year as a percent of original estimates

Knowing how well you have estimated development expense in the past can be a good predictor of future performance and also an indication of the need to change estimation practices.

Use this information to perform retrospectives to identify areas of improvement in your ability to forecast accurately or accelerate development

● Easy to calculate

● Results >100% Shows need to take some action, but not necessarily the root cause of the problem. (Could be either inaccurate estimation or inefficient execution)

click here to return to article Continue



White Paper 11

White Paper



Examples of Financial Outcome Metrics in Product Development

Achievement of Time-To-Profit, or Time-To- Money

Prior to the official start of the development project, designate a date; for example, six months after the product is introduced. The time frame is up to you, but the intent is to get feedback on the success of the project soon after introduction to the market place.

Some companies establish several time-to- profit dates to track the progress and take corrective action, which is a source of lessons learned and good practices.

Forecast the actual sales, profitability, COGS or number of units expected at the designated date. What and how many items you track is up to you. We suggest they are all related to profits or the ability to generate profits.

Identify the actual result at the designated time and compare them with the initial forecast.

● Good metric to identify the early success or failure of a new product. You don’t have to wait years to find out.

● Good metric to determine the accuracy of the marketing forecast.

● Provides accountability and establishes a target to achieve.

Portfolio Profitability Index (PPI)

The objective is to develop a range of products that provide maximum return for the invested development effort.

PPI is the same calculation as PI (profitability index), but performed over a complete development portfolio and allows the impact of possible future roadmaps to be compared or to compare the impact of R&D between different business units to compare their efficiency as profit generators.

● A good way to measure the impact of a variety of potential portfolio directions

● For PPI comparisons, the investment period should be the same and in addition the measure of future cash flows should be for a specific period after the end of the investment period.

Return on Innovation

A way to measure the big R in R&D. Track innovation based on patents or patent applications.

Alternatively you can measure the number of granted patents that become part of a new product.

● Somewhat difficult to track due to the time frame.

Continue click here to return to article



White Paper 12

White Paper


Metric Maladies

Many organizations suffer from a set of maladies when it comes to metrics. The following list is intended not to cause any needless pain, but to help you identify and remedy the problems. We suggest that you review the list and then identify and mend the “maladies” that burden your organization — productivity and morale will improve. The most common maladies include:

Metric Proliferation: This happens when an organization develops an almost “feeding frenzy” for metrics. Where it seems that everything is measured, and much energy is

spent measuring things that are neither relevant nor helpful to

running the business effectively.

What To Do: Every metric must be challenged to ensure that it can contribute to improving the productivity of the organization. Further, metrics must be prioritized to ensure that you are measuring the vital few things that will provide the leverage to improve the performance of your organization.

What To Do: Develop a System of Metrics as outlined in this article. While it may take some effort to create your system, the productivity improvements will be worth it.

What To Do: Work with the members of your team and create an inventory of all the metrics, identify redundancies, conflicts, flaws; then call it to the attention of your managers and executives. They may not be aware of the number of metrics you have and the negative impact on productivity.

Metrics and No Action: A metric is useless if it is not used to create visible action that will result in improvement. Metrics have no reason to exist, except to serve as a way to improve performance.

What To Do: Ensure that there is a closed loop process for every metric, paying particular attention to “what” will be done with the measure, “who” will do it, and how often it should be done. Obviously, if you cannot document improvement, then it is likely that the metric is not being used properly.

What To Do: See also “Metric Proliferation” notes for more suggestions.

Metric Cacophony: This occurs when metrics are at odds with each other; i.e., not in harmony. Specifically,one metric goes against the intent of another metric causing confusion in the organization.

What To Do: All metrics must be part of a System of Metrics — a system where every metric builds on every other metric that results in the aggregate benefit. You must be able to layout all your metrics in a single document, and then you must ensure that metrics are in harmony with each other. They cannot be at cross purposes.

What to do: Conduct a thorough inventory of all metrics in your organization. Evaluate all of them against the business objectives of the organizations and then proceed to eliminate, change and re-prioritize

Metric Secrecy: This is where many good metrics are in place, but the people who could implement them do not have visibility. Metrics must be clearly communicated to those that are able to create the improvement in the organization.

What To Do: Every metric must have an “owner” — the person who is responsible for ensuring that the measure, reporting and action all take place. For every metric there should be a communication path where all stakeholders are informed and have the opportunity to contribute.

What To Do: Use modern technology to make the metric convenient to find and format it such that it’s easy for the audience to understand. An important note is that the definition of “convenient” is not up to you; rather, you should ask the audience to define it for you.

Metrics for Punishment: Sometimes, by design or accident, metrics are used as tools for punishment. This is one way to render metrics useless. It does not take too many instances of punishment to cause a negative reaction in the organization, which causes a rebellion wherein metrics are padded, hidden and otherwise sabotaged.

Appendix B: Metrics Maladies



White Paper 13

White Paper


What To Do: All metrics must be guilt free — a safe environment must exist where metrics are welcomed and used to improve productivity and employee satisfaction. Care should be applied not to create the impression that a metric can or will be used to punish or castigate anyone.

Metrics “Not for Us”: You did all the due diligence to create a metric, yet some in your team refuse to accept the clear message from your metric; thus, they start criticizing your methodology, the accuracy of the data, the phase of the moon and the meaning of life. This is classic denial — find every excuse to avoid measuring.

What To Do: Tenacity will win the day. Remember, “Data convinces, opinions confuse.” Or as someone said, “He who has the most data wins.” You should expect denial. So, be ready to deal with it. If the resistance persists, a face-to-face meeting may be in order where the objective is to diagnose the reasons for the resistance and define a clear win-win strategy. More acute cases may call for an intervention, which is a carefully planned course or action that involves people from your HR department, management and even outside experts

Metric Fossils: These are metrics that were established many years ago, and no one has questioned their validity in the business environment of today. Resources are spent in generating the metrics, but little or no value is derived.

What To Do: Identify the Fossils, then kill them and bury them in a very deep hole! And the sooner the better!

What To Do: This is where a System of Metrics is very useful. It will ensure that you have only the metrics you need and that the metrics you do have are valid and current.

Metrics “Because I Say So!”: Someone generally in a position of power establishes a metric but does not engage the organization to ensure understanding and support; consequently, the organization has no clue about the metric and support for it is non-existent.

What To Do: Executive and managers should know that

team members need the full context of a metric. For example: Why does the metric exist? What is expected from the team members concerning the metric? What

kind of support will team members expect from the person

establishing the metric?

What To Do: Ask for and demand information that will enable you to understand and support the metric.

Metric Massaging: This is when the organization begins

to play games with a metric. Due to certain forces, such as high-pressure, or fear of punishments, numbers start to be padded in order to make the results look favorable.

What To Do: Eliminate the fear by making the metric guilt free. Reinforce the value of honesty and integrity in

the organization. Review with your team the metric’s objective,

while avoiding excessive, emotional reactions.

One Size Fits All Metrics: Every project is different. Yet we often use a boilerplate set of metrics without considering the type of project. While standardization and benchmarking are desirable, use your judgment.

What To Do: Challenge the boilerplate approach. Recognize that every project is different; therefore, it may require that your metrics flex. Keep your eyes on the objectives and results of your project.

What To Do: Sometimes it makes much sense to have standard metrics in order to track progress or compare with other projects. In this instance, make certain that everyone knows the purpose of the metric and the need to

make it standard.

Orphan Metrics: This is when no one is assigned ownership for the metric or the owner of the metric is not known. In this situation, users give up on the metric because there is no feedback loop.

What To Do: For every metric there must be absolute clarity as to its owner. The owner must add the metric to his or her job description. If there is no known owner for a metric, get rid of it!

click here to return to article



White Paper 14

White Paper


What To Do: Yet another example for the need of a system of metric, which will make it easier to identify Orphan Metrics.

Metrics for Academia: A metric may be made overly complex. Generally, these metrics are perceived as irrelevant to “the real world” and are therefore dismissed.

What To Do: KISS — Keep it Simple, Sir! Keep your eyes on the ball by focusing on the outcomes, objectives and

results that you are trying to achieve. Challenge metrics that

seem overly complex to determine if their complexities are

absolutely necessary.

Misguided Metrics: This is where organizations measure the wrong things. While the intent is honest, and the metric provides accurate information, the organization cannot take improvement action. Worse yet, the metric may cause the wrong action. For example, using the number of hours employees work to measure productivity.

What To Do: Do the due diligence when developing the

metric to ensure that your objectives will be met. Involve

your team in the development of the metric, the collaborative

environment will highlight errors and omissions. Finally,

examine if your metric is causing the proper corrective action, or

if the metric is causing any action. You might discover that you

need to re-engineer your metric.

Fuzzy Metrics: This is where a metric has been poorly defined or its definition has not been communicated to all stakeholders. The consequence is that various people or managers will misinterpret the meaning or intent of the metric and create their own definitions.

What To Do: When developing the metric ensure precise definition, in plain language that is easy for all to understand. Proactively communicate the meaning and

intended use of your metric. Do not assume that though it is

obvious to you everyone else will understand

Click here to return to article

Participate in the Dialog

Click here if you have questions, comments or suggestions about “Metrics Maladies”. The authors welcome your participation in this important topic.

Do you have a “favorite” metric malady?

Continue


mailto:jose%40rapidinnovation.com?subject=I%20have%20a%20comment%20about%20your%20article%20on%20the%20MRD


White Paper 15

White Paper


Leading Metrics Lagging Metrics

Level 3Supports Levels 2 and 1

Level 1Drives all

other levels

Level 4Individual KPI’s —support Level 1

The King Metric

Leading Metrics Lagging Met-

Lagging Met- Lagging Metrics Leading Met-

Level 4 metrics are generally at the “individual contributor” level; i.e., what each employee measures to support all the metrics in the system.

4

Level 3 metrics are “departmental” measures or “team” metrics." Beware of the proliferation of metrics. Ensure that your organization truly needs a level 3.

3

The “The King Metrics” are the highest level of measures, generally at the company or organizational level — most commonly financial (ROI, ROE, ROO, Etc.) or some other critical “outcome metric”. These metrics drive all other metrics in your organization or team.

All projects and functional groups must synchronize to the King Metric — this will ensure that everyone is in alignment with the ultimate objectives of the organization or the team.

1

Level 2 metrics are generally at the “functional” level and include a combination of “leading” and “lagging” metrics.

2

The Mental Model of the System of Metrics

(The above template for a System of Metrics will need to be adapted to your business objectives, the size of your organization and type of business.)

The model shows a series of “levels” starting at the top with the King Metric — Level 1 (item 1 ). Notice that at the top of the pyramid there is always an outcome metric — Level 1 (item 1 ). At the product line level, the King Metric should be also an outcome metric, but it can be Product Margin, forecasted sales, cost. The King Metric can be a short set of “outcome” metrics depending on your business objectives and priorities. We recommend — as with all metrics — that fewer is always better than many

Below that, there are several levels depending on the size and type of organization — Level 2 & 3 (items 2 and 3 ). For example, for a small hardware company, a two-level pyramid may be sufficient. We always recommend that the fewer metrics the better, this is where prioritization and your business objectives play a critical role.

Level 4 metrics are the KPI’s (Key Performance Indicators) for individual contributors or for managers. These KPI’s are in support of the System of Metric

Leading Metrics

Glossary:

Outcome or Lagging Metric: A result metric that measure the end-point of a process, for example a the volume os sales of a new product — after the end of the development process

Process or Leading Metric: Metrics used to monitor a process, for example number of days to complete a milestone during development

Level 2Supports Level 1

Appendix C: The System of Metrics




White Paper 16

White Paper


1

4

3

2

High Maturity ● Effective, easy to use System of Metrics in place. Metrics regularly tracked, and continually improved. Vivid commitment to take decisive action based on the System of Metrics

● Best practices process fully functional (capture, use, improve and purge)

● A culture of managing by metrics is the norm and no longer a novelty.

● A closed-loop for Corrective Action system is in place and working

● Metrics are “guilt free”, embraced by the organization, managers and executives

The maturity curve shown above is simply a mental model, which is offered in this document as way to encourage you and your team to have an open dialog about how well you are utilizing your metrics, how effective they are, and if they are used to further the corporate objectives of your company. The curve is not comprehensive, but it does set a starting point for a self evaluation.

Start by determining the current level of maturity in your organization. This can be done by surveying your organization or by having an extended meeting to discuss the curve versus your current capabilities in metrics. At this stage, a qualitative result is acceptable as it is all you need to for improvements from one level to the next higher up. Do not allow the lack of defensible and quantitative information to keep you from initiating your efforts, quantitative information will come later, when it is needed to further your progress.

Strong elements of maturity ● A “System of Metrics” is in place and is reviewed regularly

● No formal best practices process. Some effective best practices are documented and used

● The culture understands the roles of metrics, but has not embraced them

● A closed-loop process for corrective action is in place and but used sporadically

● Management by fact-and-data becoming part of the culture. Executives model the way

Good maturity NPD process ● Metrics in place and reviewed regularly. Some metrics in place, lack consistency, the focus is only on the financial metrics

● Best practices are communicated through casual interaction.

● The culture understands the roles of metrics, but has no passion to utilize them

● Corrective action is sporadic, but mostly not working

● Managers and Executives “manage-by-metrics“ but mostly financial

Low maturity NPD process “Ad Hoc” ● Metrics not in place or not reviewed regularly. Management by anecdotes and by today’s problems

● No best practices

● The culture does not value metrics, some see them as a threat, many make jokes about metrics

● No corrective action in place, manage-by-crisis

● Metrics are used to punish and to lay blame!


Metrics to Reduce Time-to-Market & Accelerate Time-to-ProfitWhite Paper


Note that in this example, Level 4 is about KPI’s for the Eng. & Mktn leadership

Example for an Engineering and Marketing Organization for a Specific Development Program

● The above example illustrates a system of metrics for the Engineering (R&D) and Marketing organization of a Business Unit.

Please note that it is an example to illustrate the model using a fictitious scenario.

● (In this scenario) The initial model of this System of Metrics was created after approval of the Product Proposal (The Business

Case)

● As the development project evolves, all metrics will be monitored regularly during Program Review, to ensure that all

the objectives are achieved — Particularly the King Metric, which in this example is a margin of 47% at time of public

announcement

● Note that it does not include other functions due to a hypothetical choice of the “owner” of the System of Metrics, which in this

example is the General Manger.

Points of interest:

1. Note that the King Metric (Level 1), in this example, is for a target product margin. The Metric Owner could have selected other

target or targets. Perhaps, there is a strategic initiative to improve product margin.

2. All other metrics below the King Metric are in alignment; that is, support the achievement of the King Metric and most

importantly do not prevent the target.

Appendix D: One Example

A System of Metrics for a Hypothetical Development Program

Target COGS: 53%Warranty: $32K for

year 1, 2 and 3 after announcement

Target Price (ASP) to achieve 47% margin: $40,666 per unit

Product Margin: 47%

at public announcement

Engineering Up-time: 80%Tooling costs: $160KR&D Outsource: 8% — $1.25M

• Engineering Managers to Ensure Zero De-fect for all Outsourced R&D

• Product Management to ensure Zero Defects for the MRD

All Engineering Managers and Leads to en-able Up-time of 80% throughout the design phase

Level 1

Level 2

Level 3

Level 4

Critical Assumptions

• Public Announcement (Time-to-market) to occur 16 months after the Project Kick-off meeting

• Time-to-profit: 12 months after Public Announcement

Quality of the MRD >95%(Measured through a survey of R&D)



White Paper 18

White Paper


About the Authors

About Rapidinnovation, LLC.A few of the critical questions we help answer:• How can we get closer to our customers in order to create more profitable products?• What market segments are profitable, not-profitable, growing and where should we invest our R&D

and Marketing Dollars?• How do we grow beyond our core?• How can we reduce time-to-market and shorten time-to-profit?• How can we make Agile work for us?• How do we make our schedules more predictable?

We have been serving the high-technology community around the world for the last 17 years. Our ability to concentrate on high technology has given us a unique insight into the dynamics of the fastest-moving market around. Every one of our Business Consultants is a seasoned veteran of corporate life in high-tech companies. All have engineering degrees with advanced degrees in business plus years of experience helping technology companies achieve their business goals — credentials that are brought to bear to solve our customers’ challenges.

We work with our customers by focusing on the high-leverage areas of the company: R&D and Marketing where we collaborate, train, coach and facilitate every aspect of product development, innovation, and profitability. We also work with Senior Staff to develop measurable growth management strategies and then help in the clear execution.

Our list of satisfied customers includes some of the leading global technology companies. Contact us to help you solve the most thorny problems confronting you.

Visit us at: www.rapidinnovation.com

About the Authors


http://www.rapidinnovation.com

White Paper 19

White Paper


Jose Campos, Founder Rapidinnovation

Jose and his team of seasoned consultants collaborate with B2B companies around the world on the methods, tools and cultural shifts needed to rapidly and consistently deliver superior value to customers.

He has spent his career working to define and deliver innovative products to market. For 30 years he has passionately concentrated on developing new methods to discover and deliver customer value in the tumultuous world of high-tech.

Jose has written several acclaimed books, including Voice of the customer for Product deVeloPment, a methodology for discovering, prioritizing and using customer requirements to create clearly differentiated products faster than the competition, and risk management for Product deVeloPment, which includes strategies and methods to reduce the impact of uncertainty on your investment. flexible Project management for Product Development is his latest publication. Jose was also coauthor of Project management toolbox in collaboration with Dr. Dragan Milosevic.

His focus on high-tech and their technical background allow Jose and his team of experienced consultants to provide meaningful and practical coaching to technology professionals. They have traveled the world to help R&D and Marketing teams improve their ability to create innovative products customers love.Contact Jose: [email protected]

Bob Buxton brings more than twenty years of experience in both R&D, marketing and business segment management within high-technology companies His R&D experiences were primarily in connection with Radar and Communications equipment design. His marketing and business management experience spans the range of strategy development, portfolio management, new product definition and development, creation and execution of go-to-market strategies and product life cycle management. Bob holds a masters degree in Microwave

and Modern Optics from University College, London and an MBA from George Fox University, Newberg Oregon. He is a Chartered Engineer and a Member of the Institution of Engineering and Technology

Dave Worsley is a hands-on professional with over 30 years experience working for manufacturing and service companies in Europe and USA. He specializes in, and coaches, operations effectiveness, project management, client interaction, using continuous improvement methodologies, process and maturity mapping, cost reduction programs, with statistics and statistical process control.

Dave thrives on process improvement by applying his engineering disciplines to drive focus on metrics, Lean Six Sigma 5S behavior. He currently works with Energy Trust of Oregon, supporting their offerings to help industrial companies incorporate energy management as part of their strategic business effort, and to help utility customers meet their energy savings goals.

Dave earned a BSc (Honors) in Electronics from London University, is a senior member of Institution of Electrical and Electronic Engineers (IEEE) and a Certified Six Sigma Black Belt. Dave also has certifications from Portland State University and Washington State University on operations effectiveness and Lean training. Dave has been granted three patents for the engineering design for paper handling equipment and has received various awards from Fortune 500 companies

About the Authors


mailto:jose%40rapidinnovation.com?subject=


White Paper 20

White Paper


Additional ResourcesRapidinnovation has published a series of books on product development. They were written by experienced product developers for

those who must implement critical business processes in product development environments.

The Voice of the Customer for Product Development

Your illustrated guide to obtaining, prioritizing and using customer requirements and creating winning products

Step 2: Organize Your Visits

85 For comments, questions or additional copies please e-mail the authors: [email protected]

CONTINUED

Detailed Process: Discussion Guide

SET THE FOUNDATION

Ensure that the objectives for the customer visits are documented and nal.Circulate the objectives and any other relevant material to the members of your team and any other relevant stakeholder.

1

CALL A BRAINSTORMING MEETING

Decide who needs to attend based on the objectives and each person’s ability to contribute.Schedule the meeting for at least three hours. This may reduce attendance, but quality is more important than quantity.Secure a conference room with ample space.

2

HintPRIORITIZE TOPICS FOR THE MEETING

In step #3 (Brainstorm), you are defi ning and prioritizing the topics of conversation. A good way to start the brainstorm is to ask your team members to write down all the questions they wish to ask their customers (use Yellow Sticky Notes).

BRAINSTORM

Use the Af nity Diagram method or any other formal framework for brainstorming.

Brainstorm all possible • topics.Organize the topics or • questions.Consolidate the topics.• Prioritize the topics.•

You should end up with no more than three to ve topics. Prioritize based on importance to the objectives and using a clear criteria for selection.

3

HintGenerally, you will need one hour to

cover three topics, or perhaps fewer. Remember that you need to allow the customer to talk, and this takes time. It is not a survey, it is a conversation where you need to probe and let the customer talk and think.



Types of Customers Subsegment A Subsegment B

Customers you have lost. 1. That is, customers who no longer buy from you for a negative reason.

Customers you never had. 2. That is, customers who ought to have bought from you, but never did.

Customers who have stopped buying from you.3. That is, you have not lost them, but they have not bought from you in some time. Different than Item 1 above.

Future customers.4. That is, customers in new markets, new companies, new applications, etc.

Customers who are buying much less from you.5. Those who, in the last six, 12 or 18 months have shown a drastic reduction in purchases.

Customers who are buying much more from 6. you.

Subsegments A and B are there to provide one additional level of segmentation. For example, you might make A the U.S. and B Europe or you can make A Managers and B Technicians. You need not use both A and B in your sample selection; it’s your choice.

Example: Customer Archetypes T


OUNDATION

Ensure that the objectives for the customer visits are documented and Circulate the objectives and any other relevant material to the members of your team and any other relevant stakeholder.

Understand What Customers Value


RankCustomer Value Driversin Order of Importance Category

Competitor A

Competitor B

No.Rating (1-10)

Score Rating (1-10)

Score

1 29% Ability to address existing and anticipated technical Ben/Tech 8 2.32 10 2.90 You need to

be the BEST in satisfying the most important Customer Value Drivers (CVDs).

2 18% Positive experience with the product. Ben/Emo 9.5 1.71 8 1.44

3 15% Positive experience with the company. Ben/Emo 5 0.75 9.5 1.43

4 12% Ease of operating and maintaining the system Ben/Tech 7 0.84 8.5 1.02

You can be COMPARABLE to competitors in the second tier of CVDs.

5 8% Increased asset utilization. Ben/Eco 8 0.64 4 0.32

6 5% Ease of integration into company processes and Ben/Oper 9.5 0.48 4 0.20

7 5% Cost of ownership (CoO). Cost/Money 8.5 0.43 6 0.30

8 3% Impact on the organization if something goes wrong. Cost/Risk 9 0.27 7 0.21 You can be

INFERIOR in satisfying CVDs that are the least important, provided your performance is still

9 3% Increased revenue/throughput. Ben/Eco 7 0.21 4 0.12

10 2% Purchase price. Cost/Money 6 0.12 4 0.08

All 100% Product Value Index 7.77 8.02

Example: Product Value Index

1 2 3 4

The Product Value Index (PVI) is a numerical expression of value. By quantifying the value that a product represents to a target segment, you can determine how the offering of one company ranks compared to that of another company. The PVI is calculated by first multiplying the Importance percentage (item 2) by the Rating (item 5)—this generates the Score (item 6) for each CVD—and then totalling the Scores to create the PVI (item 7).

5 6

7

T Step 3: Interview Your Customers


INTERVIEWER: How do you feel about this software product?CUSTOMER: It is very dif cult to use.

INTERVIEWER: Tell me more?CUSTOMER: The menu system is not intuitive.

INTERVIEWER: Tell me why it’s not intuitive?CUSTOMER: I get lost all the time.

INTERVIEWER: Why do you think you get lost using the menu?CUSTOMER: Well, there are too many layers and no way to get back.

ROOT MESSAGE: The menu system has too many layers and no way to get back, which makes customers feel that the product is too dif cult to use.

Q1st Why

2nd Why

3rd Why

ANote the root message — i.e., actionable input from the customer.

Example 1: The 5 Why’s

INTERVIEWER: Tell me about your printing needs.CUSTOMER: I want a printer that prints at more than 20 pages per minute, and 30 would be better.

INTERVIEWER: Why is that important to you?CUSTOMER: The one we have now does 10 pages/min and it is too slow.

INTERVIEWER: In what circumstances is it too slow?CUSTOMER: In particular when we have a queue of documents with heavy graphics.

INTERVIEWER: Tell me more.CUSTOMER: We often have to print 60-100 page reports at the last minute before a presentation, and these reports contain lots of graphics. It is not atypical to have 3-5 users printing reports at the same time.

INTERVIEWER: What about documents with text only?CUSTOMER: No problem there. It is only when we have these heavy graphics reports and when 3-5 users are queueing.

Q1st Why

2nd Why

3rd Why

AFrom feature to problem — i.e., don’t stop probing until the customer voices the problem.


4th

Edit

ion

tools fully-illustra

ted

processes

examples

templates and

worksheets

The Voice of the Customer for Product Development

This VOC guidebook ad-dresses the pressure to obtain customer requirements rapidly — and shows you how to turn this knowledge into innovative products faster and better than your competitors.It’s a step-by-step guide to every aspect of obtaining, processing and applying knowl-edge of your customers — so

Flexible Project Management for Product Development

Your illustrated guide to making project management work in tumultuous development



CONTINUED


SET THE FOUNDATION


1



2



BRAINSTORM




3

















Competitor A

Competitor B

No.Rating (1-10)

Score Rating (1-10)

Score
















1 2 3 4


5 6

7








Q1st Why

2nd Why

3rd Why








Q1st Why

2nd Why

3rd Why



4th

Edit

ion


ted

processes

examples

templates and

worksheets

Flexible Project Management for Prod-uct Development

Designed for product de-velopers who are tired of traditional project manage-ment approaches that don’t account for the real-world intricacies and challenges of product development.The step-by-step design of this guidebook makes it a valuable reference for you and your development team

Risk Management and FMEAfor Product Development

Your illustrated guide to reducing time-to-market through risk management and FMEA



CONTINUED


SET THE FOUNDATION


1



2



BRAINSTORM




3















OUNDATION





Competitor A

Competitor B

No.Rating (1-10)

Score Rating (1-10)

Score
















1 2 3 4


5 6

7








Q1st Why

2nd Why

3rd Why








Q1st Why

2nd Why

3rd Why



4th

Edit

ion


ted

processes

examples

templates and

worksheets

Risk Management and FMEA for Prod-uct Development

In product development, the imperative to reduce time-to-market is always present — and risks can cause cata-strophic delays. What if you could predict and manage risks, so they don’t interfere with your time-to-market goals?This step-by-step guide addresses every aspect of identifying, prioritizing and mitigating product develop-ment risks. It’s written by vet-eran product developers who

Debriefs and Postmortems for Product Development

Your illustrated guide to improving performance through lessons learned



CONTINUED


SET THE FOUNDATION


1



2



BRAINSTORM




3















OUNDATION





Competitor A

Competitor B

No.Rating (1-10)

Score Rating (1-10)

Score
















1 2 3 4


5 6

7








Q1st Why

2nd Why

3rd Why








Q1st Why

2nd Why

3rd Why



4th

Edit

ion


ted

processes

examples

templates and

worksheets

Debriefs and Postmortems for Product Development

Debriefs are the most effective way to improve development-team performance — and that translates to shorter time-to-market and more rewarding innovation development.This step-by-step guide covers every aspect of postmortems, identifying what to do-more-of and what to do-less-of to improve future product-development performance. It’s written by veteran prod-uct developers who understand how to obtain the right input from your team, and

Actionable Metrics for Product Development

Your illustrated guide to developing and using metrics to improve product margins and reduce time-to-market



CONTINUED


SET THE FOUNDATION


1



2



BRAINSTORM




3















OUNDATION





Competitor A

Competitor B

No.Rating (1-10)

Score Rating (1-10)

Score
















1 2 3 4


5 6

7








Q1st Why

2nd Why

3rd Why








Q1st Why

2nd Why

3rd Why



4th

Edit

ion


ted

processes

examples

templates and

worksheets

Actionable Metrics for Product Development

“What gets measured gets done” says the refrain. This is particularly true in product de-velopment due to the many competing priorities, from time-to-market, to gross margin to time-to-profit and many more.This book will guide you through an assess-ment of your current metrics and their impact on

For questions and comments: [email protected]

Coming soon

Coming soon


mailto:info%40rapidinnovation.com?subject=Send%20me%20information%20about%20the%20book%20series

White Paper - · PDF fileUse Metrics to Reduce Time-to-market and ... Bob Buxton —...

Documents

Transcript of White Paper - · PDF fileUse Metrics to Reduce Time-to-market and ... Bob Buxton —...