Findings and Recommendations from Cognitive Testing for the … · 2016. 4. 4. · – Round Two...

Findings and Recommendations from Cognitive Testing for the Survey of State Government Research and Development (SGRD) – Round Two with Resolutions

Prepared for:

Christopher Pece, National Science Foundation (NSF)

Rebecca Morrison, NSF

John Jankowski, NSF

Prepared by:

Heidi St.Onge and Kenneth Herrell, Data Collection Methods and Research Branch (DCMR)

Economic Statistical Methods Division (ESMD)

U.S. Census Bureau

June 8, 2015

Findings and Recommendations from Cognitive Testing for the Survey of State Government Research and Development (SGRD)

– Round Two with Resolutions June 8, 2015 of 68

Table of Contents Executive Summary ....................................................................................................................................... 3

Research Objectives ...................................................................................................................................... 5

Research Methodology ................................................................................................................................. 5

Limitations .................................................................................................................................................... 5

Findings and Recommendations ................................................................................................................... 5

General Findings and Recommendations ..................................................................................................... 6

Specific Item-related Findings and Recommendations .............................................................................. 10

Findings and Recommendations on the Graphic ........................................................................................ 19

About the Data Collection Methods and Research Branch (DCMR) ........................................................... 23

Description of Cognitive Interviewing Techniques ..................................................................................... 24

Appendix A: Draft Questionnaire for Cognitive Testing – Round two ........................................................ 25

Appendix B: Draft Graphic .......................................................................................................................... 57

Appendix C: Cognitive Interviewing Protocol ............................................................................................. 58



Executive Summary The Survey of State Government Research and Development (SGRD) is a biennial survey conducted by

the U.S. Census Bureau, on behalf of the National Science Foundation (NSF), that collects data on state

research and development (R&D) expenditures from state government departments, agencies,

commissions, public authorities, and dependent entities with research and development activities.

Currently, the level of data captured on this survey does not provide a complete picture of the

intergovernmental fiscal relationships of R&D in state government. Census Bureau staff from the Data

Collection Methodology and Research Branch (DCMR) conducted two rounds of cognitive interviews

with agency respondents for the SGRD. The interviews centered on a draft of a re-designed survey

instrument intended to capture additional detail to meet the needs of NSF. The additional content that

was added for testing intends to address the uniqueness of state government research and

development (R&D).

To test ideas that came up from the first round of interviews, Census and NSF modified the draft

questionnaire (Appendix A) prior to the second round. During the second round of testing, respondents

were presented with alternative versions of specific questions and instructions that presented issues

during the first round of testing. Alternative versions were provided for the following

questions/sections of the draft questionnaire:

The “What is R&D?” introductory text

Question 3 (R&D Screener)

Question 6 (Summary of internal and external R&D expenditures)

Question 9 (FY 2015 Internal R&D Expenditures by “Character of work”)

Question 10 (External R&D Expenditures by Performer)

Additionally, round two included testing of a new graphic that was designed to help aid respondent

understanding of the survey by outlining the monetary flow of funding that the survey questions are

based on. A copy of this graphic can be located in Appendix B.

In total, seventeen interviews were conducted in California, Oregon and Washington. During testing, we uncovered a number of useful findings about reactions and data availability for respondents to this survey. The results of testing will be discussed in detail in this report. Some highlights include:

Many respondents would have to consult other staff members to respond to these questions.

Respondents did not always understand the definition of “R&D” and how it would apply to their

agency, and the “What is R&D?” section did not always help respondents understand the

definitions.

Respondents preferred the original version of Question 3 with yes/no questions because of the

simplicity of answering the original version.

Respondents generally were able to understand the terms “internal” and “external,” with a few

exceptions.



Respondents preferred the grid layout of Question 6.

Respondents were not always able to reliably break down their R&D expenditures by Character

of Work (i.e., Basic research, Applied research, Experimental Development) because these are

not readily available in their records.

Respondents preferred the term “type of entity” over “performer.”

Generally, respondent reactions to the graphic were positive.



Research Objectives The Survey of State Government Research and Development (SGRD) is a biennial survey conducted by

the U.S. Census Bureau on behalf of the National Science Foundation (NSF). This survey collects data on

state research and development (R&D) expenditures from state government departments, agencies,

commissions, public authorities, and dependent entities with research and development activities. The

interviews described in this report focused on evaluating respondents’ interpretations of questionnaire

items, and on investigating the availability and reportability of both existing and new content. During

this second round of testing respondents were also provided with a graphic to determine how this

additional reference tool would impact understanding of survey content. For several questions,

respondents were provided with alternative versions to evaluate respondent preferences,

understanding, and reportability.

Research Methodology During April 2015, seventeen cognitive interviews were conducted with government respondents in the

following metropolitan areas; Sacramento, CA; Oakland, CA; Salem, OR; Portland, OR; and Olympia, WA.

Interviews were conducted by Census Bureau staff from the Data Collection Methodology and Research

branch (DCMR).

Cognitive interviews are a questionnaire development technique that asks respondents to describe their

thought process as they proceed through a questionnaire, allowing survey designers to better

understand how respondents are interacting with the survey, in the respondents’ own words. The

strategy for probing specific responses involves non-leading, open-ended questions whereby initial

questions are followed up with additional questions that probe respondents’ answers for more detail,

until a sufficient understanding of question interpretations and response strategies is achieved and any

apparent or potential problems are identified and their causes determined. The protocol used in these

cognitive interviews are included in Appendix C.

Limitations

The findings presented in this report are subject to certain limitations, considering respondents were

purposively selected based on location and having reported to past survey. Therefore, the resulting

sample should not be considered statistically representative.

Findings and Recommendations The remainder of this report presents findings and recommendations from the second of two rounds of

cognitive testing. These findings and recommendations are organized into three sections: general

findings and recommendations that apply to multiple items, item-specific findings and



recommendations, and specific findings and recommendations related to the graphic. Please note that

due to time constraints, and the need to address specific survey content, Questions 1, 7, 8, and 11 were

not evaluated during Round two.

General Findings and Recommendations

Finding #1: Definition and Use of the Term R&D

Respondents had different ideas regarding the term “research and development” and what activities

would be considered R&D within their agency. During the interviews, a few respondents indicated they

were initially unsure why they were receiving this survey and had to spend some time understanding

what information the survey was collecting.

While discussing the concept of research and development, some respondents said things like, “I’m

trying to figure out what research is” and “we don’t really have any of this,” whereas some respondents

indicated that their agency would have “research and probably development,” or “research and

innovation.” The types of R&D activities respondents described at their agencies spanned such diverse

topics as bridges, fish migration, zero emission vehicles, DUI intervention programs, and forest fire

prevention.

The majority of respondents reported they would gather data from other staff members at their

agencies, either directly, by asking other staff to provide the data, or indirectly, by relying on what other

staff had entered into their records. Commonly these employees would be located within “programs”

and sometimes were called “program managers” by respondents.

Recommendations:

Consider briefly explaining the definition of R&D in the mail-out materials or the

scripted email message from the state coordinator to whom this survey applies.

Consider adding additional examples that cover a broader range of R&D activities.

Continue to evaluate the instructions during usability testing.

Resolutions:

NSF is considering adding an insert/flyer similar to the one drafted for the Non-Profit R&D survey on the definitions of R&D. The Business R&D and Innovation Survey (BRDIS) flyer is another model, but as structured it is more about the value/services offered by Census to rather than info about what is R&D. NSF will review this with Census and make a decision.

NSF will provide examples to census to add that cover additional R&D activities (e.g., social science examples).

Census and NSF will continue to evaluate the instructions during usability testing.



Finding #2: Staff or Department Solely Dedicated to R&D

When probed, most respondents indicated their agency did not have staff or a department solely

dedicated to R&D. Some respondents indicated they might have “program managers” that may have

some role in the R&D activities. While others said they would have “managers over the project” but

these managers were not just managing R&D, and they would have other duties “but not only research.”

Recommendation: Consider removing Question 3A from the screener, as this question may not

help respondents identify that they have research and development.

Resolution: This recommendation will not be incorporated into the survey at this time. NSF will

monitor the responses to the screening questions during the next survey cycle and will re-

consider changes for future survey cycles where applicable.

Finding #3: Use of the Terms “Internal” and “External”

Based on the findings from the first round of cognitive interviews, the terms “intramural” and

“extramural” were reverted to “internal” and “external” during Round two; “internal” and “external”

are the terms used in the survey in years past. During this second round of testing it seemed that some

respondents who did not view the graphic were still unsure of the meaning of the terms “internal” and

“external” R&D within the context of the survey. However, it seemed most respondents who saw the

graphic at the beginning of the interview were able to describe the difference between “internal” and

“external” R&D expenditures. These respondents were able to identify the differences between these

types of expenditures while reviewing the graphic as well as while responding to Questions 4 and 5 in

the survey.

Recommendations:

Continue to evaluate respondent understanding of the terms “internal” and “external”

during usability testing.

Consider using “R&D activities conducted by your agency’s staff” or “R&D activities

funded by your agency but conducted by other entities” in place of “internal” and

“external” R&D activities, respectively.

Resolutions:

The terms “internal” and “external” will be evaluated further during usability testing.

The recommendation to use the phrase “R&D activities conducted by your agency’s staff” or “R&D activities funded by your agency but conducted by other entities” in place of “internal” and “external” R&D activities will be considered, although there is concern about the question getting too wordy. NSF will review final text and consider the alternatives.



Finding #4: Time to complete the survey

While some respondents indicated the survey would take more time to complete than last time, most

respondents estimated the current version of the survey would take about the same amount of time to

complete as the previous version of the survey they completed in 2013. Since we did not review all of

the survey questions during these interviews, it was difficult to determine whether respondents were

considering all of the survey questions or just the subset we discussed during these interviews when

creating this estimate.

The amount of time respondents estimated it would take to complete this version of the survey varied

from about 15 minutes up to about 8 hours and spanned between one day and a work week. These

respondents said things like, “identifying the accounts and getting the numbers would be the hard part,”

and “we’d spend more time in conversation with the lead guys to keep them on the same page of how

this is interpreted.” It was sometimes difficult for respondents to quantify the amount of active time

that would be spent completing the survey since several respondents delegated parts of the survey to

other employees within their agency.

While discussing the process for collecting data for this survey, some respondents indicated they would

spend time creating other materials (e.g., Excel files, shortened instructions, emails) to help the other

employees who are reporting these data understand what to include and exclude. One respondent

indicated the “spreadsheet took a couple of hours” to create. Respondents also indicated that they like

having printable or downloadable copies of the questionnaire to distribute to the people who help them

with their response.

One respondent said, “the timeframe was hard to juggle” and that they would suggest giving agencies

more time to complete the questionnaire. However, this respondent recognized that the timeframe

might have been impacted by the amount of time it took to route the survey through their agency.

Recommendations:

Continue to investigate the time it would take to complete this survey during usability

testing.

Continue to investigate the types of reference materials respondents would create and

the amount of time this takes to help employees respond to the survey request.

Consider developing other tools that could be created by NSF or Census to aid

respondents, and reduce the need for respondents to create tools themselves.

If necessary, update the OMB burden statement to reflect a change in burden hours.

Resolutions:

Census and NSF will continue to evaluate the time it would take to complete this survey




Census and NSF will continue to investigate the types of reference materials

respondents would create and the time it takes for employees to respond to internal

requests.

NSF does not have the capacity at this time to develop additional tools (e.g.,

supplemental reporting spreadsheets), but both NSF and Census will consider these

options in future iterations of this survey. A pdf version of survey will be developed by

Census, as required under the IAA for operations; we assume respondents will use that

to send to additional sub-reporting units.

NSF agrees with possibly updating the OMB burden statement to reflect a change in

burden hours. They will evaluate estimates from cognitive testing. NSF may consider

adding a question to the survey to ask them how long it took them to complete the

survey, with an eye toward adjusting the burden in the future. NSF tried to do

something similar on the 2012 Higher Education R&D Survey

(http://www.nsf.gov/statistics/srvyherd/surveys/srvyherd_2012.pdf) but was instructed

by OMB to remove it for future survey cycles. We anticipate similar guidance from OMB

were we to try it again for the Survey of State Government R&D.

Finding #5: Gathering the data

As mentioned in Finding #4, several respondents we interviewed during Round Two indicated they

would reach out to others while completing this survey. Some respondents indicated they would reach

out to “the programs” or “program managers.” Additionally, respondents described needing access to

different types of records in order to provide this information. Some respondents indicated they would

query their “financial system” while others would review the specific grants associated with this work.

Additionally one respondent said, “I can’t just go to a database and look it up,” and, “It’s not something I

already have.”

Recommendation: None at this time.

Resolution: None needed.

Finding #6: How the data are used

In general, it seemed respondents were unaware of the purpose of the survey and the uses of the

results and recommended that the uses and goals of the survey be communicated more clearly. A

number of respondents said things like, “What do you do with the data? What decisions do you make?”,

“How is this used? Do you need 100% or can it be 98?”, “What is the purpose for this survey and how

you’re going to use it?”, and “What do you use the data for?”

Recommendations:

http://www.nsf.gov/statistics/srvyherd/surveys/srvyherd_2012.pdf



Consider adding additional introductory text to the Web instrument, cover letter,

and/or survey invitation materials that briefly describes the use of the survey data (e.g.,

within the log-in page).

Consider adding additional text and/or links to prior publications of survey data.

Provide respondents with a way to request results from the survey.

Resolutions:

NSF agrees with providing additional text and/or links to prior publications of survey

data and providing respondents a way to request results from the survey.

Although a link to the NSF survey page may already appear in the Governor’s Letter, NSF

is also considering providing this link on the log-in screen and on the survey’s business

help site.

Specific Item-related Findings and Recommendations Findings within this section will be discussed based on the order in which these items appeared on the

draft questionnaire (see Appendix A).

Finding #7: Fiscal Year – Question 2

Almost all of the respondents we interviewed were able to answer this question with relative ease. One

respondent indicated it was “handy” to clarify the timeframe for the survey up front because “pooled

funds can be calendar year or fiscal year.” Some of the respondents we interviewed in Oregon, as well as

one agency in Washington, indicated their state or agency follows a biennial budget. Although they

operate on a biennial budget, respondents indicated they would be able to report data based on the

fiscal years we were requesting without a problem. Additionally, several respondents indicated their

annual reporting would be completed prior to the collection period for this survey so there would be no

problem reporting for these years. None of the respondents we interviewed indicated the hyphenated

fiscal year (i.e., 2013-14) was difficult to understand.

While probing on this question respondents were asked what the gray boxes indicated to them.

Although some respondents understood this was a pre-filled box that was not editable, most

respondents were unsure what the gray color indicated. It was not clear if this finding was a function of

using paper as a proxy for a Web instrument.

Two respondents in California showed some confusion about whether or not we were asking for the

state or federal fiscal year; they either did not notice that we were asking about “your agency’s fiscal

year,” or the fact that we were asking for their agency’s fiscal year still did not make it clear that we

wanted their state’s fiscal year.

Recommendations:



Conduct additional testing with respondents during usability testing to identify how

respondents understand and report using the gray boxes.

Put the word “state” somewhere in the question to make it clear we are asking for their

state’s fiscal year (e.g., “your state agency’s fiscal year”).

Resolutions:

Respondent understanding and reporting using the gray boxes will be tested further


NSF agrees with the recommendation to include the word “state” within the question to

make it clear we are asking for their state’s fiscal year and will implement prior to

usability testing, if possible. Census will make this change prior to usability testing.

Finding #8: What is R&D Screens – with Alternative

Before discussing this section, several respondents described that they were confused about the

definition of “R&D.” These respondents indicated they would like a better understanding of what R&D

is within the survey. Some respondents indicated they were unsure why they had received this survey.

It was unclear in the interviews whether these definitions helped them understand why they received it.

Respondents were presented two versions of this section during this round of testing. These two

versions had roughly the same content, but the content was presented in different formats and with

different headers (see Appendix A). It seemed respondents from the second round of testing spent

more time reading through these definitions than respondents from round one, although this finding

may be a function of presenting two versions of the content for comparison.

Overall, respondents indicated a preference for the alternative version of these screens. Regardless of

their overall preference, several respondents indicated they liked the bulleted list that describes “What

is NOT R&D.” A couple of respondents indicated they would like these instructions to be clear, direct,

and succinct, saying things like, “it is important to make the definition clear, it’s better for everybody,”

and “I like the idea that it’s on one page.”

Similar to findings from Round one, some respondents described having difficulty finding their work

within these examples saying things like, “it is tough in a standard survey to see what we should be

counting based on the examples.” Several respondents indicated they would like to see more social

science examples listed to help them identify how to classify their agency’s activities. Several

respondents also noted that the examples seem to skew towards basic research (or their understanding

of what basic research is), with one respondent saying that these instructions “don’t count applied

research” and that their agency only conducts applied research. One respondent explained that they

summarized the old version of the instructions before sending them out to the program areas to

complete the survey.



As far as the content on these pages, respondents had differing reactions to the information that was

presented. One respondent honed in on one part of the definition and said, “hasn’t been done before?

Is there anything that literally hasn’t been done before?” One respondent felt these instructions were

helpful recognizing that, “there were things that I thought were R&D and when I read [this] I realized

was not”

A number of respondents indicated there were items within the “not R&D” list that they would expect

to be included. Respondents said things like, “State planning and research does allow technology

transfer in these,” and “On the ‘Not R&D’ list there are two things we’d consider – business

development services and economic/policy/feasibility studies we do some of this for research

purposes.” Furthermore, a number of respondents felt that policy was difficult to exclude saying,

“Almost all research we do effects policy so it’s a little tricky,” “I couldn’t separate the time spent on

research versus policy,” and “to a degree all research is used for policy.” Other respondents complained

about excluding things like market research, information systems, and monitoring. Additionally, some

of these respondents indicated they could not exclude everything that we ask them to exclude.

Some respondents noted that a few of the requirements for R&D are not very limiting. One respondent

noted that many things that they do are novel, or create solutions that are useful to others, or have

uncertain outcomes, but might not necessarily be considered R&D.

Lastly, one respondent noted that it is not clear if a project must meet all of the aspects of R&D for it to

be included in the survey, or only one of these aspects. For example, if a project creates solutions that

are useful to others, but it is not novel and the outcome is certain, it is not clear if it should be

considered R&D on this survey.

Recommendations:

Include the instructional text entitled “What is Research and Development (R&D)?” on

the same screen as Question 3 (screener).

Use the original version of the “What is Research and Development (R&D)?” instructions

with adjustments made to the order and presentation of the information. The order

should be as follows:

o Definition and bulleted lists of what is R&D

o The bulleted list of what is not R&D

o The matrix that includes what is and is not R&D

o The instructions on reporting units

o The instructions on using estimates

o The matrix that classifies R&D from four activities

o The original screener question (i.e., yes/no questions)

Add more social science examples to help respondents identify their activities.



Resolutions:

NSF agrees with the recommendation to include the instructions and screener question

on the same screen in Centurion. Census will make the change prior to usability testing.

Recommendations regarding the order of instructional text and matrices have been

accepted.

NSF agrees with the recommendation to add more social science examples, where

applicable, and will provide examples to Census to add to the instructions.

Finding #9: R&D Activity (Screener) – Question 3 with Alternative

Overall, more respondents preferred the original version of this question, stating that the yes/no format

was easier to answer and that there was a significant amount of overlap between the questions in the

alternative version. While observing respondents reactions to the alternative to Question 3, it appeared

more cognitively burdensome for respondents, and required more attention and thinking than the

original version. Respondents said things like, “(the original version) is easy to answer and the second is

definitely more work,” and “performance is ok but funding is tricky.” In general, it seemed respondents

understood what the original question was requesting and most respondents were able to answer the

questions, or would know how to obtain the answers. Additionally, one respondent said that the term

“work toward” would include “anything that is at least 50% done.”

One respondent noted that the instructions here are redundant with the “What is R&D?” section. They

also said that being redundant is the best-case scenario, because if these instructions are not redundant,

“then it’s a problem because you’re defining it two different ways in the same document.”

Recommendations:

Use the original version of the Screener question (i.e., yes/no questions) for the survey.

Include the instructional text entitled “What is Research and Development (R&D)?” on

the same screen as Question 3 (screener).

Resolutions:

NSF agrees with the recommendation to use the original version of the Screener

question.

NSF agrees with the recommendation to include the instructions and screener question

on the same screen in Centurion. Census will make this change prior to usability testing.

Finding #10: Expenditures for R&D Performed Internally (Internal R&D) - Question 4

Overall, it appeared respondents understood the term “internal R&D” when responding to this question,

although a couple of respondents thought that Question 4 and Question 5 were the same question upon

first glance. Some respondents indicated that they would not report anything here. Respondents said



things like, “this is hard,” “I would say no to internal because there is no way to find it,” “there is no way

to capture this R&D,” “we don’t code time the way you’re asking,” and “don’t have hours on contract

versus policy.” Some respondents indicated they would have to reach out to “the programs” or other

employees within their agency to respond to this question. A few respondents in Oregon and

Washington spontaneously referenced the graphic that was presented to them at the beginning of the

interview while reviewing these questions.



Finding #11: Expenditures for Extramural R&D by Performance – Question 5

Similar to Finding #10, respondents who reviewed the graphic at the beginning of the interviews seemed

to understand the meaning of the term “external R&D.” Some respondents indicated that determining

how much was spent on R&D activities externally would be difficult. A number of respondents reported

they would have to refer to other employees in order to answer this question, saying things like, “it is up

to the program to be able to break it out by entity, by vendor, or project code,” “the programs keep

track of this.”

Some respondents indicated that the exclusion of “pass-through” funds could be difficult since they

consider all of this money to be “passed through” their agency to the external entity. Respondents said

things like, “excluding pass-through is a big decision point,” “’retain oversight’ is tricky because I am not

sure how much, it depends on how the contract is written,” “no administrative oversight or control is

tricky – what degree of oversight?” and “I’d interpret this as including pass-through funds.” One

respondent indicated the term that they use for this type of pass-through funding is “proviso” or

“provided solely for.” This respondent was concerned that by excluding this amount something would

be missing, saying, “who reports it?”



Finding #12: Total R&D Expenditures Summary - Question 6 with Alternative

While some respondents indicated that either of these options would be suitable, in general, more

respondents seemed to favor the grid format as a summary of their reported data. Respondents said

things like, “I like the chart. It’s more detail oriented.” One responded was confused by the text-based

version of this summary and said, “this is asking for another split of Federal, it seems redundant.”

When discussing their interaction with this section, many respondents indicated they would check their

responses to be sure the math was computed correctly saying things like, “I would look to see if the total

is right and keep going.” However, some respondents indicated they would simply assume our math

was correct and move on without reviewing. A few respondents were unsure if they would be able to



edit the data listed in the table. One respondent said, “’update as needed’ means go back to 4 and 5 to

do that because this is auto or pre-filled.”

A few respondents indicated that this is not a question, with one respondent saying, “number 6 is not

really a question, it is more of a confirmation. Does this look reasonable?” When asked about the

scrolling that would occur on this page most respondents did not seem concerned about scrolling. One

respondent said, “I would prefer to scroll, less than 3 scrolls is fine.”

Recommendations:

Use the table format of this question on the questionnaire to summarize Questions 4

and 5.

Consider evaluating the functionality of the table during usability testing (e.g., can

Centurion update a table that appears on the same page as questions? If not, consider

moving the summary table to its own page.)

Evaluate respondent reactions to scrolling through these items during usability testing.

Evaluate if respondents check responses within the table during usability testing (e.g.,

do they check for errors and if so do they make changes within the table, within the

question, or not at all.)

Consider referring to this as a summary table and not a separate question, and evaluate


Resolutions:

NSF agrees with the recommendation to use the table format on the questionnaire to

summarize Questions 4 and 5. Census will apply this to Centurion before usability

testing.

NSF will discuss the options for functionality of this table with the Centurion staff prior

to usability testing. If it is not possible to update the table in “real time”, it is possible

this table will be presented on a separate page and respondents will be asked to review

prior to proceeding. A decision will be made after discussing options with Centurion

staff.

Census and NSF will evaluate respondent reactions to scrolling during usability testing.

Census and NSF will evaluate respondents’ use of the table during usability testing.

NSF will develop alternative text ahead of usability testing to identify the matrix as a

summary of the agency’s R&D.



Finding #13: Total FY 2015 Internal R&D Character of Work by Federal vs. Non-Federal–

Question 9 – with Alternatives

Similar to findings from the previous round of testing, most respondents had difficulty splitting their

R&D by character of work. For most respondents, the breakout of “basic research,” “applied research,”

and “experimental development” is not a meaningful distinction. Moreover, several respondents

indicated that this would not be a breakout that would exist in their records and that coming up with

this breakout would be challenging and time consuming, saying things like, “to me research is research,”

and “we don’t have it broken out this way.” One respondent became frustrated after reading the

question and said, “I’d throw up my hands. Really, who cares? Especially at our dollar level. Do you

really need to know that?”

Some respondents described that they would estimate the breakouts for this question saying things like,

“zero basic, almost all applied, and some in experimental development,” and “most expenditures in

applied, some in experimental development (because basic would be out of scope for their contracts).”

Other respondents indicated they would spend time trying to locate these numbers, saying things such

as, “I have to apply my contracts back to these definitions and read the fine print,” and “I would have to

look under the project to see what category it falls into.”

Some respondents indicated they would read through the definitions at the bottom of the page before

reading the instructions, saying things like, “I skipped the boxes and went to the definitions toward the

bottom that made more sense to me,” “I’d put the definitions first,” “definitions are more easy to

understand, the examples are more abstract because they don’t relate (to my agency),” and “I had to

get through the examples before I saw the definition. Put the examples behind the questions.” Some

respondents did not notice the definitions at the bottom of the page at all, even as they were looking at

that part of the page, and focused only on the examples.

Some respondents would expect others would be better suited to answering these questions saying,

“budget isn’t the logical person but we don’t have a good candidate,” “certainly program managers

would answer this,” and “I’d ask the program people.”

In general, respondents did not realize that this question was asking for only internal R&D expenditures.

Several respondents who stated they would not report anything in Question 4 went on to describe how

difficult this question was.

Additionally, respondents did not understand this question was asking them to report only those

expenditures within fiscal year 2015. These respondents did not immediately recognize how the

response boxes for this question were different from previous questions within the survey. When

probed, several respondents indicated they would be reporting for both fiscal years in this question.

One respondent indicated adding an alert to indicate this is asking for only FY 2015 would be helpful.

Others said things like, “bold that this is for fiscal year 2015 only,” and “why is this just one year when

everything else is two years?”



When presented with all three versions of response boxes for this question, respondents from the

Oregon/Washington trip indicated a slight preference for the item that categorized the responses by

year first and then had options for Federal and non-Federal expenditures; California respondents were

near unanimous in preferring the version that did not have any boxes for FY 2014. One respondent in

Oregon suggested breaking by FY and then stacking the Federal and non-Federal boxes.

Most respondents were unsure what the shaded gray boxes indicated, and several said that this would

be confusing or frustrating. One respondent noted that if he were to print out the version that had FY

2014 and then try to pull the data based on reading the printed copy, he would have no idea that we did

not want FY 2014 data, and would have wasted his time pulling data that we did not want.

Two respondents in California were asked how much of an increase in burden it would be to ask for FY

2014 on this question. One respondent said that it would nearly double the burden of the survey,

because going through all of their projects and classifying each by character of work would be the most

burdensome part, so this would essentially be asking them to double that work. However, the other

respondent said that this would add almost no burden, because the hard part was simply in identifying

the projects that should even be considered R&D, and they would have already done that for the

previous questions on the survey that ask for FY 2014.

Recommendations:

There are a number of reasons to believe the accuracy of responses to this question is

compromised due to the number of issues that have presented over two rounds of

cognitive testing. Removing this question would be the best alternative.

Consider encouraging respondents to only provide estimates for this question to avoid

having them attempt highly-scientific and time-consuming methods to classify their

projects in this way (e.g., with instructive text such as “using your best estimate only”).

To maintain consistency across all questions on the questionnaire, use the option that

presents the response boxes with the two gray (non-functioning) FY 2014 boxes for

Federal and non-Federal first followed by the two functioning, white FY 2015 boxes for

Federal and Non-Federal.

Program skip patterns into the online questionnaire to be sure respondents who answer

“0” to Question 4 are not asked this question, as this is a sub-set of Question 4 (e.g.,

internal R&D expenditures).

Move the examples below the question so that respondents can view them if they want

to. Consider adding links from “basic research,” “applied research,” and “experimental

development” to the examples below.

Emphasize further that this is for internal R&D only and for FY 2015 only. Currently,

these instructions tend to blend in with the rest of the instruction.



Continue to investigate respondent understanding of the gray boxes during usability

testing. Also, consider adding “not applicable” (or other suitable language) to each gray

box to provide a visual indication that FY 2014 data is not being collected.

Continue to evaluate the amount of burden it would require to answer this question for

FY 2014 as well.

Resolutions:

Given the NSF needs for the National Patterns of R&D and the SGRD as an input to that

report, removing this question is not a realistic option for NSF at this time.

NSF agrees with the recommendation to incorporate text like “using your best estimate

only” to encourage estimation into this question prior to usability testing.

NSF agrees with the recommendation to use the option that mirrors the response field

layout used in prior questions: present the two gray (non-functioning) FY 2014 response

(i.e., Federal and non-Federal) on the left and the two functioning, white FY 2015

response boxes on the right (i.e., Federal and Non-Federal).

NSF will review the skip patterns with Census to ensure they are correct throughout the

survey.

NSF will move the example table to appear below the response boxes for this question.

Census will add links within each question that link to the examples for “basic research,”

“applied research,” and “experimental development” below.

NSF agrees with the recommendation to add “n/a” or “not applicable” to each of the

gray boxes in attempt to emphasize and visually indicate that FY 2014 data is not being

collected. Census will implement this change prior to usability testing.

During usability testing Census and NSF will investigate respondent understanding of the

gray boxes and the use of “n/a” or “not applicable”.

Census and NSF will continue to evaluate the amount of burden it would require to

answer this question.

Finding #14: Extramural R&D by Performer - Type – Question 10

For the most part, respondents were able to report the information requested in this question, although

it was sometimes difficult to determine if respondents understood this question was specifically for

“external” R&D expenditures. Overall respondents preferred the use of the term “type of entity” over

the use of the word “performer” within the title and text of this question. Respondents said things like,

“performer sounds artsy,” “by performer seems casual to me,” “I saw performer previously but it didn’t

make any sense,” “I don’t know what performer is,” and “performer is more like an individual person.”

A few respondents indicated they would like to report additional information in Question 10C saying,

“there is no place to put Federal government. It doesn’t fit in any so I’d put it in other,” and “add a

bullet to capture everything else.”



Recommendations:

Replace the term “performer” with the phrase “type of entity” within this question. If

applicable, replace this term throughout the questionnaire.

Consider adding “Federal government” to the description of “other entities” in 10C.

Resolutions:

NSF agrees with the recommendation to replace “performer” with “type of entity” for

this question. Census will implement this change prior to usability testing.

The recommendation to add “Federal government” to the description of “other

entities” in Question 10c will not be taken. In general, states rarely transfer money to

the Federal government. This would rarely apply for this survey and could cause

confusion.

Findings and Recommendations on the Graphic Based on findings from round one, we decided to create a graphic to represent the flow of R&D funding.

Due to the limited amount of time between rounds one and two and the time needed to create and edit

the graphic, we were only available to present the draft during interviews in Oregon and Washington.

This section will only be referring to 10 of the 17 interviews during this round of testing.

Finding #15: Interpretation of the Graphic

For the most part, respondents seemed to be able to describe the content within this graphic.

Respondents described the graphic by saying things like, “pretty standard way the Feds think, a standard

flow chart.” One respondent did begin to interpret the image incorrectly by thinking of “inputs and

outputs” but as they worked through the graphic realize it was actually, “money flow and not who

benefits.” Another respondent indicated, “wasn’t sure if this was budget or expenses.”

In general, it seemed respondents had a better understanding of the terms “internal” and “external”

R&D based on their interpretation of the graphic. One respondent said, “internal and your agency,

that’s clear to me.” Although, it was unclear during testing whether respondents understood that

internal R&D would include salaries and wages spent to manage external R&D. One respondent felt that

the box labeled “Your Agency’s Total R&D” meant “total funding stream.” Similarly, one respondent said

that in the title the term “monetary flow makes me think of laundering money.”

Recommendations:

Further evaluate respondent understanding of the graphic during usability testing.

Change the title of the graphic to “R&D Funding Structures.”

Resolutions:



The understanding of the graphic will be tested further during usability testing.

NSF will consider some alternative titles for the graphic.

Finding #16: Focus on the Images versus the Text

Although some respondents indicated they would look at the images and text together to gain an

understanding of the graphic, several respondents seemed focus more on the text within the graphic.

Respondents said things like, “I was focused on the words I guess” and “I looked at both, I’m a visual

person.” Several respondents indicated the images would have no meaning without the text to

accompany them.



Finding #17: Image Descriptions

For the most part, respondents seemed to think the images that were used throughout the graphic were

appropriate. One respondent said, “I like lower right box. That was the quickest route through the

graphic. Showing the microscope I thought this is people doing R&D. It fit well.” Although some

respondents thought there were some icons that could be modified, several respondents indicated they

would not know of any other images that would be more suitable.

When probed on the icon labeled “your agency,” respondents said things like, “I can’t think of a better

image” and “I do appreciate the agency is made of people.” When probed on the icon labeled,

“companies and individuals” most respondents felt the current image was appropriate and would not

prefer the use of gears as an icon for this description.

A few respondents spontaneously suggested that there were more male icons than female icons saying,

“they’re all wearing neck ties, it doesn’t seem inclusive” and “I want one of them to be a girl.”

Recommendation: Add some female icons to the graphic.

Resolution:

NSF agrees with the recommendation to add female icons to the graphic and will attempt to implement this recommendation prior to usability testing.

NSF also had some additional considerations for the graphic that were not based on the usability testing. These changes include:

o Make all of the icons the same color (e.g., black or gray, to help focus attention on the colors used for the lines).

o In the "Other funding sources" box, change the gears icon to the two-person icon used for "companies and individuals" below.



o In the "External" box, switch the order of "Companies and Individuals" and "Other Entities Performing R&D."

o Change the icon for "Other Entities Performing R&D" to the government and non-profit icons used in the "Other funding sources" box.

o To assist color-blind individuals, or people who are printing the graphic on B&W printer, consider using a visual device other than color to distinguish among the three sources. One way to do this would be to use three different line styles (e.g., solid, dashed, dotted). There may be value in using line style + color to make the distinction, but input from the Center for New Media and Promotions (the area working on the graphic) would be appreciated.

Finding #18: Placement and Use of the Graphic

Some respondents were unsure where this graphic would be the most useful to them while completing

the survey. A few respondents indicated this graphic would be most useful presented at the beginning

of the questionnaire saying things like “near the front of the document and label it as a conceptual

framework,” and “sign in or intro page,” and “cover memo and graphic.” During testing, several

respondents mentioned the graphic while discussing the content for Question 4 and Question 5, but

none of the respondents indicated they would actually refer back to this graphic while completing the

survey. One respondent said, “If it wasn’t shown from the get-go I wouldn’t look for it.”

Recommendation: Conduct further evaluation of the best location to present this graphic during

usability testing.

Resolution: Additional testing regarding the placement of the graphic will be conducted during

usability testing. There will likely be issues regarding the placement of the graphic. One option

is to include it on the flyer with definitions on one side and the graphic on the other, but will

require some additional conversations. Additional consideration may be needed given that the

graphic may be seen most on the Business Help Site as guidance.

Finding #19: Miscellaneous Findings

One respondent noticed a typographical error in the text. The graphic is missing a “)” in the “Other

Funding Sources” section.

A few respondents indicated they would consider other governments to be included in the external

group. One respondent said, “In the other group you may want to have ‘other government’.” Another

respondent mentioned including “state government” within this category.

Recommendations:

Add the missing “)” to the description of the group labeled “Other Funding Sources”.

Consider adding “other governments” to the group of other external R&D entities.

Resolutions:



The missing “)” will be added to the “Other Funding Sources” description.

NSF agrees with the recommendation to add “other state and local governments” to the

Other funding sources on the graphic.



About the Data Collection Methods and Research Branch (DCMR) The Data Collection Methods and Research Branch (DCMR) in the Economic Statistical Methods Division

(ESMD) assists Economic Directorate survey program areas and other governmental agencies with

research associated with the behavioral aspects of survey response and data collection. The mission of

DCMR is to improve data quality in surveys while reducing survey nonresponse and respondent burden.

This mission is achieved by:

Conducting expert reviews, cognitive pretesting, site visits and usability testing, along with post-

collection evaluation methods, to assess the effectiveness and efficiency of the data collection

instruments and associated materials.

Assisting program areas with the development and use of nonresponse reduction methods and

contact strategies.

Conducting empirical research to help better understand behavioral aspects of survey response,

with the aim of identifying areas for further improvement as well as evaluating the effectiveness

of qualitative research.

For more information on how DCMR can assist your economic survey program area or agency, please

visit the DCMR intranet site or contact the staff chief, Amy Anderson Riemer.

http://cww.census.gov/econ/oepi/RIRS/

mailto:[email protected]



Description of Cognitive Interviewing Techniques Respondents were asked to review the questionnaire and go through the thought process of answering each question (given the amount of research and correspondence it typically takes to complete the survey, it would not have been feasible for most respondents to attempt to provide actual answers). After respondents completed each section, they were asked protocol questions covering every item on the form and other features of interest (e.g., instructions, section headers, etc.). The strategy for probing specific responses involves non-leading, open-ended questions (e.g., “What does that question mean to you?”, “How did you go about answering it?” – see Appendix A for the full interview protocol). Initial questions are followed up with additional questions that probe respondents’ answers for more detail, until a sufficient understanding of question interpretations and response strategies is achieved and any apparent or potential problems are identified and their causes determined. For items with findings of “no problems observed,” we concluded that there were no indications of problems because respondents demonstrated the following types of behaviors: Paraphrased the question without distorting its meaning; Verbalized qualified answers or other contextual information congruent with intended meaning of question.



Appendix A: Draft Questionnaire for Cognitive Testing – Round two

FY 2014 and FY 2015 Survey of State Government Research and Development

The FY 2014 and FY 2015 Survey of State Government Research and Development (R&D)

measures state government funding of R&D. Responses from your state are used in

conjunction with other data to estimate national totals for R&D.

For questions, please contact:

Lisa McNelis

U.S. Census Bureau

(800) 622-6193

[email protected]

UserID:

Password:

OMB Burden Notice

mailto:[email protected]



Question 1 Review Agency Contact Information

Please review the contact information below and update if needed. (*required fields)

Primary Respondent

Your Name*

Title*

Agency Name*

Street Address Line 1


City*

State*

Telephone*

Email*

Additional Respondent (if applicable)

Your Name*

Title*

Agency Name*



City*

State*

Telephone*



Your Name*

Email*

Save and Continue



Question 2 Agency Fiscal Year

2a. When did your agency’s Fiscal Year 2013-14 end?

Note: For the rest of this survey, we will refer to this as Fiscal Year as FY 2014.

2014

MM DD YYYY

2b. When did your agency’s Fiscal Year 2014-15 end?

Note: For the rest of this survey, we will refer to this as Fiscal Year as FY 2015.

2015

MM DD YYYY

Save and Continue



What is Research and Development (R&D)?

Research and development (R&D) is creative work conducted systematically to extend

scientific knowledge, or devise new applications, which may include materials, products,

devices, processes, systems, methods, or services.

What makes it R&D?

When it is R&D When it is NOT R&D

R&D is novel.

It increases our

knowledge of the

subject.

It hasn’t been done

before.

You are testing blood

samples as part of a

research project to find out

the side effects of a new

cancer treatment.

You are collecting information

from samples of patients to

estimate the incidence of

chicken pox in the state’s

population. (You are using a

standard approach to

estimate the spread of

chicken pox.)

R&D creates solutions

useful to others.

Others might benefit

from the findings.

The findings can be

generalized to other

situations and

locations.

You are testing a pavement

on your highways that is

currently used only at

airports. Other states will

want the results.

You are testing pavement on

your state’s highways to

estimate how much you need

to budget for pavement

replacement over the next

five years. (Other states will

not benefit from your specific

state information.)

The outcome of R&D is

uncertain.

The solution isn’t

obvious to an expert

in that field.

Your research involves

monitoring streams to

determine whether a new

program is increasing the

population of a particular

type of fish.

You are monitoring streams

as part of plan to implement

long-term monitoring for a

particular type of fish. (The

monitoring plan has already

been tested and you are

certain of the quality of the

plan.)



Examples of classifying R&D in four situations

Activity It IS R&D

It is NOT R&D

Technical assistance You hire a technical

consultant to test the

disease resistance for the

new fish species you are

developing. (The assistance

addresses the uncertainty of

the science/technology

aspects of the product or

service.)

You hire a technical

consultant to help you

design the graphic design

for the package to ship your

new fish species when you

begin offering it for sale to

other states. (The assistance

addresses the uncertainty of

the marketing/production

aspects of the product or

service.)

Help for new businesses You provide funding to new

businesses to help them

with the costs of building

prototypes of products they

are developing.

You provide funding for new

technology companies to

help them acquire basic

skills to market their new

products.

Consulting You use a consultant to plan

testing of a highway

pavement material that your

transportation research

center is trying to develop.

You use a consultant to help

you secure health and safety

approval for your new

pavement material.

Developing a product from

your research

You hire a university

research center to test a

new type of grass you

developed to test whether it

will survive actual conditions

along the coastline.

You hire a law firm to help

you with the process for

patenting the new grass that

you developed.

Reporting unit

The reporting unit is your department, agency, commission, public authority (herein

referred to as agency) including all divisions and offices regardless of location that either

perform or fund R&D.



Estimates are acceptable

Please report all items to the best of your ability.



What is Research and Development (R&D)?




What is R&D?

R&D is a new way of thinking about a problem

o It has not been done before

o It may produce findings that could be published in academic journals

o It includes ideas that could be patented

R&D increases our knowledge of the subject

o Helps create new products or applications

R&D creates solutions that others may find useful

o Findings can be generalized to other situations and locations

R&D outcomes are uncertain (because it’s never been done before)

o Solutions are not always obvious or expected

What is NOT R&D?

o Construction and acquisition of land and facilities used primarily for R&D

o Fixed equipment used primarily for R&D

o Program planning and evaluation

o Business development services for new companies

o Commercialization (includes promoting/producing the products/services from

R&D projects)

o Economic/policy/feasibility studies

o General patient services

o Information systems

o Management studies

o Marketing of products/services

o Market research or analysis

o Routine data collection/dissemination

o Routine monitoring/testing

o Strategic planning

o Technology transfer




You are testing blood samples as part of a

research project to find out the side

effects of a new cancer treatment.

You are collecting information from

samples of patients to estimate the

incidence of chicken pox in the state’s

population. (You are using a standard

approach to estimate the spread of

chicken pox.)

You are testing a pavement on your

highways that is currently used only at

airports. Other states will want the

results.

You are testing pavement on your state’s

highways to estimate how much you need

to budget for pavement replacement over

the next five years. (Other states will not

benefit from your specific state

information.)

Your research involves monitoring streams

to determine whether a new program is

increasing the population of a particular

type of fish.

You are monitoring streams as part of

plan to implement long-term monitoring

for a particular type of fish. (The

monitoring plan has already been tested

and you are certain of the quality of the

plan.)

You hire a technical consultant to test the

disease resistance for the new fish species

you are developing. (The assistance

addresses the uncertainty of the

science/technology aspects of the product

or service.)

You hire a technical consultant to help you

design the graphic design for the package

to ship your new fish species when you

begin offering it for sale to other states.

(The assistance addresses the uncertainty

of the marketing/production aspects of

the product or service.)

You provide funding to new businesses to

help them with the costs of building

prototypes of products they are

developing.

You provide funding for new technology

companies to help them acquire basic

skills to market their new products.

You use a consultant to plan testing of a

highway pavement material that your

transportation research center is trying to

develop.

You use a consultant to help you secure

health and safety approval for your new

pavement material.




You hire a university research center to

test a new type of grass you developed to

test whether it will survive actual

conditions along the coastline.

You hire a law firm to help you with the

process for patenting the new grass that

you developed.

Reporting unit

The reporting unit is your department, agency, commission, public authority (herein

referred to as agency) including all divisions and offices regardless of location that either

perform or fund R&D.

Estimates are acceptable

Please report all items to the best of your ability.



Question 3 R&D Activity in FY 2014 and FY 2015




R&D includes:

o New products or applications

o Research findings that might benefit others

o Findings that can be generalized to other situations and locations

o Ideas that could be patented

o Findings that could be published in academic journals

R&D excludes:



o Commercialization (e.g., promoting/producing the products/services from R&D

projects)











o Construction and acquisition of land, facilities, or fixed equipment used primarily

for R&D (this is reported separately in Question 11)

Did your agency do any of the following during FY 2014 or FY 2015?

3a. During FY 2014 OR FY 2015, did your agency have a

division, branch, or office devoted to R&D?…………………………… Yes

No

3b. During FY 2014 OR FY 2015, did your agency perform

R&D activities using agency staff? ………………………………………… Yes

No

3c. During FY 2014 OR FY 2015, did your agency fund R&D

at another state or local government? …………………………………… Yes

No

3d. During FY 2014 OR FY 2015, did your agency fund

universities or other nonprofit organizations to perform

R&D activities? …………………………………………………………………………… Yes

No





No

3e. During FY 2014 OR FY 2015, did your agency fund

companies or individuals to perform R&D activities? …………

Yes

No

3f. During FY 2014 OR FY 2015, did your agency perform

or fund other R&D activities not specified above? …………. Yes

No





No

Question 3 R&D Activity in FY 2014 and FY 2015




R&D excludes:



o Commercialization (includes promoting/producing the products/services from

R&D projects)











o Construction and acquisition of land, facilities, or fixed equipment used primarily

for R&D (this is reported separately in Question 11)

During FY 2014 or FY 2015, which of the following best describes each of your

agency’s research and development (R&D) activities? (Select all that apply.)

3a. Research and development (R&D) work toward creating new findings,

products, or applications?

Performed it within your agency

Funded it (a separate organization performed the work)

Not applicable



3a. Research and development (R&D) work toward creating new findings,

products, or applications?

3b. R&D work toward producing research findings that might benefit others?



Not applicable

3c. R&D work toward identifying findings that can be generalized to other

situations or locations?



Not applicable

3d. R&D work toward creating any ideas that were (or could be) patented?



Not applicable

3e. R&D work that might lead to findings that could be published in academic

journals or similar publications?



Not applicable



3f. Other R&D work not described above?

Please specify: ___________________________________________________



Not applicable

[Note: Questions 4, 5 and 6 will appear on the same webpage.]



Question 4 Expenditures for R&D Performed Internally (Internal R&D)

During FY 2014 and FY 2015, what were the total expenditures for internal R&D

performance from federal funds, state funds, or other funds?

Include

R&D performed by your agency’s employees

Services performed by others in support of an agency R&D project (e.g. lab testing)

Administration and management of R&D performed external to your agency (e.g.,

administration or R&D-related contracts or grant recipients)

Salaries

Benefits

Supplies and movable equipment for R&D

Travel

Indirect costs

Purchased services

Exclude

R&D performed by higher education institutions

R&D performed by businesses and individuals

R&D performed by non-profit organizations

R&D performed by other governments

Construction and acquisition of land or facilities used primarily for R&D (report in

Question 11)

Fixed equipment used primarily for R&D (report in Question 11)

4a. Federal Funds used for internal R&D

For example:

Grants

Awards

Contracts FY 2014 FY 2015

Appropriations from the United States Government ………… $ $

4b. State Funds used for internal R&D

For example:

Appropriations from the state legislature

General funds

Restricted funds

Revenue funds

State grants

Tobacco settlement funds

Lottery proceeds

Funds from other agencies within your state ……………………… $ $



4a. Federal Funds used for internal R&D

For example:

4c. Other Sources used for internal R&D

For example:

Funding from other state governments

Funding from state/regional partnership agreements

Funding from local or tribal governments

Funding from nonprofit organizations

Funding from business ………………………………………………………… $ $

These are your total expenditures for internal R&D:

(sum of 4a through 4c) ……………………………………………………………

$ Auto Populate $ Auto Populate



Question 5 Expenditures for R&D Performed by External Entities (External R&D)

During FY 2014 and FY 2015, what were the total expenditures for external R&D

performance from federal funds, state funds, and other funds?

Include

R&D funded by your agency and performed by:

o higher-education institutions

o agricultural experiment stations

o businesses and individuals

o non-profit organizations

o other state or local governments

Exclude

pass-through funding for which you have no administrative oversight or control

5a. Federal Funds used for external R&D

For example:

Grants

Awards

Contracts FY 2014 FY 2015

Appropriations from the United States Government ………… $ $

5b. State Funds used for external R&D

For example:

Appropriations from the state legislature

General funds

Restricted funds

Revenue funds

State grants

Tobacco settlement funds

Lottery proceeds

Funds from other agencies within your state ……………………… $ $

5c. Other Sources used for external R&D

For example:

Funding from other state governments

Funding from state/regional partnership agreements

Funding from local or tribal governments

Funding from nonprofit organizations

Funding from business ………………………………………………………… $ $

These are your total expenditures for external R&D:

(sum of 5a through 5c) ……………………………………………………………




Question 6 Summary of Reported R&D Expenditures

This is a sum of the reported total R&D expenditures for FY 2014 and FY 2015

reported in Question 4 and Question 5.

6a. Your agency’s total R&D expenditures (including all sources of

funds for both internal and external R&D) are: ………………………

FY 2014 FY 2015


This is a sum of the reported total R&D expenditures from federal funding sources

reported in Question 4A and Question 5A.

6b. Your total R&D expenditures from federal funding sources

are: ……………………………………………………………………………………………


Save and Continue



Question 6 Summary of Reported R&D Expenditures

Note: The following charts show the answers you reported in Question 4 and Question 5 above. You

may update the content that appears in the white boxes as needed.

FY 2014 FY 2015

Internal External Total by

Source Internal External Total by

Source

Federal $ Prefilled $ Prefilled $ Auto Federal $ Prefilled $ Prefilled $ Auto

State $ Prefilled $ Prefilled $ Auto State $ Prefilled $ Prefilled $ Auto

Other $ Prefilled $ Prefilled $ Auto Other $ Prefilled $ Prefilled $ Auto

Total by

Performer

Total by

Performer

$ Auto $ Auto $ Auto $ Auto

TOTAL $ Auto TOTAL $ Auto



Question 7 Total R&D Expenditures by Function

Of your total R&D expenditures [insert amounts from 6a respectively], how much

were for each category below?

7a. Agriculture

For example:

Animal husbandry

Aquaculture

Crop management FY 2014 FY 2015

Food and commodities ………………………………………………………… $ $

7b. Energy

For example:

Alternative fuels (e.g., ethanol, biofuels)

Energy conservation/efficiency

Oil and gas

Other fossil fuels (e.g., coal, clean-coal, lignite)

Renewable energy (e.g., wind, solar)………………………………… $ $

7c. Environment and Natural Resources

For example:

Air and water quality

Fish, game, and wildlife

Forestry

Geological studies

Marine and aquatic environments

Parks and preserves

Soil and water conservation………………………………………………… $ $

7d. Health

For example:

Biomedical research

Mental health and addiction

Pharmaceuticals

Public health………………………………………………………………………… $ $

7e. Transportation

For example:

Aviation

Highways, roads, and bridges

Ports and waterways

Public transportation

Rail and freight…………………………………………………………………… $ $



7f. Other functions

For example:

Corrections and criminal justice

Education

Labor

Public safety

Social services ……………………………………………………………………… $ $

If Other, please specify:

Total (sum of 7a through 7f) ………………………………………………… Auto Populate Auto Populate



Question 8 Funds Received from Federal Agencies

8. Of the total R&D expenditures from federal funds [shown in Question 6b], what

amount came from the following agencies? Please add any agencies that are not

already listed. If none, please report $0.

Federal Agency

FY 2014 FY 2015

Department of Agriculture $ $

Department of Commerce $ $

Department of Defense $ $

Department of Education $ $

Department of Energy $ $

Department of Health and Human Services $ $

Department of Homeland Security $ $

Department of Housing and Urban Development $ $

Department of the Interior $ $

Department of Justice $ $

Department of Labor $ $

Department of State $ $

Department of Transportation $ $

Department of the Treasury $ $

Department of Veterans Affairs $ $

Environmental Protection Agency $ $

National Aeronautics and Space Administration $ $

National Science Foundation $ $

Other: [Please specify] $ $

Total (sum of all above) …………………………………………… $ Auto Populate $ Auto Populate



Question 9 FY 2015 Expenditures for R&D Performed Internally by Character of Work

Examples

Basic Research Applied Research

Experimental

Development

You are studying the

properties of blood to

determine what affects

coagulation.

You are conducting research

on how a new chicken pox

vaccine affects blood

coagulation.

You are testing a newly

developed chicken pox

vaccine with various ages of

school children before

implementing it statewide.


properties of molecules

under various heat and cold

conditions.


on the properties of

particular substances under

various heat and cold

conditions with the objective

of finding longer lasting

components for pavement.


developed pavement under

various types of heat and

cold conditions prior to

using it on your state’s

highways.

You are studying the heart

chambers of various fish

species.

You are examining various

levels of a toxic substance

to determine the maximum

safe level for fish in a

stream.

You are designing a new

system for monitoring a

stream that will try out the

results of your recent

research in a real world

situation.

You are studying the effects

of various strategies for

teaching math in order to

understand which is the

most effective.


implementation of a specific

math curriculum to

determine what factors lead

to successful

implementation by teachers.

You are using research in

the field to develop

education software and

support tools for math

curriculum.

In FY 2015 only, what amount of your agency’s internal R&D expenditures [insert

Question 4 FY 2015 total] were for basic research, applied research, and

experimental development, from federal and non-federal sources?

9a. Basic Research

Research undertaken primarily to

acquire new knowledge without any

particular application or use in mind.

Federal Funds Non-federal Funds

$ $

9b. Applied Research



9a. Basic Research

Research conducted to gain the

knowledge or understanding to meet

a specific, recognized need. ……………

$ $

9c. Experimental Development

The systematic use of the knowledge

or understanding gained from

research directed towards the

production of useful materials,

devices, systems, or methods,

including the design and development

of prototypes and processes. ……………

$ $

FY 2015 Federal FY 2015 Non-federal

Total (auto calculate): ………………… $ $



Question 9

FY 2015 Expenditures for R&D Performed Internally by Character of Work

Examples

Basic Research Applied Research Experimental

Development




coagulation.




coagulation.









conditions.













highways.



species.





stream.






situation.





most effective.



math curriculum to


to successful






curriculum.

What amount of your FY 2015 internal R&D expenditures [insert Question 4 FY

2015 total] were for basic research, applied research, and experimental

development, from federal and non-federal sources?

9a. Basic Research FY 2014 FY 2015




Federal

Non-Federal

Federal

Non-Federal

$ $ $ $




9a. Basic Research FY 2014 FY 2015




$ $ $ $









$ $ $ $

Federal

Non-Federal

Federal

Non-Federal

Total (auto calculate): ………………… $ $ $ $



Question 9

FY 2015 Expenditures for R&D Performed Internally by Character of Work

Examples

Basic Research Applied Research Experimental

Development




coagulation.




coagulation.









conditions.













highways.



species.





stream.






situation.





most effective.



math curriculum to


to successful






curriculum.

What amount of your FY 2015 internal R&D expenditures [insert Question 4 FY

2015 total] were for basic research, applied research, and experimental

development, from federal and non-federal sources?

9a. Basic Research Federal Non-Federal




FY 2014

FY 2015

FY 2014

FY 2015

$ $ $ $




9a. Basic Research Federal Non-Federal




$ $ $ $









$ $ $ $

FY 2014

FY 2015

FY 2014

FY 2015

Total (auto calculate): ………………… $ $ $ $



Question 10 Expenditures for R&D Performed Externally by Performer

10a. Of the total expenditures for external R&D performance

reported in Question 5, [amounts], how much was to

higher-education institutions? ………………………………………………

FY 2014 FY 2015

$ $

10b. Of the total expenditures for external R&D performance


businesses or individuals? ……………………………………………………

$ $

10c. Of the total expenditures for external R&D performance


Other entities?

For example:

Other state governments

Local governments

Non-profit organizations …………………………………………………

$ $

If other, please specify:

Total (sum of 10a through 10c) ……………………………………………… $ $



Question 10 Expenditures for R&D Performed Externally by Type of Entity

10a. Of the total expenditures for external R&D performance


higher-education institutions? ………………………………………………

FY 2014 FY 2015

$ $

10b. Of the total expenditures for external R&D performance


businesses or individuals? ……………………………………………………

$ $

10c. Of the total expenditures for external R&D performance


Other entities?

For example:

Other state governments

Local governments

Non-profit organizations …………………………………………………

$ $

If other, please specify:

Total (sum of 10a through 10c) ……………………………………………… $ $

Question 11 R&D Expenditures for Land and Facilities



This question asks about land and facilities (including fixed equipment, such as reactors,

wind tunnels, and particle accelerators) used primarily for R&D. It includes the acquisition

of, construction of, and major repairs or alterations to structures, works, equipment,

facilities, or land for use in R&D activities.

Construction and acquisition of land and facilities used primarily for R&D includes major

costs for construction and purchase of buildings to be primarily used as R&D facilities.

Include:

Only expenditures related to the R&D activities of your agency.

Exclude:

Expendable or movable equipment (e.g., spectrometers, microscopes).

Office furniture and equipment.

Costs of pre-design studies (e.g., those undertaken before commitment to a specific

facility).

NOTE: The expenditures reported on this question are separate from the R&D expenditures

asked throughout the rest of the survey, and should not be included in the expenditures on

any other question in the survey.

11. During FY 2014 and FY 2015, what were your

agency’s R&D expenditures for construction and

acquisition of land and facilities (including fixed

equipment) used primarily for R&D? ……………………………

FY 2014 FY 2015

$ $



Appendix B: Draft Graphic



Appendix C: Cognitive Interviewing Protocol

Cognitive Interviewing Protocol

State Government R&D Round two

April 23, 2015

(This protocol is a guide – the questions presented here will not necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)

Research Objectives: Is the requested information available?

What records is the information stored in?

How does the respondent comprehend/interpret the terminology?

Are graphic images helpful in conveying the flow of money that we’re asking for?

What is the respondent burden for answering these questions?

Expected Length of Interview: 60-90 minutes

Materials Needed: 2-3 copies of the test questionnaire

2-3 copies of the alternative questions

Respondent’s answers to previous SGRD questionnaire (get from Census)

Consent form(s)

Digital recorder

Business cards

Introduction: Introductions:

o How long at agency, position/title and responsibilities

o Census/NSF observers

Meeting purpose:

o To understand the process used to answer the new R&D questions



o To get feedback on the questions – we are NOT testing the respondent, but we are

testing the questions (getting information about interpretations, impressions and

opinions, does the order of the questions make sense)

o No right or wrong answers – all feedback is useful!

Meeting structure:

o General and specific questions about the questions. Some questions may seem odd or

redundant because we are not accountants or R&D project managers, and because

we’re meeting with various government agencies that have different ways of keeping

records.

Permission to audio tape? Have respondent sign consent form.

General Probes: Can you tell me more about that?

In your own words, what was that question asking?

What terms do you use to describe this?

Can you tell me how you would arrive at that answer? What would you include? What would

you exclude? What records would you use?

Do your records contain this information? What do these records look like? What are they

called?

Would you use book figures, or would you have to estimate to report for this item?

Would you have to manipulate the book figures to get an answer? How?

Would you need help from other coworkers to get this data?

I want to make sure I understand, can you explain that again?

Reflect back on R’s answer and ask R to correct (“I want to make sure I have it right. I think you

said, “….?”)

Sample Probes for Question Alternatives: Was this version helpful or unhelpful?

o Why or why not?

o What was helpful/unhelpful about it?

What, if anything, could be changed to make this version more helpful for you?

Which version do you prefer? Why?

Could you tell me a little bit more about that?

Background Questions: Could you tell me a little about the activities of this government agency? What are your roles

and responsibilities?

What does “R&D” mean to you? Could you tell me a little more about that?

What types of R&D are conducted at this agency?



(If we cannot speak to R&D manager) Do you interact with R&D managers, scientists, or other

R&D personnel? Please tell me about your relationship with R&D personnel.

Are there staff solely dedicated to performing R&D? How about staff dedicated to managing

R&D projects?

Section 1: R&D Graphic At this time, I will be sharing a graphic that we have created that we are considering including with the

survey materials.

Graphic #1 – Inputs and Outputs of Your Agency’s R&D Expenditures What do you feel like this graphic is telling you?

What are your overall reactions to this page?

o How do you feel about seeing a graphic like this on a government survey?

What does the title “Your Agency’s Total R&D” mean to you?

o What are you including and excluding here?

(Repeat the following questions for each of the icons of interest: Other funding sources, your

agency, Companies and individuals)

o How do you interpret this image/icon?

o Based on the title, what type of image would you expect to see? (for companies and

Individuals – are gears or humans a better example)

Do you find these icons/images confusing?

o Would a text-based diagram be more or less confusing? How so?

What is the word “internal” indicating to you? How about “external”?

o Where would be the best location for these terms?

Where would you expect to see an image like this?

o We cannot put this directly on the page with the question. Knowing this, where would

you like to see it instead?

Section 2: R&D Survey At this time, I will be sharing the current draft of the survey instrument with you. This is a draft and will

eventually be used to create an electronic version of the form. Please bear with us if there are breaks in

the pages. I would like you to complete the form as you normally would. Please let me know if you

have to access your records or if you would have to consult someone else at any point. This is not a test

of ability or knowledge, and there are no right or wrong answers. As we work through the

questionnaire, I may ask you to review additional versions of certain questions to get your impressions

and opinions. Please be honest, this exercise is designed to help choose the format that would be the

best for your department/agency.



Please let me know if you have questions at any time and we will do our best to answer them for you.

Introduction & Login Screen Note any respondent reactions to this screen

Question 1 - Agency Contact Screen Note any respondent reactions to this screen

Question 2 – Agency Fiscal Year What would you report for Question 2A?

o Could you tell me a little more about how you arrived at your answer?

Could you explain the Note to me in your own words?

What does the gray box indicate to you?

What is R&D Screens What does ”R&D” mean to you?

o What types of terms do you use to describe this type of work at your agency?

What are your overall reactions to these pages?

What information are you looking for, if any?

What information stands out to you?

How would you use this information?

What is R&D Screens (alternative) If you were to choose a design from these two options, which would you choose?

o Could you tell me a little more about that?

o How did you come up with that selection?

Was version 1 helpful or unhelpful?



Is there any other format you would prefer?

Question 3 – R&D Activity in FY 2014 and FY 2015 What are your reactions to this page?

How would you go about answering this question?

Could you tell me in your own words what the include statements are telling you? What about

the exclude statements?

Could you go through and answer these questions for me?



o What records, if any, would you need to consult in order to answer these questions?

What time period were you considering?

Question 3 – R&D Activity in FY 2014 and FY 2015 (alternative) Could you go through and answer these questions for me?

o What records, if any, would you need to consult in order to answer these questions?

What does the phrase “work towards” mean to you?

o What stage or stages of the project would this be applicable to? (i.e., before a project

starts, projects in progress, completed projects, other)

Comparing the two sets of questions, which ones are easier for you to understand?

Which set of questions do you think gives us an accurate picture of your R&D activity?






Question 4 – Internal R&D Expenditures by Source of Funds What does this question mean to you?

o How did you go about answering this question?

o What records did you need to consult in order to answer this question?

o Did you need help from other government personnel to get this data?

What does the term “internal R&D performance” mean to you?

o Do you use this term?

o (If no) What term or terms do you use to describe this type of activity?

o Could you give me some examples of these type of expenditures?

What would you include here?

What would you exclude from this item?

How does this item relate to other questions on the form? Could you tell me a little more about

that?

Question 5 - External R&D Expenditures by Source of Funds How did you go about answering this question?

What does “total expenditures for external R&D performance from federal funds, state funds,

and other funds” mean to you?

o Do you have line items for this information?

(If yes) Where is this information located? What does this include?



(If no) Is it clear what this item would include? How difficult would it be to break

this out?

What are you including?

What are you excluding?

What records would you use to respond to Question 5a?


What records would you use to respond to Question 5b?


What records would you use to respond to Question 5c?


Are you including or excluding pass-through funds?

o (If yes) could you tell me a little more about that?

Question 6 – Summary of Total Reported R&D Expenditures In your own words, what are we trying to represent to you?

o What are your overall reactions to the layout of this item?

o Is the way this item is presented clear or unclear to you? Could you tell me a bit more

about that?

Thinking about the information on this page, what would you do at this point?

o Would you review these totals in any way?

(If yes) How would you go about doing that?

o Would you be able to cross check these values against any records or documents? If so,

which ones?

Would you have to estimate any of these totals? (If yes) please tell me more about this.

o How accurate do you think your answer is to this question?

Did you need help from other government personnel to get this data? Who would you seek help

from?

Question 6 – Summary of Total Reported R&D Expenditures (alternative) In your own words, what are we trying to represent to you?

o What are your overall reactions to the layout of this item?

o Is the way this item is presented clear or unclear to you? Could you tell me a bit more

about that?

Thinking about the information on this page, what would you do at this point?

o Would you review these totals in any way?

(If yes) How would you go about doing that?

o Would you be able to cross check these values against any records or documents? If so,

which ones?

Would you have to estimate any of these totals? (If yes) please tell me more about this.



o How accurate do you think your answer is to this question?

Did you need help from other government personnel to get this data? Who would you seek help

from?

If you were to choose a design from these two options, which would you choose?



Thinking about the records you would need to use to complete this question, which format

would be easier for you to report your data into?





Is there any other format you would prefer?

Question 7: Total R&D Expenditures by Function How did you go about answering this question?

Can you tell me what this question is asking for in your own words?

What records did you need to consult in order to answer this question?

o Did you have to estimate the answer for this question? Please tell me more about this.

o How accurate do you think your answers are to this question?

o Did you need help from other government personnel to get this data?

o How easy or difficult was it to determine the answer to this question?

Could you provide some feedback on the examples?

o Did the categories make sense to you?

Why or why not?

o Were any of them particularly helpful or not helpful to you?

Question 8 – Funds Received from Federal Agencies What does this question mean to you?

How did you go about answering this question?

o What did you include in in this answer?

o Is this information that you would have access to?

o Do you have the federal department/agency listed in your records?

(If yes) What records would you need to access?

(If no) Is it available elsewhere?

o How would you report or list other entities that do not appear on this list?

How easy or difficult would it be to collect and report?

o Would you know where each department/agency fits?

o What type of information would you have available? Agency name? Branch? Other?



Question 9 – FY 2015 Expenditures for Internal R&D by Character of Work What does this question mean to you?



What years are you reporting for? 2014, 2015 or both?

Could you explain to me what the examples at the beginning of this section in your own words?

o Which of these categories do you do? Could you give me some examples?

What did you include under “Basic Research”?

o What sort of projects would you include here? Can you give me some examples?

What did you include under “Applied Research”?


What did you include under “Development”?


Did you have any projects you were unsure where to report?

How did you determine which category your expenditures would be classified under?

How easy or difficult was this question to answer?

What time period or periods are you considering while answering this question?

Question 9 – FY 2015 Expenditures for Internal R&D by Character of Work

(alternatives 1 &2 ) What do the gray boxes indicate to you?

If you were to choose a design from these three options, which would you choose?



Thinking about the records you would need to use to complete this question, which format

would be easier for you to report your data into?






Is there any other format not presented here which you would prefer?

Question 10 – Expenditures for External R&D by Type of Entity What does this question mean to you?



What are you including in Question 10A



o How would you go about getting the total for “higher education institutions”?

What does that term mean to you?

o Is that total available in your records?

o What sort of details do you have in your records? In what ways can you split out this

total?

How would you go about getting the information for “businesses and individuals”?

o Would you have a separate line item in your records for this?

o (If no) What would you have to do to come up with a response here?

What would you include in the “Other entities” category?

o What types of details do you have in your records on these entities?

o How do you come up with an answer to this item?

Is there anything else we are missing here?

Question 10 – Expenditures for External R&D by Type of Entity Was version 1 helpful or unhelpful?


Which title do you prefer? Why?

Question 11 - R&D Facilities and Fixed Equipment What does this question mean to you?

Could you explain the note to me in your own words?

o Does that change how you think about the other questions on the survey? Why or why

not?

How would you go about answering this question?

o What records would you need to consult in order to answer this question?

o What would you include here?

o Was there anything you would exclude?

Could you give me some examples of large fixed equipment that your agency has (or would like

to have) that are dedicated for R&D work?

o Is there anything that you are considering but are not sure if it is dedicated for R&D

work or not? Could you tell me a bit more about that?

Section 3: Card Sort Activity (Only if there is time to complete) (For the card sort activity each respondent will be handed a stack of 10 cards (attached) that include

different examples of research and development activities. In addition to these cards, respondents will

also be presented with three sheets of paper that have the following categories written on them: 1)

Basic research; 2) Applied research; and 3) Development)



Examples for State Government R&D Card Sort Note: Each bulleted point will be a separate card.

Basic Research:

Study and document the movement and behaviors of wild deer populations through a

contaminated/restricted area into populated-residential areas. Study deer through radio collar

tracking, and random hunts to tranquilize deer for measurements and take blood samples for

lab analysis.

Study the structure and properties of a newly identified gray-shale formation that was

discovered while building a road. We have not seen this formation elsewhere in the state and do

not have data on its properties.

Issued a matching grant to a university hospital to study possible changes in autoimmune

receptors in women before, during, and after pregnancy.

Applied Research:

Experimenting with a specific bacterium for the control of invasive species of mussels in open

water such as rivers and lakes.

Study the roosting behaviors and densities of specific bat populations to identify the best type,

placement, and number of artificial bat-houses to better control insects in agricultural areas to

decrease use of man-made pesticides.

Study the porous capacity of a specific layer of sedimentary rock formations to understand if the

reservoirs serve as potentially suitable sites for carbon sequestration lease permits.

A special study was conducted with victims of domestic violence to assess current depression

and anxiety symptoms, self-reported health status, psychosocial functioning, and

medical/mental health treatment received prior to and post-trauma. The lead investigator is

looking to develop better treatment protocols.

Development:

Develop and test (in classrooms) a new methodology for teaching children with pervasive

developmental disabilities to read with improved comprehension.

Test the effectiveness of a specific bacterium to control black fly larvae. Earlier research showed

it to be both effective in controlling black fly larvae as well as environmentally safe.

Develop a new model in which Department of Health data and Department of Human Services’

(DHS) data can be used to identify possible predictors in a new methodology for referring cases

to DHS for evaluation of child abuse/neglect.

At this time, I would like to discuss one of the sections we discussed earlier in a bit more detail. I am

going to present you with a set of cards that contain examples of research and development activities. I



would like you to read each card and create a separate stack on each of these three sheets of paper.

Please create one stack for Basic research, one stack for Applied research and one stack for

Development. There are no right or wrong answers. Do you have any questions?

Respondents will be asked questions during the activity and after the sort is completed:

How did you arrive at your answer?

Could you tell me a little bit more about that?

Note: If R is having difficulty, probe on the process

o Could you tell me a little about what you are considering?

o What could we do to make this clearer?

Wrap-up: How long would you estimate it would take to complete the 11 questions we reviewed today?

How does that compare to the previous times you completed this survey?

How easy or difficult would it be to complete these questions – very difficult, somewhat

difficult, neither easy nor difficult, somewhat easy, or very easy?

Would there be anyone else that would help you respond to this form? Review the form? (If

yes) Who?

Do you have any other comments or suggestions?

That is all the questions I have for today. Thank you for your time and assistance!

Findings and Recommendations from Cognitive Testing for the … · 2016. 4. 4. · – Round Two...

Documents

Transcript of Findings and Recommendations from Cognitive Testing for the … · 2016. 4. 4. · – Round Two...