The

KEYSPOT Model

A home away from home An evaluation of the Philadelphia

Freedom Rings Partnership

Seeta Peña Gangadharan, PhDKistine Carolan, MSSKayshin Chan, MPH

2

Acknowledgments➜

Open Technology Institute

DirectorSascha Meinrath

Director of Field OperationsJoshua Breitbart

Evaluators for the Freedom Rings PartnershipSeeta Peña Gangadharan, PhDKistine Carolan, MSSKayshin Chan, MPH

Research and Technical Support for the Freedom Rings Partnership evaluation

Georgia Bullen Greta Byrum April GlaserDarby HickeyJosh King Chiehyu LiDan MeredithLeticia MirandaBincy Ninan-Moses Preston RheaHannah SassamanItir SonuparlakSeamus Tuohy

Freedom Rings Partnership Evaluation Working Group

City of Philadelphia Office of Innovation and TechnologyDrexel UniversityFree Librar y of PhiladelphiaMedia Mobilizing ProjectPeople’s Emergency CenterPhiladelphia Parks and RecreationPhiladelphia FIGHTPhiladelphia Opportunities Industrialization CenterRutgers UniversityUrban Affairs Coalition

IT Support for Workstation User Survey implementationCognis IT

Report graphic designJeff Ferzoco, linepointpath LLC

Creative Commons License info:https://creativecommons.org/licenses/by-nc-nd/3.0/

Thank you to the KE YSPOT par ticipant s and Webguides, who contributed time and shared their personal stories in focus groups; to KEYSPOT staff members who shared documentation and insights with us through meetings and interviews; and to staff members of allied organization who shared their helpful perspective from beyond the partnership. The authors also greatly appreciate the external reviewers who provided feedback on the report.

3

5 Executive Summar y 9 Introduction11 Background15 Histor y of the Freedom Rings Par tnership21 Evaluation Methodology31 Findings32 Progress On NTIA Deliverables 33 Q1a: Who par ticipates at KEYSPOTs?38 Q1b: What par ticipant needs are met

through KEYSPOTs?40 Q2: What are the ef fects of KEYSPOT access

and training on broadband adoption?43 Q3a: What are the ef fects of KEYSPOT access

and training on par ticipants’ employment status?

45 Q3b: What are the ef fects of KEYSPOT access and training on par ticipants’ educational attainment?

46 Q4: What are the ef fects of KEYSPOT access and training on community engagement?

48 Q5: In what ways has the FRP increased par tners’ capacity?

53 Q6: How well has the FRP functioned?

59 Discussion67 Conclusions70 Endnotes 72 Appendices

Table of Contents

4

Image: D

rexel Freedom Rings Partnership

5

Executive Summary

I n S e p t e m b e r 2 010 , b y m a n d a t e o f t h e A m e r i c a n Reinvestment and Recover y Act (AR R A), the National Telecommunications and Information Agency (NTIA) aw a rd e d a tot a l of $18 . 2 m i l l ion to a d i ve r s e g rou p of Ph i l a de lph ia or g a n i z a t ion s u nder t he B roa d ba nd Te c h n o l o g y O p p o r t u n i t i e s P r o g r a m ( B TO P) . A n additiona l $8.9 million in matching f unds provided by partners brought the total budget to $27.1 million. The group, later formalized as the Freedom R ings Partnership (FR P or “Partnership”), implemented a Public Computer C e n t e r ( P C C) g r a n t a n d a S u s t a i n a b l e B r o a d b a n d Adoption (SBA) grant under a single umbrella branded as the KEYSPOT Program. These two programs, led by a core group of eleven academic, nonprofit, and government i n s t i t u t i o n s , w o r k e d j o i n t l y t o e x p a n d c o m p u t e r and Internet access and training to vulnerable populations in Philadelphia.

Open Technolog y Institute (OTI) led the planning a nd i mplement at ion of t he pa r t ner-d r iven eva lu at ion f rom Ja nu a r y 2011 to Ja nu a r y 2013 . Th rou g h a need s assessment and shared decision-making process, partners identified a set of goals for the Partnership and measurable indicators towards those goals, moving their overall vision beyond the NTIA-prescribed deliverables. These goals address the ability of the FRP to reach Philadelphia’s most

underserved populations; the effects of the FRP programs on broadba nd adoption, jobs, educationa l atta inment, and communit y engagement; and the capacit y and well-functioning of the Freedom R ings Partnership to meet its shared goals. OTI then collaborated with partners to develop a mixed methods evaluation plan. Though this Report represents the culmination of the evaluation, it does not represent the final picture of the Partnership. This Report represents the findings from data collection end ing in Ja nua r y 2013, but the PCC a nd SBA gra nts officially end in June and August, respectively.

What the evaluation does show is a complex portrait of a n i n novat i ve d i g it a l i nclu sion prog ra m i n one of t he cou ntr y’s la rgest a nd poorest cit ies . The FR P ha s uniquely embedded public computer access and digital literacy programs within a net work of social ser vices and commu nit y-ba sed work . With this approach, the FR P has helped bring Philadelphians online and helped them harness broadband ’s positive socioeconomic ef fects. It ha s been most successf u l in cu ltivating socia l suppor t for new users interested in broadband, and applying an expanded, Partnership-developed definition of broadband adoption, which includes a range of activities related to digital l iteracy and technolog y use.

6

The key points from this evaluation include both participant and Partnership outcomes.

➜➜ SBA and PCC partners have partially met and/or exceeded their original training, awareness, and public computer center outputs, though not yet met their grant deliverables for new household and business broadband subscribers, or average number of public computer center users per week.

➜➜ K E Y S P O Ts p r e d o m i n a n t l y s e r v e A f r i c a n Americans in Philadelphia’s poorest neighborhoods, w h i l e s p e c i f i c p a r t n e r s s e r ve d i ve r s e t a r g e t popu lations, including the homeless, veterans, people recovering from substance abuse, youth, sen iors , En g l ish la n g u a ge lea r ners , a nd ot her vulnerable populations.

➜➜ K E YS P OTs s e r v e i n d i v i d u a l s w i t h v a r i o u s broadband needs, including those who: (1) lack funds to purchase home broadband or who do not have a computer at home; (2) may have tools at home but lack the confidence to use them; (3) do not have home Internet but feel broadband is highly relevant to their lives.

As of January 2013:

➜➜ Participants learn about K EYSPOTs through personal recommendation and on-site promotion of digital and non-digital services, via cross-referrals of sites’ existing health and human service programs and the KEYSPOT lab and training programs.

➜➜ KEYSPOTs provide critical access points to the Internet for participants, helping them overcome fear of technology and increase their digital literacy skills with the help of supportive frontline staff who create a safe, welcoming, and engaging space for new Internet and computer learners.

7

Because this Report only represents interim results of the FR P progress towards its goals, the authors recommend a longitud ina l study track ing a cohor t of pa r ticipa nts over time to docu ment those outcomes. Similarly, the long-term outcomes of Partnership hea lth can only be determined after the program has finished and a follow-up study conducted to see how the Partnership inf luenced each partner organization and their relationships w ith each other. O vera l l , t h is eva lu ation i l lu m inates a n impor t a nt bro a d b a nd a d o pt ion mo d e l of re a c h i n g t hos e mos t isolated from broadband via embedding computer centers

a nd t ra i n i n gs i n socia l ser v ice a nd com mu n it y-ba sed agencies—referred to in this Report as the K EYSPOT Model of Broadband Adoption. This model demonstrates how i nd iv idu a ls w ho recog n i ze t he cr it ic a l releva nce of d i g it a l tool s i nter a c t w it h t r u s ted so c ia l set t i n g s t hat prov ide a comfor table lea rning env iron ment a nd engaging staff who inspire and stimulate digital learning and broadband use, whether the use takes place in home or in public settings. The FR P's experience with BTOP re s u lt e d i n t he c re a t ion of a n i n nov a t i ve mo d e l for broadband adoption, bringing broadband and its benefits to Philadelphia’s most underser ved communities.

➜➜ Participants visit KEYSPOTs for a variety of reasons, most ly per t a i n i n g to work force de velopment activities and training.

➜➜ While participants do not initially visit KEYSPOTs for the purpose of community engagement, they discover a KEYSPOT community of learners, as well as participate in their own communities at KEYSPOTs, both on-site and online.

➜➜ The Freedom R ings Partnership has increased the IT capacit y of partner organizations; supported professiona l development of its digita l l iterac y trainers; and improved ser vice deliver y through e x t e n s i v e c o l l a b o r a t i o n i n s i d e a n d o u t s i d e the partnership.

➜➜ Partners unanimously agree about the importance o f c o n t i n u i n g d i g i t a l l i t e r a c y t r a i n i n g s a n d public computer centers in Philadelphia, but they are uncertain as to how and with whom to sustain the work.

➜➜ Though more research is needed to determine the impact of KEYSPOTs on participants’ employment status, educational attainment, and home broadband subscription status, this eva luation establishes that K EYSPOTs laid the foundation for future improvements in those three areas.

8

9

O p en Tec h nolog y I n s t it ute (OT I) pre sent s it s f i n a l su m mative Repor t of t he Freedom R ings Pa r tnersh ip ( “ F R P ” o r “ t h e Pa r t n e r s h i p ” ) . T h e R e p o r t a s s e s s e s t he Pa r t ner sh ip's pro g re s s tow a rd s it s sh a re d go a l s , w h i c h i n c l u d e b u t a r e n o t s o l e l y d e f i n e d b y g r a n t deliverables outlined in its applications to the National Telecommunications and Information Administration's Broadband Technology Opportunities Program. Spanning a period of t wo years from Januar y 2011 to Januar y 2013, the Report represents a collaborative or partner-driven evaluation process. The Report provides a qualitative and quantitative portrait of the Partnership’s current progress towards shared goals, including stimulating broadband adoption, promoting broadband ’s socioeconomic effects for Ph i ladelph ia’s most u nderser ved popu lat ions, a nd impacting the health and capacit y of the Partnership.

The Repor t is str uctu red a s fol lows. The f irst t wo sections provide context for understanding the evaluation and findings: OTI describes relevant characteristics of Philadelphia in order to understand broadband adoption a n d p re s e nt s a n ove r v i e w o f t he Pa r t n e r s h i p, g r a nt deliverables, a nd progra m activ ities . Next , the Repor t outlines a l l sta ges of the eva luation, f rom pla nning of shared goals, methods, and evaluation activities, to analysis of the data.

The heart of this Report lies in the Findings section, which has been organized by shared goals identified by the Partnership. The section with an assessment of the Partnership’s progress in reaching deliverables outlined in its grant applications to the NTIA, as of January 2013. As this interim Report was written prior to the conclusion of the program ( June 2013), these numbers offer a point of comparison for understanding the findings from OTI’s evaluation activities. The Findings section then addresses t he f i rst set of sha red goa l s a nd a n s wers pa r t icipa nt-level questions of who attends K EYSPOTs, what needs are met by K EYSPOTs, and the impact of K EYSPOTs on pa r t icipa nt broadba nd adopt ion, employ ment a nd education, and community engagement. The latter half of the Findings section answers Partnership-level questions around building capacit y of the KEYSPOT partners and the functioning of the Partnership.

The Discussion and Conclusion sections synthesize t h e s e f i n d i n g s a c r o s s g o a l s t o p r e s e n t ov e r a rc h i n g principles and lessons from the Philadelphia experience, comparing the model of digital l iteracy and broadband adoption to other program literature. For digital inclusion prov iders , resea rchers , polic yma kers, a nd f u nders, the Repor t recommendations prov ide actionable steps for d e ve l o p i n g m e a n i n g f u l a n d s u s t a i n a b l e b ro a d b a n d adoption programs and policies in the future.

Introduction

10

Poverty: Families at 200% of Poverty level (ACS, 2010)

KEYSPOT

11

To properly eva lu ate t he work of t he Freedom R i n gs Pa r t n e r s h i p , i t i s c r i t i c a l t o u n d e r s t a n d t h e s o c i o -economic context in which this initiative has operated. T h i s contex t rema i n s profou nd ly cha l len g i n g. A f ter decades of popu lation a nd job loss, Philadelphia faces high povert y and unemployment rates, poor educational outcomes, and an over-burdened municipal budget at the start of the 21st centur y. Philadelphia a lso lags behind the countr y and other large cities in Internet access and broadband adoption. One recent assessment estimates that by 2035 almost 40 percent of the cit y—six hundred thousand residents—will lack the skills to participate in a digital economy.1 This section outlines the initial setting in wh ich t he Pa r tnersh ip lau nched ef for t s to improve digital literacy and Internet access for Philadelphia’s most marginalized communities.

Profile of the City in 2010Over the past t wo decades, Philadelphia has witnessed a pattern of changing demographics. In 2010, the population of the fifth largest cit y in the U. S. showed its f irst signs of grow th since 1950 —a gain of 0.6 percent since 2000. 2

From 19 9 0 to 2 010 , t he c i t y ’s H i s p a n i c p o pu l a t i on increa sed by 110 percent , wh i le t he nu mber of W h ite residents decreased by almost a third.3 According to the 2010 Censu s , Ph i ladelph ia’s 1,52 6 ,0 0 0 resident s were 43 percent A f r ic a n-A mer ic a n , 41 percent W h ite , 12 percent Hispanic and 7 percent Asian or Pacific Islander.4

Immigrants represented approximately 12 percent of the cit y’s population. Some 14 percent of Philadelphians had a disabilit y.5 In comparison to similar cities, Philadelphia had relat ively fewer you ng adu lt s a nd more seniors a s percentages of the overall population. 6

In 2010, Philadelphia faced stark povert y issues. One in four households fell below the poverty line, with higher rates for African Americans (31 percent) and Hispanics (41 percent).7 While the pover t y rate rema ined steady from 2004 to 2010, the number of residents living close to the povert y l ine grew by a lmost 50 percent. Indeed, Philadelphia’s median annual household income in 2010 was among the lowest for large cities, at $34,400, and had varied little in recent years. 8 This was partially the result of the city’s loss of jobs—almost 90,000 (more than one in ten jobs) bet ween 1990 and 2010.9 Increasingly, the t ypes

Background“This is a city that is struggling for employment, adequate education,

enough health care, food. [We’re] like a lot of other cities, but… we’re in the top ten. We’re also the poorest of those top ten. We’ve got a huge number of folks who qualify for free lunch. We’ve got a fifth of the population that’s been incarcerated.”

Staff member at a Managing Partner of the Freedom Rings Partnership

12

of jobs available to Philadelphians focused on “Eds and Meds”—educationa l and hea lth ser vices—and required higher levels of education than the manufacturing jobs that once dominated the job market.10 Compounding this bleak jobs situation, in 2010, more than half of the adult population had low literacy sk ill levels, mak ing reading basic instructions or filling out job applications difficult or impossible.11 A f ter yea rs of l it t le cha nge (a lt hou gh generally a percentage point above the national number), Philadelphia’s unemployment rate spiked after the start of the “Great Recession,” almost doubling bet ween 2007 (6 percent) and 2010 (11.5 percent).12

T h e “ G r e a t R e c e s s i o n” a n d t h e a c c o m p a n y i n g s u b p r i m e m o r t g a g e c r i s i s a l s o n e g a t i v e l y a f f e c t e d homeowners in some of the city’s poorest neighborhoods, w h e r e m e d i a n h o m e v a l u e s d ro p p e d by a s m u c h a s 5 0 p erc e nt . 13 R e nt er s s t r u g g le d a s we l l — t hos e w it h incomes under $35,000 (essentially Philadelphia’s median income) were much more likely to pay proportionately h i g h e r r e n t s t h a n t h e i r w e a l t h i e r c o u n t e r p a r t s . 1 4 Ph i l a d e lph i a h a d one of t he lowe s t r a t ios of s t re e t-b a s e d h o m e l e s s n e s s t o ove r a l l p o p u l a t i o n i n 2 010 ; a t t h a t t i m e t h e c i t y ’s p u b l i c h o u s i n g a g e n c y w a s t he fou r t h l a r g e s t i n t he c ou nt r y. T he Ph i l a de lph i a H o u s i n g A u t h o r i t y ( PH A) s e r v e d a p p r o x i m a t e l y eight y thousand people, including many of the residents in the neighborhoods targeted by FR P (of which PHA is a partner).15

The recession also affected social ser vices funding, f r o m f e d e r a l , s t a t e , a n d c i t y g o v e r n m e n t s , a s w e l l a s ph i l a nt h ropic fou nd a t ion s . 16 S p e nd i n g c u t s were especially ominous for the already troubled Philadelphia S chool Di st r ic t . W h i le st a nd a rd i z ed te st score s had been increa sing increment a l ly (up by about 4 percent e a c h ye a r f rom 2 0 0 5 to 2 010), Ph i l a de lph i a’s pu bl ic schools faced a budget shortfall and steadily decreasing enrollment.17 These pressures on the educational system had a disproportionate effect on already disenfranchised demograph ics: most students were A f rica n-A merica n (56 . 2 percent) or Hispanic (18. 6 percent), and the vast majority was poor (80.6 percent).18 The same demographic g rou ps were d i s prop or t ion a t e l y re pre s e nt e d a mon g

Philadelphians lacking high school diplomas (19 percent of the general population) and without college degrees (77 percent of the general population).19

Broadband Adoption & Internet AccessWhen the FR P launched, Philadelphia ranked well below the nationa l average in broadband adoption—as might be expected given the cit y’s challenging socioeconomic c h a r a c t e r i s t i c s a n d t h e d o c u m e n t e d l i n k b e t w e e n povert y and lack of Internet connectivit y. 20 In 2010, an est imated 55 percent of Ph i ladelph ia resident s lacked home Internet access, and in certain neighborhoods with h igher concentrations of poor A f rica n-A merica n a nd Hispanic residents, as many as 80 percent of households did not have broadband at home. 21 These neighborhoods were a lso cha racter i zed by lower rates of educat iona l atta inment—a nother factor associated nationa lly w ith lack of home broadband—than the cit y at large. 2 2 Lastly, in Philadelphia (and mirrored in national trends), research has demonstrated that seniors are significantly less likely than younger groups to use the Internet. 23 As a result , these commu nities of ten depended on public l ibraries to go online. 24

In an earlier effort to address disparities in broadband access, Philadelphia stood out as a pioneer among large c it ie s by promot i n g mu n ic ip a l w i re le s s . 25 Howe ver, Wireless Ph i ladelph ia , a s t he 20 0 4 ef for t wa s k now n, w a s l a r g e l y u n s u c c e s s f u l . C o n n e c t i o n s p e e d s w e r e s l o w e r t h a n e x p e c t e d , t h e n e t w o r k w a s n o t c o m p l e t e d , a n d i n 2 0 0 8 t h e p r o j e c t e n d e d . 2 6 The follow ing year, the cit y rolled out a new initiative c a l le d D i g it a l Ph i l a de lph i a — e nv i s ione d a s a “re s e t ” o f Ph i l a d e l p h i a’s a p pro a c h to t e le c om mu n i c a t i on s . Init ia l pla ns included u sing w ireless net work s to help br id ge t he d ig it a l d iv ide, increa sing cit y workers’ u se of technolog y and access to the Internet, offering better a nd more mu n icipa l ser v ice s on l i ne , a nd cre at i n g I T business enterprise zones. 27

City plans to address the digital divide are an essential part of efforts to keep pace with the ever-shifting digital

13

environment. Approximately 80 percent of Fortune 500 companies, such as Walmart, Comcast, and McDonald ’s, now only accept job applications online. 2 8 Government a gencies a nd public hea lt h orga nizations increa singly send cl ients to the Internet to access records or apply for benefits. And in 2014, the GED will shift to an all-d igita l format—test-ta kers w i l l have to ta ke t he exa m online. 29 As noted in seminal research about broadband a d o p t i o n i n l ow- i n c o m e c o m m u n i t i e s , w h i l e “u nt i l recently, a supplement to other channels of information a nd com mu n ic at ion ,” t he I nter net now “ ha s become increasingly a basic requirement of social and economic inclusion.”30 Indeed, “access to the Internet is not a choice: it is a necessit y, shaped by a complex array of barriers to access.”31 Those without digital literacy or reliable Internet face being left behind as the rest of the world progresses

into a digital future.A g a i n s t t h e c i t y ’s s o c i o e c o n o m i c c h a n g e s a n d

h i s tor ic a l c h a l len ge s to br i n g i n g broa d ba nd a nd it s benefits to Philadelphia’s most underser ved populations, the Freedom R ings Partnership launched in 2010. The Partnership is a diverse collection of communit y-based a n d s e r v i c e or g a n i z a t i on s t h a t re c e i ve d f u n d i n g to provide digital l iteracy and public computer centers to Philadelphians. The next sections explain the early history of this endeavor and establish the scope and natu re of this evaluation.

Percent Broadband Connectionsby census tract. (FCC, 2010)

14

Youth Outreach & Adolescent Community Awareness Program (YOACAP) Youth services, community organizing, health services

One Day At A Time (ODAAT)Substance abuse recover y, supportive housing, case management

Drexel University/Community College of Philadelphia/Philadelphia Housing Authority (Drexel/CCP/PHA)Supportive housing, workforce readiness, re-entry services, case management

People's Emergency Center (PEC)20 PCCsHomelessness, economic development, workforce readiness, domestic violence, services for immigrants, seniors, youth

Philadelphia FIGHT27 PCCsHIV/AIDS services, substance abuse recover y, youth health, advocacy and services for those living with disabilities, homelessness

Media Mobilizing Project (MMP)6 PCCsMedia literacy, advocacy around education, immigrant rights, unemployment, and healthcare, services for immigrants

Managing Partners & Host Site Issue Areas

PCC

PCC

& S

BA

SBA

Philadelphia Opportunities Industrialization Center (OIC)2 PCCs, one mobile labGED programming, workforce readiness training, mobile lab with GED and job training

Philadelphia Parks and Recreation (PPR)19 PCCs After school programming for youth, recreational programming for general public, seniors, youth

Free Library of Philadelphia (FLP)3 PCCs, one mobile labCommunity-based services for unemployed, childcare, youth, mobile lab serving neighborhoods and organizations without computer access

15

I n S eptember 2 010 , u nder m a nd ate of t he A mer ic a n Reinvestment and Recover y Act (AR R A), the National Telecommunications and Information Agenc y (NTIA) awarded a total of $18.2 million to a group of Philadelphia o r g a n i z a t i o n s u n d e r t h e B r o a d b a n d Te c h n o l o g y Oppor t u n it ie s Prog ra m (BTOP). A n add it iona l $8 .9 million in matching funds provided by partners brought t h e t o t a l b u d g e t t o $ 2 7.1 m i l l i o n . T h i s g ro u p, l a t e r forma lized a s the Freedom R ings Pa r tnership (FR P or “Partnership”), successfully applied for the funding under t wo BTOP grant categories: Public Computer Centers (PCC) and Sustainable Broadband Adoption (SBA). The Partnership had also submitted a request for approximately $31 million from the NTIA's Comprehensive Community I n f r a s t r u c t u r e (C C I ) d i v i s i o n o f B TO P t o b u i l d a middle mile f iber net work in the cit y, but this proposal was not successful.32

According to the NTIA, PCC projects would “expand public access to broadband ser vice” by establishing public c o m p u t e r c e n t e r s a t c o m m u n i t y i n s t i t u t i o n s , w h i l e SBA projects wou ld “promote broadband demand ” and provide “broadband education, awareness, training, access,

equipment or support.” BTOP funding overall included a s p e c i f i c e m p h a s i s o n r e a c h i n g “ u n d e r s e r v e d ” o r “v ulnerable” populations.33

A t t h e t i m e o f w r i t i n g , t h e c o r e P a r t n e r s h i p management consists of eleven academic, nonprofit , and govern ment institutions. The lead a gencies (“Primes”) on t he g r a nt s a re t he Cit y of Ph i l a de lph i a O f f ic e of Information Technolog y (OIT) for PCC and the Urban Affairs Coalition (UAC) for SBA . Un d e r t h e m a n a g e m e n t o f O I T, s i x “ M a n a g i n g Partners” provide public computer center facilities at their ow n sites, and a lso have created and manage additiona l public computer centers at community anchor institutions (referred to as “ host sites”) throughout Philadelphia. Un d e r t h e m a n a g e m e n t o f UAC , s i x M a n a g i n g Partners (three of which also receive PCC monies) provide digital l iteracy training and work to increase broadband adoption rates in the cit y.3 4 Prior to BTOP funding, each of these SBA partners had provided some digital l iteracy training. With the BTOP f u nding , these SBA partners suppor ted a nd expa nded tra ining at their ow n sites , a s well as at PCC sites and other A llied Organizations in

History of the

Freedom Rings Partnership

16

Training: classroom-based, instructor-led, scheduled trainings.

Subscriber: A household subscriber must have a paid or subsidized home broadband Internet connection. A household subscription may include more than one individual member of the household, but is still counted as a single household.

Awareness participant: attendees at outreach events and participants who reference

specific media channels (newspaper, magazine, website, etc) when describing how they heard about the program.

Awareness impression/target audience: total number of intended target audience, including estimates of media market (online, tv, radio, newspaper, and other outreach) impression.

SBA15,000training participants

5,000home subscribers

50 business subscribers

5,000netbooks distributed to Housing Authority residents

75,000reached by awareness campaign

➜ program portal/website

➜ e-learning system

Grant Deliverables to the NTIAAs stated in the PCC and SBA grant applications, the Partnership has aimed to achieve the following goals:

NTIA Definitions of Grant Deliverables

PCC77 public computing centers

15,000people per week

17

Freedom Rings Partnership Stakeholders

SBA PCC

UAC OITPrimes

Managing Partners Drexel

CCPPHA

YOACAP

ODAAT

MMP

PEC

FIGHT

80+ Community Anchor organizations

PPR

FLP

OIC

18

Philadelphia.35

Joi nt l y, t he pu bl ic computer center s a nd d i g it a l literacy trainings are branded as the KEYSPOT Program.36 For purposes of clarity, this Report refers to the SBA and PCC grants as FR P programs and to the physical places w here t he F R P prov ide s publ ic computer acce ss a nd digital literacy trainings as KEYSPOTs. SBA instructors and PCC lab assistants are k nown as “ Webguides.” Two different sources of data guided the development of the PCC deliverables. The Prime sur veyed prospective c o l l a b o r a t o r s d u r i n g t h e g r a n t w r i t i n g p r o c e s s t o understand their existing digital literacy program capacity, whether or not a n ex isting computer lab needed to be updated, and where new potential sites could be hosted. These prospective partners then selected site locations based on data collected by the Knight Center for Digital Excellence, which showed areas of broadba nd need. A later analysis commissioned by OTI confirmed that the placement of computer centers matched a rea s of high broadband need.37

To achieve the goal of f ive thousand new household Inter net subscr ipt ion s , SBA pa r t ners have promoted the K EYSPOT Discount Broadband Plan as well as the Comcast Internet Essentials program.38 In May 2011, UAC (the Prime) started negotiations with Mobile Citizen, a vendor of Clear Wireless ser vices, and Wilco Electronic Services, a minority-owned cable provider in Philadelphia, to cre ate a nd i mplement a low-cost Inter net pla n for K EYSPOT pa r ticipa nts (referred to in t h is Repor t a s the “K EYSPOT Discount Broadband Plan”). The FR P contributed $125,000 to subsidize the program. A year later, at the end of June 2012 , SBA partners piloted the program with Philadelphia Housing Authorit y (PHA). In November 2012 , they finalized the eligibilit y criteria: any KEYSPOT participant without the Internet at home can purchase a wireless home Internet plan for $14.99 per month, with a subsidized modem included.39

To ach ieve t he goa l of d istr ibut ing f ive t hou sa nd netbooks, SBA partners Drexel Universit y, Communit y College of Philadelphia, and the PHA created a separate t r a i n i n g p r o g r a m f o r PH A r e s i d e n t s . C o m m u n i t y

C o l l e g e o f P h i l a d e l p h i a i n s t r u c t o r s , a l o n g w i t h Drexel student interns, help PHA residents learn computer and Internet basics in t wo four-hour training sessions. A f t e r t h e y c o m p l e t e e i g h t h o u r s o f t r a i n i n g , P H A r e s i d e n t s c a n r e c e i v e a n e t b o o k . D r e x e l o v e r s e e s t h e m a n a g e m e n t of the program and the distribution of the netbooks. For the training deliverable, SBA partners set their goal by estimating an average of fourteen training hours per participant. To achieve the goal, each SBA partner implemented its own trainings and curriculum. To track lea r ners’ prog ress a nd t he d if ferent t y pes of t ra in ing , SBA pa r t ners proposed in t heir g ra nt applicat ion t he development of an e-learning management system. Lastly, to achieve the awareness deliverable, Drexel led a Partnership-wide branding campaign to “promote broadba nd awa reness i n v u l nerable popu lat ions .”4 0 In addition to traditiona l media channels, Drexel worked w it h pa r tners to create the K EYSPOT website (w w w.ph i l lykeyspot s .org ), a cent ra l repositor y for host site information and training schedules. To manage the grants, the FRP Primes and Managing Pa r t ners set up a ser ies of gover na nce st r uct u res a nd meet i n g sc hedu le s . Fi r s t , bot h g ra nt s held bi week l y PCC Steering Committee meetings and SBA Steering Committee meetings where staff representatives from each Managing Partner would meet to discuss implementation succe s se s a nd c h a l len ge s , qu a l it y i mprovement , a nd general resource sharing. The Partnership also created four sub-committees, or “ Work ing Groups,” focused on training, technolog y, awareness, and evaluation. Staff from Managing Partners h a d v a r y i n g le ve l s of p a r t ic i p a t ion i n e a c h of t he s e Working Groups, depending on their own level of interest. On the SBA side, the Training Working Group discussed t he t ra i n i n g cu r r icu lu m a nd d i g it a l l iterac y le a r n i n g goa ls for pa r ticipa nts . On the PCC side, the Tra ining Working Group partners primarily discussed professional development act iv it ies a nd goa ls for Webg u ides . The Te c h n o l o g y Wo r k i n g G r o u p p r i m a r i l y f o c u s e d o n PCC g ra nt need s , such a s pu rcha si ng , i nst a l l i ng , a nd

19

t rouble shoot i n g tech nolog y at t he computer centers . The Awareness Working Group focused on the branding a n d p u b l i c a w a r e n e s s c a m p a i g n o f t h e Pa r t n e r s h i p . Lastly, the Evaluation Working Group engaged partners in t he d e ve lo p m e nt of OT I ’s e v a lu a t ion pl a n , w h ic h i s discussed in the next section. As proposed in the original grant applications, the FRP has sought to further strengthen its programmatic reach by developing severa l collaborations bet ween the PCC and SBA partners. Several PCC sites offer SBA trainings, particularly those partners who receive both PCC and SBA funding. As the PCC grant focuses on establishing new public computer centers, SBA trainings ser ve as a natural complement to these centers. In addition to this diverse composition of BTOP-funded partners, the Partnership has also pursued collaborations with other entities such a s sen ior centers , t he School Dist r ict of Ph i ladelph ia , t he Mayor’s Com mission on Literac y, a nd t he Mayor’s Office of Reintegration Services for Ex-offenders, among other groups. Compared with other recipients of PCC and SBA grants nationally, the FRP is one of the largest that is focused on a single metropolitan area. The dollar amount and quantity

of project deliverables are outpaced by only a handful of BTOP awardees in other major U. S. cities, such as New York, Chicago, and Los Angeles. While many other large BTOP grant recipients utilize existing ser vice net works, like a statewide library system, the FRP constitutes a diverse network of organizations that have differing missions and target populations.41 The Managing Partners and host sites (collectively “partners”) offer public health services, GED preparation and other continuing education programs to support new immigrants, independent living support services for people w ith disabilities, drug and a lcohol addiction cou nseling , youth and seniors progra mming , and more (see p. 14). Members of the FR P work with those facing unemployment, homelessness and unstable housing, reentry following incarceration, low educational attainment, and other social dynamics common in Philadelphia. The provision of public broadband access and computer training complements this wide range of existing services and support networks for vulnerable populations.

20

Evaluation Question

WU

S/SE

S

GIS

Focu

sG

roup

s:

Part

icip

ants

Focu

s G

roup

s:

Web

guid

es

Inte

rvie

ws:

M

anag

ing

Part

ners

Inte

rvie

ws:

Prim

es

Inte

rvie

ws:

H

ost

Site

s

Inte

rvie

ws

wit

h Al

lied

O

rgs.

Prim

es’

Rep

ort

to

NTI

A

1a W h o p a r t i c i p a t e s a t KEYSPOTS?

1b What par t icipant needs are met through KEYSPOTS?

2W h a t a r e t h e e f f e c t s o f KEYSPOT access and training on broadband adoption?

3aW h a t a r e t h e e f f e c t s o f KEYSPOT access and training on participants’ employment status?

3bW h a t a r e t h e e f f e c t s o f KEYSPOT access and training on par ticipants’ educational attainment?

4W h a t a r e t h e e f f e c t s o f KEYSPOT access and training on communit y engagement?

5I n w h a t w a y s h a s t h e F R P b e e n a b l e t o i n c r e a s e t h e capacit y of its par tners?

6 H o w w e l l h a s t h e F R P functioned?

21

About the EvaluatorsO p e n Te c h n o l o g y I n s t i t u t e (OT I ) c o m p l e t e d t h i s evaluation as a named partner on the SBA grant and sub-contractor for the PCC grant.

OT I ’s i n vo l ve m e nt i n t h e F R P b e g a n w h e n t h e K night Center of Digital Excellence contracted OTI to assist Philadelphia partners in the formation of a BTOP proposa l . OTI a ssisted w it h one u nsuccessf u l su ite of appl ic at ion s for t he f i rst rou nd of N T I A Broadba nd Tech nolog y Oppor t u n it ies Prog ra m f u nd s, a s wel l a s t h e s e c o n d , s u c c e s s f u l p a i r. 4 2 I n 2 0 0 7, s t a f f a t OT I produced a report assessing Wireless Philadelphia , the Cit y's effort to bridge the digital divide with a cit y wide wireless net work.43

In order to complete the evaluation, OTI formed an interdisciplinar y tea m of qua litative, quantitative, and geospatial analysts, as well as technologists with experience in developing databases and online tools. Two analysts based in Philadelphia ser ved as the main liaison to the broader team.

For the writing of this Report, OTI relied on feedback from t wo independent, anonymous reviewers established

in the field of digital inclusion evaluation and research. OT I a l so sol ic ited feed ba c k a nd i nput f rom Pr i me s a n d p r e s e n t e d d r a f t s f o r r e v i e w t o t h e E v a l u a t i o n Work ing Group.

Discovering Common GoalsS i n c e 2 011, O p e n Te c h n o l o g y I n s t i t u t e (OT I) h a s developed an evaluation in collaboration with Primes and Managing Partners.4 4 The logic behind this collaborative approach relates to the diversit y of stakeholders involved i n t he F R P a nd t he i n a bi l it y of N T I A’s e s t a bl i she d metrics to capture the breadth of the project's ambitions, achievements, and challenges. The Partnership's work to establish sha red goa ls a nd eva luation instr u ments wa s itself a formative and formidable process of synthesizing visions and synchronizing documentation. This Report refers to t he e va lu at ion a s a pa r t ner- d r i ven one , a nd it d i f fers f rom top-dow n eva lu at ions , w here a f u nder or ot her inst itut iona l sou rce of control d ict ates what met r ic s prog ra m i mplementers need to me a su re a nd achieve goals.45

As the evaluator, OTI led the development of shared

Evaluation Methodology

22

goa l s , st a nd a rd i z ed i nd ic ators , a nd a comprehen si ve e va lu at ion pla n . OT I a l so c re ated a nd i mplemented the necessar y research tools and reporting platforms to assist the FR P in documenting progress towards NTIA grant deliverables.

For evaluation planning, OTI engaged in a t wo-step process that involved needs assessments and facilitated discussion. First, in the spring and summer of 2011, OTI interviewed all Managing Partners to distill each partner’s: (1) understanding of their own goals in providing public computer access or d igita l l iterac y tra inings; (2) idea s for questions whose answers would demonstrate success and address challenges; (3) preferred evaluation methods for a ns wer ing t hose quest ions . OTI a lso conducted a “Global Audit” of partners’ numerous existing evaluation instruments to determine if partners had common metrics bet ween them.

Ref lecting this information back to the Evaluation Work ing Group (EWG), OTI t hen faci l itated severa l discussions to help leadership a gree upon both shared goals and evaluation instruments.

In t he cou rse of t h is process , pa r tners ex tensively d i s c u s s e d N T I A s u b s c r i p t i o n m e t r i c s a n d t h e def inition of broadba nd adoption (see below). Severa l Managing Partners along with OTI questioned whether NTI A’s targets of increasing home subscribership a nd broadband demand adequately ref lected the complexit y of bro a d b a nd a d o pt i on a nd bro a d b a nd ’s i mp a c t s . 4 6 Partners agreed that this definition (compared with the NTIA’s definition) more accurately ref lected the type of work being done in Philadelphia , and thus adopted the expanded definition.

“An Adopter is a person who has increased their computer skills or gained new computer skills in order to obtain information they need, perform transactions, and communicate more effectively and thereby improve his/her quality of life."

"A Subscriber is a person who has a paid or subsidized subscription to the Internet, who becomes an Internet user of a library, computer center or other public access point for no cost, or who gets wireless access to the Internet through handheld devices such as cell phones, smart phones, iPads, or netbooks. A Subscriber is someone who previously did not access the Internet in one of these ways." 47

"Broadband Adopter: An Adopter is a person who has increased their computer skills or gained new computer skills in order to obtain information they need, perform transactions, and communicate more effectively and thereby improve his/her quality of life."

"Broadband Subscriber: a person who obtains a 1) fully-paid, 2) subsidized or 3) free subscription to the Internet (for example becomes an Internet user of a library, computer center or other public access point for no cost) and uses a a) computer or b) handheld devices such as cell phones, smart phones, iPads, or netbooks for access to the Internet. A New Subscriber is someone who previously did not access the Internet in any of these ways prior to program participation."

Tra i n i n g l iterat u re (d ated Au g u st 2011) produced by FIGHT for new digital literacy instructors (Webguides) ref lected a refined definition:

In July 2011, on a BTOP discussion list that OTI hosted and that included a wide variety of BTOP awardees from ac ros s t he cou nt r y, Ph i ladelph ia F IGH T— a pa r t ner responsible for managing one third of all K EYSPOTs in the Partnership—said:

23

Partnership reaches and serves target populations.

Participants adopt broadband.

Participants increase their educational attainment and improve their employment status.

KEYSPOTs serve as hubs for community engagement.

The Partnership strengthens the capacity of partners to advance a collective agenda.

The Partnership functions well.

Inte

rnal

Fun

ctio

ning

Participant Outcom

es

1

3

2

4

6

5

Shared goals

of the

Freedom RingsPartnership

In December 2011, a total of six goals emerged from the needs assessments and collaborative discussions.

These shared goals (across both grants) include both pa r ticipa nt-level outcomes as well as pa r tnership-level ones , w it h t he a ssu mpt ion t hat pa r t nersh ip dy na m ics interact with the abilit y of the FR P to help participants benefit from broadband technologies. Goals 1, 5, and 6 relate to the Partnership’s internal functioning and form t he fou nd at ion for t he pa r t icipa nt-level outcome s of adoption (Goal 2), education and employment (Goal 3), and communit y engagement (Goal 4).

T he goa l s a l so ref lec t lon g-ter m outcome s , such a s posit i ve c h a n ge s i n employ ment a nd educ a t ion a l a t t a i n m e nt d u e to e x p o s u re to F R P pro g r a m s , a n d short-term ones, such as increased digital l iteracy sk ills,

workforce sk ills, and educational opportunities. Due to the lack of standardization bet ween diverse

p a r t n e r s ’ e v a l u a t i o n i n s t r u m e n t s a s d e t e r m i n e d by t he G lob a l Au d it , OT I le d a pro c e s s of d e ve lopi n g Pa r t n e r s h i p - w i d e e v a l u a t i o n re s e a rc h i n s t r u m e n t s , including specific questions to be asked in sur veys, focus groups, a nd inter v iews for rev iew a nd approva l by t he E WG . One pr i ma r y poi nt of d iscu ssion wa s how t he evaluation would fairly represent the range and diversit y of programming offered by host sites. The Partnership never issued standard curricula or a set of trainings for host sites, and the host sites all serve different populations with var ying levels of concern over data collection (e.g., people living with HIV/AIDS or women who are victims of domestic violence). Standardizing a set of indicators

24

across a non-standard program was a formidable challenge. A lso of con siderable debate wa s t he d i f f icu lt y i n

match ing long-term goa ls w it h shor t-term ind icators , p a r t ic u l a r l y for me a su r i n g home su bs c r ipt ion s . For example, while trainings stimulate interest in subscription, ac t u a l su bsc r ipt ion s m i g ht not occ u r w it h i n a shor t t ime f ra me. Since these long-term sha red goa ls wou ld b e d i f f ic u lt to c a pt u re b ot h i n t he s hor t t i m e (t wo yea rs) a l lot ted for prog ra m eva lu at ion a nd in l ig ht of environmental factors external to the Partnership’s work (e.g., an economic downturn), the EWG instead focused on mea su ring shor t-term ind icators such a s increa sed skills, increased knowledge, and improved opportunities in each of the goal areas.

Broadband ’s positive socioeconomic and community effects presented additional challenges for measurement. For employment outcomes, the EWG members identified t he need to u se qu a l it at ive i nd ic ators such a s stor ie s of sk i l ls development a nd stories of entrepreneu rsh ip to mea su re prog ress . E WG members a lso recog n ized t he d i f f icu lt y i n docu ment i n g i mproved educ at iona l outcomes—a long-term investment—in the shor t t ime allotted to FR P programs. Additionally, due to sensitivity of collecting data on minors, OTI excluded youth from this evaluation, thereby limiting the scope of evaluation to adu lt educ at iona l i mpac t s . 4 8 A s a re su lt , members d iscu ssed col lecting stor ies of how adu lt pa r t icipa nt s a c c e s s a nd c on ne c t w it h e d u c a t ion a l opp or t u n it ie s prov ided by K EYSPOTs, in add ition to a ny stories of participants using KEYSPOTs to advance their education. For com mu n it y enga gement i nd icators , OT I quer ied Managing Partners and Primes about their examples and interpretation of communit y engagement. The resulting range of activities included attendance at a communit y meeting at a K EYSPOT, peer-led lea rn ing , a nd u sing technolog y for communit y-focused work.

A s for i nc re a si n g broa d ba nd a dopt ion , pa r t ner s a greed on t he expa nded def init ion of adoption noted previously. Nevertheless, the EWG struggled to arrive at indicators due to differences in the nature of NTIA grant deliverables for the t wo programmatic sides of the FR P. PCC Managing Partners did not need to include a focus

on home broadband subscription as an indicator, while SBA Managing Partners and the SBA Prime did. For the SBA side, capturing specific subscription numbers was challenging, given that there were no publicly available d at a set s of su bscr ipt ion rate s t hat matched t he t i me frame of the evaluation. A lso, even though K EYSPOT pa r t ner s promoted t he C omc a s t I nter net E s sent ia l s program, Comcast was unwilling to share the number of subscribers to the program.49 After careful deliberation, t he E WG a r r i ved at a set of i nd ic ators t hat ref lec t a range of activities related to a spectrum of adoption. The group agreed that qualitative and quantitative measures wou ld prove u sef u l to track a nd focu sed on gathering information from the Work station User Su r vey, focus groups, and inter views.

I n order to t r a c k pro g re s s tow a rd s t he s e sh a re d goa ls u si n g t he i nd ic ators a g reed upon by t he E WG , OTI conver ted each goa l into a n a nswerable question or questions.

After the Partnership approved the final evaluation pla n i n Apr i l 2012 , OT I subm it ted a nd received IR B exempt ion s for t he pla n by t he I n st it ut iona l R e v ie w Boards at the New America Foundation and Philadelphia FIGHT (April and May 2012). Then, after being certified by NTIA as having met criteria for an exemption from IR B review under the Common Rule for the Protection of Human Subjects ( June 2012),50 OTI carried out a revised data collection plan across a total of seven months, July 2012-Januar y 2013.51

It shou ld be noted that throughout the eva luation planning and implementation process, the entire tea m routinely solicited feedback from Primes and EWG to help design or approve evaluation instruments and their deployment. For example, OTI drafted a sampling plan a nd set of questions for the Work station User Su r vey (explained below), which the EWG reviewed, amended, a nd sa nctioned to f it its need s a nd concerns. A s wel l , OTI produced quar terly formative eva luation repor ts, which contained prelimina r y data a nd a na lysis so that the Partnership cou ld ref lect on its progress and make any needed changes per preliminar y findings.

25

Identifying Key Data SourcesTo complete t he eva lu at ion, OT I t a rgeted fou r ma i n populations: key leadership in the Partnership (staff at Primes, Managing Partners, and host sites), key leadership affiliated though not funded by BTOP (staff at Allied Organizations), frontline staff (Webguides), and participants.

Eva luators a lso relied on severa l key docu ments to piece together the histor y and examine FR P milestones including monthly reports (written by Managing Partners, and a lso referred to throughout as “Partners’ Monthly Repor ts”), NTI A repor ts (w ritten by Primes, a nd a lso refer red to t h rou g hout a s eit her “ Pr i me s’ Q u a r terl y Report to the NTIA” or “Primes’ Annual Report to the NTIA”), and American Recover y and Reinvestment Act (AR R A) reports. These documents contain information such as the number of PCC users, PCC training hours, SBA training hours and participants, and home broadband subscription data.

Collecting & Analyzing DataTo measure the indicators agreed upon by the partners, OT I d re w f rom Pa r t nersh ip doc u ment s a nd quer ied st a f f a nd pa r t icipa nt s u si n g a va r iet y of qu a nt it at i ve a n d q u a l i t a t i ve m e t h o d s , o u t l i n e d b e l ow. D e t a i l e d information regarding each evaluation instrument, such as response rates, composition of focus group participants, and protocols for quoting individua ls, can be found in the appendices.

➜W o r k s t a t i o n U s e r S u r v e y ( W U S) : I n collaboration with the EWG, OTI developed two versions of an adu lt participant sur vey, k nown as the Short and Long W US . The Shor t W US col lected demog raph ic information from KEYSPOT participants on an ongoing basis at PCCs via a computer script that opened the survey in a web browser whenever a participant logged into the computer. For SBA partners, instructors shared a unique link for classroom participants to fill out during trainings.T h e L o n g W U S c o l l e c t e d t h e s a m e d e m o g r a p h i c i n for mat ion , but a l so posed add it iona l que st ion s on how the par ticipa nt traveled to the center or training ,

reasons for coming, frequency of Internet use, and home subscription status. OTI deployed this sur vey by using the same process described previously. The Long W US was deployed for t wo weeks, one week in October 2012 and one week in December 2012 . For both versions of the sur vey, only participants 18 years or older were invited to complete the sur vey. Youth-ser ving host sites did not deploy the W US, including all of the Philadelphia Parks and Recreation sites. OTI analyzed both the Short W US and Long W US data for t his repor t to u ndersta nd ta rget popu lations’ demographics and broadband adoption characteristics.

➜Geospatial analysis (GIS): OTI mapped WUS participant data to examine demographics and compare neig hborhood s of K EYSPOT pa r t icipa nt s w it h a rea s w here broadba nd subscr ipt ion rates a re low. Inter net Ser vice Providers must report this information annually to the FCC (in Form 477).52 At the time of the writing o f t h i s R e p o r t , F o r m 4 7 7 d a t a s e r v e s a s t h e m o s t a c c u r a t e b a s e l i ne for a s s e s s i n g c it y w ide bro a d b a nd subscription rates.53

➜Standardized Exit Survey (“Exit Survey”): At the direction of the SBA Prime and in collaboration with partners, OTI designed an online Exit Survey for classroom-based, instructor-led trainings, as defined by the NTIA. The Exit Survey assessed the FR P's impact on individual participants, specifically asking about job skills, educational opportunities, broadband use and access, and interest and intent to purchase home subscription after training.

Focus groups wit h par t icipant s: OT I g a t h e r e d s t o r i e s f r o m p a r t i c i p a n t s t h r o u g h f o c u s g r o u p s . D i s c u s s i o n t o p i c s i n c l u d e d r e a s o n s w h y p a r t i c i p a n t s a t t e n d e d K E Y S P O Ts , t h e i m p a c t o f K EYSPOTs on participant employment and education, community engagement, and adoption, as well as general ref lec t ion s on pa r t ic ipa nt s’ K E YSPOT e x per ience s .

Focus groups wit h We bguides: A t t h e request of Managing Partners, OTI exa mined the role of Webg u ide s , w ho f u nc t ioned a s f ront l i ne st a f f a nd si mu lt a neou sly were ser ved by t he F R P. T he se focu s

26

groups with Webguides explored two themes: Webguides' ow n profe ssiona l ex per ience s w it h i n t he Pa r t nersh ip a n d t h e i r o b s e r v a t i o n s o f s t u d e n t s a n d c o m p u t e r center attendees.

I n t e r v i e w s w i t h P a r t n e r s h i p s t a f f (Primes, Managing Partners, and host sites): OT I i nter v ie wed t he se st a f f members to u nderst a nd t heir perspectives on how t he Pa r tnership f u nctioned a nd inf luenced pa r tners’ orga nizationa l capacit y. OTI a lso quer ied st a f f about what t y pes of outcomes t hey observed for participants. It was important to interview all t h ree le vel s of t he Pa r t nersh ip st a f f , f rom Pr i me s to Managing Partners and host sites, in order to fully capture t he va r ied ex per ience s of pa r t icipat i n g i n t he mu lt i-stakeholder Partnership.

I n t e r v i e w s w i t h s t a f f a t A l l i e d Organizations: OTI interviewed Allied Organizations that did not receive BTOP funding but worked with the Partnership in some capacity. As described previously, the Partnership strengthened its work through collaboration w it h ot her d iverse com mu nit y-ba sed orga nizations in Ph i ladelph ia . OTI sol icited feedback f rom Ma na g ing Partners for suitable candidates to inter view. Inter views focused on the organizations’ external perspectives of the Partnership and its achievements and challenges.

Primes’ Repor ts to NTIA:5 4 A l l Ma na g i n g Pa r t ners t rack ke y out put s for t he F R P prog ra m s i n Partners’ Monthly Reports, which the Primes then review and compile into Primes’ Quarterly and Annual Reports to the NTIA. OTI relied on Primes’ Quarterly Reports to track key outputs for each grant: (1) SBA training hours, training participants, subscribers, netbook distribution, a nd nu mber of pa r t icipa nt s exposed to out reach a nd awareness materials, and (2) PCC sites and average users p er we e k . OT I a l so e x a m i ne d t he Pr i me s’ 2 011 a nd 2012 A n nu a l Repor t s to t he N T I A to deter m i ne t he range of training hours and topics offered by PCC and SBA programs. Fi n a l l y, over t he cou r se of t he projec t , pa r t ner s expressed interest in creating a visual of the relationships built within the Partnership, as well as with organizations

outside of the Partnership. OTI developed an initial list of partnerships from a participator y exercise with partners du r i n g a Ma rch 2012 meet i n g. In Ja nu a r y 2013 , OT I solicited partner feedback about additional partnerships to include. Using Gephi, an open source data visualization soft ware program, OTI mapped this list of internal and e x ter n a l pa r t ner sh ips . T hou g h t h i s a n a l y si s wa s not included in the initial evaluation plan, creating this visual aligned well with Goals 5 and 6 (increased partner capacity and Partnership functioning). The visual represents an i n it ia l at tempt to demonst rate t he i nter na l dy na m ic s and widespread reach of the Partnership (see Appendix H, also p. 50). B y c r e a t i n g a m i x e d m e t h o d s a p p r o a c h t o evaluating FR P impacts, OTI aimed for a rich, textured understanding of the Partnership and its progress towards shared goals. The timeline (see p. 29) demonstrates when each of these evaluation activities occurred, along with some key Partnership milestones.

Doing the Analysis OT I conduc ted qu a nt it at i ve a na lysi s u si n g PSPP, a n o p e n - s o u rc e ve r s i on o f SPS S , a s t a t i s t i c a l s o f t w a re packa ge, a nd Excel. Maps were created using A rcGIS . Where appropriate, OTI ran chi-square tests and t-tests to determine statistical significance between demographic groups and home broadband subscription rates.

OTI conducted qualitative analysis using a team of coders and reviewers to code, summarize, and extrapolate key themes according to three factors:

➜➜ Frequency of the theme, as determined by number of stories or number of people

➜➜ Saturation of the theme, meaning that the theme came up across all focus groups or interviews

➜➜ Triangulation of theme, meaning that the theme c a me u p a c ros s d i f fere nt p a r t ic ip a nt g rou ps (K EYSPOT participants, trainers, staf f, A llied Organizations)

27

Shared Goal Evaluation Questions

SelectedIndicators

Partners attract and meet needs of target populations.

1a: Who participates at KEYSPOTS?

1b: What participant needs are met through KEYSPOTS?

Demographic profile of KEYSPOT participants

Reasons for using KEYSPOTS

Participants adopt broadband.

2: What are the ef fects of KEYSPOT access and training on broadband adoption?

Number of subscriptions

Stories of participants overcoming fears, finding relevance, sharing broadband benefits with others

Range of Internet access options and use

Participants increase their educational attainment and improve their employment status.

3a: What are the ef fects of KEYSPOT access and training on participants’ employment status?

3b: What are the ef fects of KEYSPOT access and training on participants’ educational attainment?

Stories of educational attainment and workforce development

KEYSPOT training outcomes

BTOP sites serve as hubs for community engagement.

4: What are the ef fects of KEYSPOT access and training on community engagement?

Stories of community building and support at KEYSPOT

Partnership strengthens the capacity of partners to advance a collective agenda.

5: In what ways has the FRP been able to increase the capacity of its partners?

Stories about staff development, technical infrastructure, resource sharing

New internal and external collaborations

Partnership functions well.

6: How well has the FRP functioned? Stories about building the partnership, communications

Reporting ResultsTo focu s at tent ion on t he FR P a nd it s succe sse s a nd cha l lenges, t h is Repor t f irst presents a section on t he most recent progress made on NTIA deliverables. Then, OTI synthesizes and presents quantitative and qualitative findings by the Partnership’s shared goals. As indicated by the methods alignment table on page 20 and as mentioned in the Discovering Common Goals section above, all but one goa l—which relates to increased capacit y—rely on both quantitative and qualitative data. The Report uses this structure to demonstrate the holistic impact of the FR P in any given area. Quantitative findings may provide a quick assessment of a particu lar goa l, but qua litative

data furnish the rich and more nuanced meaning behind t he nu mbers a nd pa int a more comprehensive pictu re of t he Pa r t ner sh ip’s i mpac t . I n a dd it ion , qu a l it at i ve instru ments provide answers to explorator y eva luation questions, produce meaningful themes that do not reduce experiences of participants or staff to rigid categories, and allow participants and staff to speak in their own language.

A s a r u le , to m a i nt a i n t he a nony m it y of sou rce s of qu a l it at ive d at a , t h is Repor t refers to i nter v iew or focu s g roup subject s w it h genera l ident if iers , such a s “ interviewee,” “staff person,” “staff member” (for interview su bjec t s), Webg u ide s (for foc u s g roup su bjec t s w ho work as frontline staff ), and participants. To minimize the invasiveness of data collection, OTI did not ask for

28

demographic information from participants during focus groups; they are simply referred to here as “participants.”

LimitationsOT I faced l i m it at ions i n conduct i n g t h is eva lu at ion . Some were com mon to col laborat ive eva lu at ions a nd some unique to the field of digital literacy and broadband adoption. Fi rst , t he process of developi n g sha red goa ls a nd indicators across diverse pa r tners was a t ime-intensive proce s s t h at shor tened t he t i me ava i l a ble for ac t u a l data collection and analysis. Negotiating and creating a Partnership-wide evaluation plan was a challenge given the FRP’s many diverse stakeholders, each with their own data collection processes. This timeline was further restricted by the unclear NTIA process for certif ying the evaluation pla n a s bei n g exempt f rom Hu ma n Su bjec t s R e v ie w. When the NTIA did certif y the plan, OTI adjusted its timeline and revised the scope of the original proposed evaluation activities to focus on primar y data collection and analysis. As mentioned previously, data collection ran from July 2012 through Januar y 2013, the latter months of the funding cycle. Second, the scope of the Human Research Subjects p r o t o c o l — a p p r o v e d b y t w o s e p a r a t e I n d e p e n d e n t R e v i e w B o a rd s — l i m i t e d t h e a b i l i t y t o c o l l e c t d a t a f rom a l l progra m pa r ticipa nts . The protocol specif ied t h a t e v a l u a t o r s c o u l d n o t c o l l e c t a n y i n f o r m a t i o n f rom ind iv idu a ls u nder 18 , despite t he fact t hat ma ny K EYSPOTs ser ve youth. In an ef fort to minimize this l i m it at ion, OT I inter v iewed st a f f f rom yout h-ser v ing orga nizations to indirectly collect information on the prog ra m’s i mpact for t h is popu lat ion. The sect ion on progress on NTI A deliverables does include a l l youth usage of public computer centers and trainings. Youth o r g a n i z a t i o n s , s p e c i f i c a l l y Ph i l a d e l p h i a Pa r k s a n d Recreation, a lso developed its ow n track ing protocols separate from this evaluation.55 T h i r d , O T I d i d n o t s y s t e m a t i c a l l y e x a m i n e t he impact of t he awa reness ca mpa ign on st imu lating b r o a d b a n d d e m a n d . D r e x e l , t h e m a i n M a n a g i n g

Pa r t ner spea rhead i ng t he awa reness ca mpa ig n ef for t , tracked events and impressions in order to fulfil l NTIA reporting requirements. Fourth, like any impact measurement tools, the ones t he Pa r t nersh ip chose had l i m it at ion s . S el f-selec t ion and social desirabilit y bias are problems that skew both quantitative data and qua litative data towards positive ou tc ome s , p a r t ic u l a r l y a mon g p a r t ic ip a nt f i nd i n g s . Generally, people who participate in evaluation research express ver y positive reviews of their experiences;56 for participants who had average or negative experiences, they may not feel compelled to fil l out a sur vey or participate in a discussion to share those experiences. In all of OTI’s evaluation instruments, participants were encouraged to provide honest answers, as no identif ying information was ever collected, but the problem of social desirabilit y bias stil l occurs. In focus group research, this problem is compounded by a “group think ” phenomenon where participants tend to a g ree w it h one a not her, a nd pa r ticipa nt s who have d if fer ing opin ions may hesit ate to spea k . To m itigate t h i s p r o b l e m , O T I m o d e r a t e d f o c u s g r o u p s w i t h probing questions designed to elicit differing opinions. Additionally, recruitment of Webguides took place with the help of staf f at Managing Partners, some of whom told OT I t hat t hey i ntent iona l ly su g ge sted na me s of i nd i v idu a l s w it h va r ied opi n ion s of K E YSPOTs a nd F R P pro g r a m s . T hu s , Web g u ide fo c u s g rou ps were not as prone to group consensus. B e yo n d t h e s e c o m m o n b i a s e s i n r e s e a r c h , t h i s evaluation encountered challenges with Partnership-wide creation and implementation of the sur veys. Given that these tools were collaboratively developed, OTI did not collect data deemed unessential, including education or income level of participants and participant addresses. The lack of this information then l imited the t ypes of possible analysis, particularly for geospatial work which requires more granular data than ZIP code for reliable analysis. In terms of implementation, some partners chose not to use one or both sur veys. Fewer Managing Partners and K EYSPOTs used the Exit Sur vey compared to the WUS, likely because it was created later in the evaluation

29

process a nd t he su r vey d id not load automat ica l ly on computer ter m ina ls for pa r t icipa nt s . A s bot h su r veys were deployed online with a computer, these sur veys also self-select for participants who have a basic familiarit y with typing and keyboard skills. These issues all limit the representativeness of the quantitative findings. Finally, while the FRP is ambitious and large in its scope, this evaluation design is not longitudinal, does not follow a cohort population, and therefore does not measure pre- and post-effects of the FRP’s programs. With this short-term design, it does not track any long-term indicators, such as

number of participants who received jobs after attending a KEYSPOT, or number of participants who apply to college as a result of KEYSPOT. Similarly, it cannot attribute any changes in citywide employment, economic indicators, home subscription rates, or other statistics directly to the program. This report focuses solely on the short-term impacts of the program on participants, with the understanding that a variety of external factors (e.g., funding of digital inclusion programs, funding of other social services and community-based programs, economic crises, etc.) inf luence long-term effects and cannot be obser ved until much later.

FRP MilestoneOTI Evaluation PlanningOTI Data CollectionApplies to Both Grants

SBAPCC

SeptemberNTIA Announce UAC and OIT Grant Awards

July1st Focus

Group

November3rd Focus

Group

MayFinal PCC Launched

JulyWUS (Short)

andExit Survey

AugustPCC Reporting

Platform Launched

May2nd FRPTrainer’s

Roundtable

MayFinal Report

JanuaryFRP Launches

KEYSPOTSBrand

January1st PCC

Launched

JunePCC Ends

September2nd Focus

Group

OctoberPCC Needs

Assessment

JanuaryFRP

Trainer’sRoundtable

DecemberFirst SBATraining

AugustSBA NeedsAssessment

NovemberGlobal Audit/FRP Goals

MarchIRB Approval

MayWUS (Short)and Exit Survey Pilots

OctoberWUS (Long)

DecemberWUS (Long)

AugustSBA Ends

SeptemberWilco/MobileCitizen Offer

MayFRPSustainabilityForum

AprilEvaluationPlan

JuneNTIAApproval

JanuarySBA ReportingPlatformLaunched

20102011

20122013

30

31

I n t h i s sec t ion of t he R epor t , OT I f i r s t pre sent s a n updated look at the Partnership’s progress on achieving its NTIA deliverables. Then, OTI presents findings from primar y data collection related to each of the six goals to provide a more comprehensive picture of the FRP’s success bringing broadband and its benefits to Philadelphia’s most underserved communities. Findings related to KEYSPOT target popu lations a nd their needs come f irst , thereby prov id ing contex t to t he t h ree pa r ticipa nt-level goa ls focused on K EYSPOT effects on broadband adoption, jobs and education, and communit y engagement. This section ends by addressing the two final Partnership-level goals related to capacit y and the functioning of the FR P.

Each goa l is presented as a question, followed by a summar y of the findings and methods used to validate

t he f i nd i n g s s t a t e m e nt s . A s m e nt ione d i n pre v iou s sections, ma ny of the eva luation instr u ments u sed for this Repor t represent a narrow v iew of the popu lation sur veyed, making it difficult to draw general conclusions about K EYSPOT pa r ticipa nts, Webg u ides, host sites , or A llied Organizations. However, where findings that per t a i n to t he se popu lat ion s t r ia n g u late w it h i n a nd bet ween qualitative and quantitative data, the results are genera lly strong. A f u ll audit of the representativeness of the Findings for each goal is available in Appendix I .

Because of the wealth of data, questions pertaining to who participates (Goal 1) and how well the Partnership has functioned (Goal 6) are subdivided into categories t hat prov ide a more ref i ned a nd complex por t ra it of participants and the FR P.

MethodologyTotal Organizations

RepresentedHost Sites

representedTotal Number of

Responses or Interviews

WUS Data (Short and Long) 8 of 9

Managing Partners54 of estimated 60

sites who serve adults3148

Adult Responses (46% Response Rate)

Long WUS/GIS 8 of 9 Managing Partners

43 of estimated 60 sites who serve adults

538 Adult Responses

(51% Response Rate)Standardized Exit Survey (SES) 7 of 9

Managing Partners(OTI did not collect this

information)105

Adult Responses (93% Response Rate)

Participant Focus Groups 4 of 9

Managing Partners11 of 80+

Host Sites28

KEYSPOT Adult Participants

Webguide Focus Groups 8 of 9

Managing Partners13 of 80+

Host Sites13

Webguides

Interviews with Partnership Staff 9 of 10

Managing Partners*

2 of 2 Primes

5 of 80+ Host Sites

18 Staff Interviews

Interviews with: Allied Organizations 6

Allied Organizations(Not Applicable)

7 Allied Organization

Staff Interviews* 10 Managing Partners includes one managing partner who ceased operations earlier in the Partnership. One managing partner did not respond to requests for interviews.

Findings

Methodologies and Populations Represented

32

A review of the FR P’s grant deliverables to the NTIA forms part of the Partnership’s process of a ssessing its sha red goa ls . The Partnership felt these deliverables provided a narrow view of broadband adoption and it s i mpact s but , never t heless , mon itored a c h ie ve me nt of t he s e de l i ver a ble s . T he section below reports on the progress as of Januar y 2013.

Public Computing Centers T h e Pa r t n e r s h i p h a s o p e n e d 7 9 p u b l i c computer centers—two more than promised. The lat ter ha lf of 2011 w it nessed a rapid i ncre a se i n nu mber of computer centers openings. Openings slowed down at around the midpoint of the grant period.

In terms of its weekly attendance numbers, the number lies below PCC partners’ stated target of serving 15,000 users per week. As of January 2013, the PCCs collectively served 4,654 visitors per week . This number peaked in July 2012 (5,359 visitors, or nearly 36 percent of its target).

Sustainable Broadband Adoption The SBA grant was responsible for key outputs related to three areas: subscriptions, training, and awareness. For each of these issues, they tracked key outputs as defined by the NTIA .

SBA partners partially met its subscription targets. For exa mple , pa r t ners demon st rated ste ady prog re ss i n t he d ist r ibut ion of netbook s . A s of Ja nu a r y 2013 , the Drexel/PHA/CCP program had given out 4,253 of the 5,000 netbooks to program participants enrolled in their trainings.

In-home and business subscription numbers remained low (see p. 2 2 for a f u l ler d i sc u ssion of su bscr ipt ion metrics data collection). However, a noticeable increase

in numbers took place in Januar y 2013, not long after the finalization of the FR P Discount Broadband Plan with Wilco/Mobile Citizen.

SBA partners met and exceeded their training targets. By Januar y 2013, they had ser ved 21,470 trainees with a total of 200,766 training hours. SBA partners hit their target of 15,000 trainees by July 2012 , nearly a year before the conclusion of the grant. On average, they provided about nine hours of structured training per participant, fewer than the proposed fourteen hours per training.57

SBA also exceeded awareness targets, reaching more than 460,000 people through marketing and advertising of K EYSPOT programs on a variet y of traditional and new media platforms. Drexel, the SBA partner that led the awareness campaign, completed the construction of t he Pa r tnersh ip’s por t a l (w w w.ph i l lykeyspot s .com) in Januar y 2012 . Bet ween Januar y and December 2012 , the site attracted 88,363 visits, 27 percent of which count as new visits, according to site analy tics.

Deliverables: Percent to GoalMM/Y Y

PCCs opened Netbooks

Average Visitors

Netbook Household

Netbook Business

Training Participants

Awareness Impressions

Training Hours

1/11 2.60 0.00 0.14 0.00 0.00 0.13 0.00 0.072/11 2.60 0.00 0.23 0.00 0.00 0.53 0.00 0.303/11 2.60 0.00 0.48 0.02 2.00 1.85 0.12 1.324/11 6.49 0.00 1.04 0.02 2.00 4.09 0.35 2.995/11 9.09 0.00 2.37 0.02 2.00 5.97 0.98 4.366/11 18.18 0.00 4.20 0.02 2.00 10.56 32.16 6.017/11 31.17 0.00 4.15 0.02 2.00 12.30 34.74 7.558/11 53.25 2.02 9.82 0.02 2.00 16.14 36.83 11.699/11 62.34 2.26 11.39 0.02 2.00 19.78 50.46 16.03

10/11 75.32 4.78 16.05 0.06 2.00 25.95 65.73 22.0311/11 79.22 11.58 18.96 0.06 2.00 32.38 79.64 27.1912/11 81.82 17.58 18.02 0.06 2.00 39.05 96.53 32.921/12 84.42 24.72 24.04 0.22 2.00 45.70 112.26 38.492/12 89.61 30.98 27.17 0.22 2.00 58.91 187.88 44.473/12 94.81 35.48 28.04 0.22 2.00 67.31 263.41 51.014/12 98.70 40.86 27.88 0.22 2.00 74.29 278.35 56.815/12 101.30 46.76 31.18 0.22 2.00 87.56 290.02 64.186/12 101.30 53.84 29.72 0.66 2.00 99.68 395.49 69.717/12 101.30 58.56 35.72 1.82 2.00 107.14 409.97 74.058/12 101.30 65.22 30.08 3.02 2.00 114.86 413.36 79.439/12 101.30 68.00 27.36 4.98 4.00 122.38 592.67 82.97

10/12 105.19 72.72 27.44 6.16 4.00 128.51 598.26 86.9011/12 105.19 75.78 31.41 7.82 4.00 134.84 600.78 90.8112/12 105.19 79.12 22.96 10.84 4.00 139.67 605.95 93.491/13 105.19 85.06 31.25 18.98 34.00 143.13 611.92 95.32Total 81 4253 5358 949 17 21470 5123466 200766Goal 77 5000 15000 5000 50 15000 75000 210630

Progress On NTIA Deliverables

exceeded deliverable not met as of 1/2013

33

Demographic Characteristics

➜➜ KEYSPOTs primarily ser ve African Americans, and age and gender are well represented across KEYSPOT users.

➜➜ K E YS P OTs d r a w f ro m e x i s t i n g p a r t i c i p a n t constituencies in their communities, and use their digital literacy programming to attract new target populations.

➜➜ Personal recommendation and reputation play a significant role in attracting target populations to KEYSPOTs.

➜➜ K EYSPOT pa r t icipa nt s pr i ma r i ly come f rom t he Nor t h, Sout hwest , a nd West Ph i ladelph ia nei g hborhood s t hat t he Pa r t nersh ip i n it ia l l y targeted.

Broadband Adopter Profile

➜➜ Most KEYSPOT participants do not subscribe to broadband at home, primarily due to cost barriers.

➜➜ For those able to afford home broadband, home subscriptions do not necessarily equate with use.

➜➜ In addition to cost issues, participants just starting out at KEYSPOTs are fearful of technology.

➜➜ Par ticipa nts a lso star t out at K EYSPOTs w ith significant (non-digital) learning challenges.

➜➜ De spite fe a rs a nd le a r n i n g cha l len ge s related to technolog y, K EYSPOT participants feel the Internet and digital literacy are highly relevant to their daily lives.

Demographic CharacteristicsKEYSPOTs primarily serve African Americans, and age and gender are well represented across KEYSPOT users.

The Workstation User Sur vey (WUS) demonstrates that eight out of ten adult participants identified as African-American or Black. Nearly half of all W US respondents were African-A merican women (47 percent). Fourteen p e r c e n t i d e n t i f i e d a s b e i n g o f H i s p a n i c e t h n i c i t y. Approx imately 8 percent of respondent s identif ied a s White with 3 percent identif ying as multi-racial.

Across the Partnership, gender was evenly represented bet ween women a nd men . Fi f t y-si x percent of W US respondents identified as women, 42 percent as men, and 2 percent as “other or self-defined.” An examination of WUS data by each Managing Partner shows that some partners ser ve most ly women or most ly men. For exa mple, t he Drexel/Community College of Philadelphia/Philadelphia Hou si n g Aut hor it y prog ra m over w hel m i n g l y ser ve s women, wherea s YOACA P a nd Ph i ladelph ia FIGHT host sites see primarily male participants.

K EYSPOT computer centers and trainings attract participants from a wide range of ages. From the W US data, the average age of adult respondents was 42 , with a ra nge from 18 to 98 . The avera ge a ge of Ex it Su r vey respondents was slightly older at 51, with a range from 18 to 98.

Who participates at KEYSPOTs?Q:

1aWUS/SES GIS FG: Participants FG: Webguides Interviews: Managing Partners Interviews: Primes Interviews: Host Sites Interviews with Allied Orgs. Primes’ Report to NTIA

123456789

Sour

ces

Use

d

34

KEYSPOTs draw from existing participant constituencies in their communities, and use their digital literacy programming to attract new target populations.

B ec au se pa r t ners repre sent a nd col la borate w it h diverse ser vice organizations, they reach constituencies who a lready tr u st a nd v isit t hose institutions, such a s people who are homeless, veterans, under/unemployed, disabled, former prisoners reentering societ y, living with HIV, and youth. In inter views, staff at several Managing Partners and the Primes reported that the placement of K EYSPOTs helped t hem reach t a rget popu lat ions i n surrounding areas. One staff inter viewee at a Managing Pa r t n e r u s e d t h e b u s i n e s s t e r m o f “ w a r m - m a r k e t ” relationships, saying, “It's easier to implement a new project or a new technolog y if you already have relationships. We already had the population to be ser ved, and they were already comfortable in our environment.” A staff member at another partner organization concurred, “It’s really the community organizations that know their people the best. In that way, you’re able to have a wide reach and a deep meaningful reach.”

In a small, though notably vocal number of instances, inter view and focus group material revealed the way in which neighborhood dynamics factor into partners’ ability to meet target popu lations. One A llied Organization’s

st a f f person sa id she tr ied to implement a prog ra m at a neighborhood K EYSPOT, but wa s u nable to attract t h e d e s i r e d t a r g e t a u d i e n c e b e c a u s e o f u n s p o k e n neighborhood boundaries. She said, "Philadelphia really is one of those places that they say is a city of neighborhoods. If you walk five blocks in one direction, you're in a ver y d i f ferent nei g h bor hood . S ome p eople d id n't l i ke to le ave t hei r neig hborhood , so ma k i n g t hat l i n k d id n't wor k .” K now i n g one’s nei g h b or ho o d i n t he pro c e s s of sit i n g PCCs is equ a l ly i mpor t a nt . One Webg u ide noted the changing demographics in his neighborhood and commented, “ The choosing of the sites is critical. . .does the communit y really need or want this? I felt l ike [my K EYSPOT] was in a neighborhood that was in an upswing… and didn’t really need those ser vices.”

Personal recommendation and reputation play a significant role in attracting target populations to KEYSPOTs.

Nearly one third of Exit Survey respondents learned about K EYSPOTs throu gh word of mouth. In focu s groups, participants frequently spoke of being recruited as well as recruiting friends and family to K EYSPOT trainings. Inter views with staff revealed similar information. As a staff person at a Managing Partner elaborated, “In this

20 40 60 80 100%

MaleFemale

YOACAP

OIC

PEC

FIGHT

ODAAT

MMP

FLP

Drexel/CCP

ExitWUS

65

55-64

45-54

35-44

25-34

18-24

100% 20 30

Managing Partner & Gender Distribution(Long W US, n=3062 )

Age Distribution(Long W US (n=2897) vs. Exit Survey (n=81))

35

subculture here, word travels fast and a person's word is the best advertisement you got. Knowing that, all I needed was a few people to have a few good experiences.” Public relations efforts, such as newspaper ads and f lyers, played a moderately significant role in attracting participants: a c c o rd i n g t o t h e E x i t S u r ve y, n e a r l y 25 p e rc e n t o f respondents said they learned about their training through newspaper ads or f lyers.

KEYSPOT participants primarily come from the North, Southwest, and West Philadelphia neighborhoods that the Partnership initially targeted.

According to the map of KEYSPOT W US respondents, high concentrations of KEYSPOT users came from North,

Southwest , a nd West Philadelphia (see p. 36). Du ring the initial grant implementation stage, the Partnership targeted these areas for K EYSPOT placement based on low broadband subscription rates. When comparing the map of W US respondents to the geographic distribution of broadband subscription rates according to FCC data, similarities appear. This indicates that K EYSPOTs are reaching areas with low broadband adoption rates.

Broadband Adopter ProfileMost KEYSPOT participants do not subscribe to broadband at home, primarily due to cost barriers.

As indicated by WUS and Exit Survey data, approximately one t h ird of K EYSPOT pa r t icipa nt s repor ted hav ing Inter net at home (E x it Su r vey: 3 6 percent , W US: 33 p erc ent). T he e x a m i n a t ion of home su bsc r ipt ion by age, ethnicit y, or race yielded no statistical differences. How e ve r, g e n d e r a p p e a r e d t o c o r r e l a t e w i t h h o m e broadband adoption: W US data showed that women are more likely than men to have Internet at home (36 percent versus 27 percent).

The exa mination of home subscription by t y pe of KEYSPOT involvement revealed that training participants were more l i kely to have home Inter net (39 percent) than participants who frequent K EYSPOTs merely for computer and Internet use (27 percent), such as to search for a job, study or learn, or read information online.

C o s t f o r h a rd w a r e a n d c o n n e c t i v i t y s e r v e d a s a s i g n i f ic a nt b a r r ier to home bro a d b a nd u s e . W US respondents cited lack of a computer (57 percent) a nd cost (30 percent) as the most common deterrents to home Internet access. Cost issues were ev ident a mong those respondents who selected “other” barriers (6 percent) in relation to home Internet subscription. Write-in answers revealed that homelessness or not having one’s own home was another key factor for lack of home Internet. No WUS respondents selected the option “It’s a waste of time” to explain lack of a home subscription.

Qu a litative f ind ings a lso conf irmed cost ba rriers .

Percentage ofParticipants Who Have Home Internet by Select Characteristics (Long W US)Gender* Women* 35.6%

Men* 27.2Race Afr.Amer/

Black32.7

Non-Afr.Amer/Black

32.6

Ethnicity Latino 32.8Non-Latino 32.3

Age 18-24 2925-34 32.335-44 35.745-54 24.555-64 41.9Over 65 28.6

Reason for coming*

Training* 38.9

Other reasons (email, job, study, etc)*

27.0* Difference between these characteristics is significant at p<0.05.

36

Percent of WUS respondents by ZIP Codes(W US, n=2868)

KEYSPOT

37

In a l l inter v iews w it h st a f f at Ma na g ing Pa r t ners a nd across a l l th ree pa r ticipa nt focu s groups, cost wa s the primar y reason cited by participants for failing to adopt broadband in the home. In one of the participant focus groups, a participant said, “I ’m [on] a ver y strict budget and I thought to tr y it . But it’s k ind of difficult .” Many pa r t icipa nt s str u g gled to meet ba sic need s . A s a st a f f person at a Ma na g i n g Pa r t ner descr ibed , “ People a re just sur viving. They're not l iv ing. They're not existing. They're just sur viving. If you're talk ing about Maslow's hierarchy of needs—food, clothing, shelter, and water—it's not even met.”

For those able to afford home broadband, home subscriptions do not necessarily equate with use.

Half of focus group participants who mentioned having home Internet or personal computers before frequenting a KEYSPOT admitted to not using these tools. Common a mong t hese pa r t icipa nt s wa s t he stor y t hat a lack of digital literacy skills limited their use of the technolog y. They reported having purchased subscriptions in the home for their children and grandchildren, not for themselves. As one participant said, “I just want to become computer-literate because ever y time I need something done, I need to go to my grandkids, or one of my children, to ask them to do things for me. They always say, ‘You need to learn computers, so you don't have to sit and wait for somebody to do somet h i n g for you .’ ” I n t he s a me foc u s g roup, another participant said, “I a lso had mine years before, because whenever my children, or my grandchildren come over, they always bring their computers. So they need to be on the Internet.”

In addition to cost barriers, participants just starting out at KEYSPOTs are fearful of technology.

Many K EYSPOT training participants come with little or no k nowledge about computers and the Internet. In t wo out of three focu s groups w ith pa r ticipa nts, f irst-time computer users expressed feeling extremely fearful of computers . De scr ibi n g h i s i n it ia l ex per ience w it h computers, a participant shared, “I used to shake when I f irst started.. . Man—it was rough.” Webguides regularly

obser ved pa r t icipa nt s’ t repid at ion. A s one Webg u ide said, many new users worr y “that at the touch or click of a button something will blow up and explode. It’s just so much power in that one button.”

Participants also start out at KEYSPOTs with significant (non-digital) learning challenges.

At t wo Managing Partners, one Prime, and one A llied O r g a n i z a t i o n , s t a f f i n t e r v i e w e e s h i g h l i g h t e d l o w l iterac y sk i l ls , or t he i nabi l it y to read , a s a ba r r ier to becoming digitally literate. One staff person at an Allied Organization focused on adu lt l iterac y said, “ We have such a high level of literacy issues in the city [and] how do you address that when you're also tr ying to address needs around digital capabilities?”

Despite fears and learning challenges related to technology, KEYSPOT participants feel the Internet and digital literacy are highly relevant to their daily lives.

A majorit y of focus group attendees stressed the critical need to learn or stay up-to-date with changing technology and the Internet. As one participant stated, “If you don’t know computers, you are lost.” Another participant added, “In today's societ y, you absolutely need a computer. That was one of the things I was tr ying to impress [on] new people where I l ived , that ou r l ives a re ju st computer-oriented now, whether we like it or not.” Ot hers expressed t he releva nce of d ig it a l l iterac y in terms of self-suf f icienc y and f u lf i l lment of persona l goals. Focus group conversations revealed the importance of de ve lopi n g d i g it a l sk i l l s to avoid de p endenc y on grandchildren or members of a younger generation for I nt e r ne t- re l a t e d c om mu n ic a t ion s a nd t r a n s a c t ion s . A n o t h e r s e t o f p a r t i c i p a n t s d i s c u s s e d r e a s o n s f o r enrollment in a training, including getting up to speed w ith secure online bank ing and improving technolog y sk ills in order to advance at work.

38

➜➜ While participants use KEYSPOTs primarily for training and workforce development, they also have many varied reasons for frequenting KEYSPOTs.

➜➜ Apa r t f rom d ig it a l l iterac y a nd computer a nd Internet access, participants access a wide range of programs while visiting KEYSPOTs.

➜➜ KEYSPOT participants are very satisfied with how programs meet their needs.

While participants use KEYSPOTs primarily for training and workforce development, they also have many varied reasons for frequenting KEYSPOTs.

When asked to pick one main reason for coming to the K E YSPOT, 4 2 percent of W US su r ve y re spondent s selec ted t ra i n i n g a nd 17 percent selec ted job -related efforts. Participants were less likely to cite other reasons such as “to use a social networking site” or watching online media as their primar y reason for attending a KEYSPOT.

Qu a l it at ive d at a echo su r vey resu lt s . Ni ne out of a tot a l of e le ven i nter v ie wee s at Ma n a g i n g Pa r t ner s and a majorit y of inter viewees at host sites highlighted how job readiness and f inding work depends on access to t e c h nolo g y a nd d i g it a l l i t e r a c y. A s a K E YSP OT host site staf f member sa id , “It’s ra re that you ca n ju st g o s om e w h e re a n d s u b m i t a [ j o b] a p p l i c a t i on r i g h t t here on t he spot—ever ybody tel ls you to go on l i ne.” Participants and Webguides also emphasized workforce development and basic computer skills as key motivators for attendance. In all three focus groups with participants, individuals shared stories of using KEYSPOTs for a range of work force development activities, including w riting

resumes, searching or applying for jobs, gaining skills for a current job, starting or promoting a business, bartering ser vices, and joining LinkedIn.

T h e p r i m a c y o f w o r k f o r c e - r e l a t e d u s a g e a s i d e , participants also shared many other motivations in focus groups. One participant noted several goals met by the K E YSPOTs , “ Fi l l i n g out [job] a ppl ic a t ion s , f i nd i n g [fa mily] that I ’ve been look ing for for a long time, and getting my schoolwork done. Tr ying to get my education, my GED. It helps me accomplish a lot of things.”

One focu s g roup pa r ticipa nt sha red a stor y about addiction and the role of the K EYSPOT program in his recover y. K EYSPOTs helped this individual to apply to universit y and subsequently become a certified recover y specia list (explained in more detail in Goa l 3b). While spea k ing , this pa r ticipa nt a lso described a deeper role that the KEYSPOT plays in his life: “This computer class opened up a lot of doors and allowed me to exercise my brain. I mean we’re talk ing about somebody who came from thirty years of addiction. My plan was only to go [to treatment] and just try to stop [using], not knowing that it would work... But filling the void with the computer class, hav ing something to do to motivate me… [It] cha nged my life.”

Apart from digital literacy and computer and Internet access, participants access a wide range of programs while visiting KEYSPOTs.

Host sites fulfil l diverse participant needs with a range of ser v ices, such a s work force development progra ms, educational trainings, case management, substance abuse recover y, hea lth education, media advocac y, and youth

What participant needs are met through KEYSPOTs?

Q:1b

WUS/SES GIS FG: Participants FG: Webguides Interviews: Managing Partners Interviews: Primes Interviews: Host Sites Interviews with Allied Orgs. Primes’ Report to NTIA

123456789

Sour

ces

Use

d

39

leadership programs. All host site inter viewees discussed how they were able to refer participants from core services of t he orga nization to K EYSPOT ser v ices . Across a l l three focus groups, participants spoke about discovering digital l iteracy trainings while benefiting from another social ser vice, and vice versa. Exit Sur vey data also speak to t he impor ta nce of onsite promotion: 32 percent of respondents reported learning about a KEYSPOT through the host organization itself.

S i m i l a r l y, t h e K E YS P OTs e x p o s e d c o m m u n i t y members to the organizations’ other offerings. All partners i nter v iewed sa id const it uent s were d raw n to labs a nd subsequently connected to other organizational programs. A staff person at one host site said the KEYSPOT helped engage youth, facilitating other programs such as a sports progra m or a mov ie night. Meanwhile, a staf f member at a Managing Partner said, “[A person] may not stop to get tested [through our STD program], but they’ l l stop for a computer training. They’ ll stop to find about GED classes. Well, we didn’t have this before the Partnership.” A not her Ma na g i n g Pa r t ner i nter v ie wee s a id , “[T he] goal is to use technolog y in a smart and relevant way so that, within the sevent y K EYSPOTs... in the multitude of nonprof it a gencies we’re work ing w ith, [technolog y is] advancing their ow n anti-povert y ef forts, instead of being this separate thing where people could just take a computer class and there’s no other context.”

KEYSPOT participants are very satisfied with how programs meet their needs.

Of Ex it Su r vey respondents, 90 percent sa id that they learned what they wanted to learn. Additionally, 86 percent of respondents sa id that they wou ld come back to the center, compared to 9 percent unsure if they would. This f inding is a lso ref lected in the W US data which shows that 60 percent of su r vey respondents had been to the host site before. An overwhelming number of respondents (95 percent) said that they would recommend the center to fa mi ly or f r iend s. Th is mirrored ent hu sia stic focu s group responses, with all participants consistently saying that they would recommend K EYSPOTs and a few even sharing stories of family and friends that they had already recruited.

A lso, in each pa r ticipa nt focu s group, at lea st one pa r t icipa nt expressed interest in lea r n ing more about K E YSP OTs lo c a t ion s i n t he c it y, prompt i n g g rou p discussion about K EYSPOT map distribution. In t wo of the three focus groups, participants shared their printed maps with others who did not k now about the maps and wanted more information.

Take a class

Study

Get news online

Look for a job

HealthVisit govt site

Banking online

E-mail

Blogging or journal

Social site

Participate in community

Other

Media (games, music, video)

Shop online

Why are people using KEYSPOTS?

(Long W US, n=531)

40

➜➜ KEYSPOTs provide an essential access point to the Internet.

➜➜ W h e n l e a r n i n g n e w c o m p u t e r a n d I n t e r n e t skills, participants rely on supportive KEYSPOT Webguides who use creative teaching strategies and alleviate participant anxieties.

➜➜ KEYSPOTs increase the digital literacy skills of participants and stimulate an interest in continued learning.

➜➜ Though Primes’ Quarterly Reports to the NTIA indicate the FR P as not yet having met its SBA broadband goals, quantitative and qualitative data suggest KEYSPOT participants credit trainings with inf luencing them to buy home subscriptions.

➜➜ For those participants who own home computers and have Internet subscriptions prior to attending a KEYSPOT, trainings help them use these items.

KEYSPOTs provide an essential access point to the Internet.

A lmost seven out of ten Exit Sur vey respondents access the Internet at a KEYSPOT, public library, or other public access point. Half of all Exit Sur vey respondents reported ut i l izi ng on ly one way to access t he Inter net , a nd for 30 percent, they only access public computers or labs to get online.

More than half of all WUS respondents said that they go online everyday (55 percent). Only 7 percent said that they never go online. Those who had Internet at home (33 percent) were much more likely to use Internet everyday (70 percent) compared to those who did not have it at home (47 percent). Of those who did not have Internet at home, nearly

half of respondents used the Internet everyday.

When learning new computer and Internet skills, participants rely on supportive KEYSPOT Webguides who use creative teaching strategies and alleviate participant anxieties.

In focus groups, participants repeatedly underscored the importance of a patient, knowledgeable, and fun trainer.

From the Webguide perspective, nine out of thirteen instr uctors d iscu ssed t heir strategies for dea l ing w it h learning cha llenges in the classroom, such as trainings that consist of students with mixed skill levels. Mixed skill level classrooms risked alienating able or advanced learners and leaving beginners behind. One participant explained the difficulties of being in a mixed skill classroom, “When [the trainer] gives an assignment, [one of the advanced students is] just sitting there and [the trainer’s] like, ‘Oh you finished already over here?’ Well, I feel intimidated because I’m still asking questions.” Meanwhile, advanced students find it frustrating to wait for beginners to catch up, preventing the class from moving for ward.

O ver t ime, Webg u ides increa sed their u se of peer teaching to keep advanced users from getting bored and supporting those with lower skill levels. One participant noted how her trainer avoided this issue by schedu ling with students and “once the class was f i l led up, the class wa s closed . So ever yone sta r ted together.” Webg u ides a lso dealt with this challenge by letting quick-learning s t u de nt s pro g re s s to t he ne x t t a sk , w h i le prov id i n g ha ndout s or one-on-one suppor t to t hose st r u g g l i ng.

What are the effects of KEYSPOT access and training on broadband adoption?

Q:2

WUS/SES GIS FG: Participants FG: Webguides Interviews: Managing Partners Interviews: Primes Interviews: Host Sites Interviews with Allied Orgs. Primes’ Report to NTIA

123456789

Sour

ces

Use

d

41

KEYSPOTs increase the digital literacy skills of participants and stimulate interest in continued learning.

Focus group data demonstrate the cumulative effects of digita l l iterac y: while a l l pa r ticipa nts sha red stories of digital literacy progress, a third of participants specifically a r t icu lated excitement a nd ea ger ness to expa nd t hei r basic skills and take other classes. One participant shared, “Once I learned how to create a document, I got excited about it . [I t hou ght ,] ‘ Th is isn’t t hat ha rd! ’ ” Sim i la rly, a not her pa r ticipa nt expressed , “I k now t he d if ference bet ween the modem and the keyboard , a nd the screen now. I lea rned that . Now I wa nt to k now more, so I ’m tr ying [a typing program].” Other digital skills of interest included different blogging platforms, web design, and intermediate/advanced computer classes.

Inter v iewees at Ma na g i n g Pa r t ners a lso obser ved p osit i ve c h a n g e s i n s t u de nt pro g re s s . “ [ T here were] i n d i v i d u a l s w h o h a d b e e n i n c a rc e r a t e d f o r t w e n t y-some years, so they weren't even introduced to personal computers or laptops or smartphones,” said a staff member at a Ma na g i n g Pa r t ner. De scr ibi n g t he exper ience of

formerly incarcerated individuals as they confronted new technolog y, he explained, “Once they came out, they were like, ‘Wow.’ To be part of the program—they learned how to t ype, they learned how to maneuver and get online for the first time. It was huge.”

Though Primes’ Quarterly Reports to the NTIA indicate the FRP as not yet having met its SBA broadband goals, quantitative and qualitative data suggest KEYSPOT participants credit trainings with influencing them to buy home subscriptions.

As reported in the previous section, Progress Towards N TI A Deliverables , t he SBA progra m ha s reached 19 percent of its target for household subscriptions and 34 percent of business broadband subscribers.

Though subscription targets are below target, some OTI data suggest that K EYSPOT use does directly lead to pu rcha se of home Inter net subscr ipt ions . In focu s groups, five participants (out of a total of eighteen queried) said they acquired a subscription following a K EYSPOT t ra i n i n g. Thou g h out of reach for ma ny pa r t icipa nt s , home subscriptions were an object of interest . Of Exit Sur vey respondents who do not have Internet at home, 87 percent said the training made them want to get Internet at home, and 45 percent said they are “ver y likely” to sign up for Internet at home in the next six months, indicating fairly high levels of interest.

Internet Access Locations (Exit Survey, n=102)

Frequency* Percentage

Public Access

70 68.6

Home** 54 52.9Work/school

22 21.6

Anywhere, using a smartphone

18 17.7

Cafe/restaurant

13 12.8

* Responses do not add up to 100% because participants can select multiple choices.

** Of those who said they used Internet at home, 25% of them also reported accessing the Internet anywhere using a smartphone. This may explain why a higher percentage of people use Internet at home compared to people who say they have Internet at home (36%).

Total Number of Internet Access Points(Exit Survey, n=102)Number of Access Points

Frequency Percent

0 7 6.91 52 512 20 19.63 11 10.84 8 7.85 4 3.9

42

For participants who own home computers and have Internet subscriptions prior to attending a KEYSPOT, trainings help them use these items.

A l l pa r t icipa nt s who had computers a nd Inter net at home before at tend i n g a K EYSPOT acqu i red a nd improved their digita l l iterac y sk il ls to use technolog y f o r p e r s o n a l re a s o n s . Fo r e x a m p l e , o n e p a r t i c i p a n t transformed from a non-user to a user with basic Internet a nd computer sk i l l s . De sc r ibi n g her pat h to u si n g a

personal computer, she said, “I got a 12-year-old grandson, and he’d be saying ‘Dang grandma, you bought a computer now, you don't k now how to use it yet?’ I started mak ing h i m sit on a computer a nd do work for me . He' d be show ing me, and then I get mixed up. So I said, ' Well, I 'm going to take a class, and I 'm going to surprise him.’ Well, I am surprising him. I'm not finished.” Several others expressed the same trajector y.

“I’m finding that navigating a computer is much easier because of the [knowledgeable] instructors.”

“I like this [KEYSPOT]

because I love the teachers. They are friendly. They are nice, and they help you with everything. Even though it’s not that many teachers, they try their best to get around to help everyone that is in the room.”

"[The trainers] met everybody at [their] level...that’s a gift you have to have - to have that patience because ... people "just weren’t getting it...[the trainers] took their time and they extended themselves. Like on a day when it wasn’t a computer class, they still said you know

what, we want to come up tomorrow, we’ll show you."

“The instructors are very good. They come to you when you have a problem, and they stay there and try to explain to you – even if they have to come back again. They seem to don't mind. At the end of the night, you know what you need to know.”

Select Participant Quotes about Supportive Digital Learning & Teaching (Focus Groups)

43

➜➜ Participants use KEYSPOTs for job seeking and job preparation.

➜➜ Within a short time span, K EYSPOTs can help participants find jobs.

➜➜ KEYSPOTs help participants keep their job skills current.

➜➜ By v ir tue of h ir ing a cor ps of Webg u ides , t he Partnership is itself an engine of job creation.

Participants use KEYSPOTs for job seeking and job preparation.

As ind icated prev iou sly, “ To look (or apply) for a job ” was the second most common reason for K EYSPOT use a mong respondents to the W US . The Primes’ A nnua l Report to the NTIA for 2012 revealed that a substantial percentage of training hours were devoted to workplace skills. For PCCs, approximately 13 percent of their total training hours were Microsof t Office trainings. About 10 p e rc e nt o f S BA t r a i n i n g s f o c u s e d o n M i c ro s o f t Office, certified training programs like the International Computer Driver’s License, and job readiness trainings. When viewed alongside focus group data, the effects of K EYSPOTs on job seek ing a nd job prepa ration come into focus. Participants stressed the critical importance of learning computer sk il ls when applying for a job.

Job preparedness also featured in obser vations shared by staff in focus groups and inter views. “ Whether it be getting them a n IT essentia ls cer tif ication, or getting them comfortable enough so that they know how to write a cover letter, [or] comfortable sitting in a job inter view. I feel l ike those goals are being met constantly,” a staff

member at a Managing Partner said.Across all three participant focus groups, participants

rout i nely told stor ie s of acce ssi n g a jobs d at aba se or uploading resumes online within the context of becoming more comfortable with computer sk ills, such as using a mouse, t yping, or web surfing. One participant relayed a stor y du r ing a focu s g roup about a n on-site v isit by prospective employers, “I ’ve been out of the work force

What are the effects of KEYSPOT access and training on participants’ employment status?

Q:3a

WUS/SES GIS FG: Participants FG: Webguides Interviews: Managing Partners Interviews: Primes Interviews: Host Sites Interviews with Allied Orgs. Primes’ Report to NTIA

123456789

Sour

ces

Use

d

“I want to really get good on my basics first before I go to the next level of web design or office work, that’s what I want to get into. I want to work.”

“I had a job interview and one of the tests was an Excel test and I passed. I passed the test from remembering... the little [KEYSPOT] class I had to bring me up to date.”

Select Participant Quotes about Workforce-Related KEYSPOT Impacts(Focus Groups)

44

for t wo years now. And I’ve been look ing. When you go [to this K EYSPOT], you get the oppor tu nit y to meet with companies. [One company] came out, and you got to meet directly with the HR person as opposed to having to put it online, and hope that someone contacted you regarding a job.”

Statements that credited K EYSPOTs w ith helping in job seek ing and preparation surfaced in comparisons of K EYSPOTs to a different state-sponsored workforce development resou rce, a nd ot her sim i la r employ ment ser v ices. One pa r ticipa nt sa id , “I needed job coaching that the [state resource] doesn't provide because a lot of [their staff ] do not take time out with you. They just sit there. You have to figure this out for yourself.” Another participant shared that when he tried to apply for a job at the state resource, the woman assisting him left him waiting for some time, “I wasted a whole fort y minutes, tr ying to fill one application and never even got it done.”

Within a short time span, KEYSPOTs can help participants find jobs.

While job seek ing featu red high ly a mong Ex it Su r vey re spondent s (6 0 percent), a bout a t h i rd of t he se job seekers cred ited K E YSPOTs w it h helpi n g t hem f i nd a job. Si m i la rly, i nter v ie ws y ielded fe w stor ie s a bout job acquisition. Three out of nine staf f inter viewees at Managing Partners and three out of five staff interviewees at host sites mentioned the direct effects of their work on employment. For example, one of the inter viewees shared stories about working with the unemployed, “They come on a regular basis, and eventually, they end up getting jobs and moving out, which is the best part.”

KEYSPOTs help participants keep their job skills current.

Focus group data reveal ways in which the employed use KEYSPOTs. One participant explained that because her super visor allowed her to train during the workday, she had rel ied on K EYSPOT t ra in i ngs to per for m bet ter at work. Two participants wanted help improving their bu si ne s s e s , s p ec i f ic a l l y t h rou g h le a r n i n g M ic rosof t Excel and desktop publishing. One shared that she used

the K EYSPOT to create f lyers and advertising materials wh ich helped her acqu ire new cu stomers at her store. Two other pa r ticipa nts empha sized sk i l ls-bu ild ing for new entrepreneurial prospects, such as bartering ser vices through the Internet and using k nowledge acquired at K EYSPOT trainings to help start a business.

By virtue of hiring a corps of Webguides, the Partnership is itself an engine of job creation.

The most recent PCC Prime’s Quarterly Report to the N T I A (Q 4 2 012) re p or te d c re a t i n g or m a i nt a i n i n g 7 7 pa r t-t i me la b a ssi st a nt posit ion s a nd 11 f u l l-t i me ma na ging positions. Similarly, according to data from AR R A reports, the SBA program created or maintained t he equ i va lent of 45 .5 f u l l-t i me posit ion s , i nclud i n g management and training positions.

Qualitative data also illuminate the ways in which the Partnership provided a path to K EYSPOT employment. I n f o c u s g r o u p s , t h r e e o u t o f t h i r t e e n We b g u i d e s ment ioned bei ng u nemployed pr ior to bei ng h i red a s K EYSPOT instructors. Other discussion revealed that the Partnership employed Webguides with var ying levels of digital literacy from diverse backgrounds, such as youth health counselors and librarians. One Webguide spoke about being a KEYSPOT student who was singled out by her instructors to apply for the Webguide positions. She described having deep fears of computer viruses and even did not know how to use a mouse before the training. With the help of her instructor, she not only progressed quickly, but also demonstrated a propensity to help her classmates. A f t e r s he f i n i s he d t he c ou r s e , t he K E YSP OT l a t e r offered her a job as a Webguide. Four staff inter viewees at Managing Partners recounted similar stories of hiring former participants into their organizations.

Ref lecting on the significance of K EYSPOT hires, one Webguide described the impact of such direct job-creation as important “not only for Philadelphians to learn the digital world, but also for us as employees, because the last thing we need to do is add on to the unemployment rate.”

45

➜➜ K EYSPOTs prov ide educationa l trainings and opportunities for adult participants.

➜➜ Few adu lt pa r t icipa nt s sha red stor ies of u sing K E YSPOTs to help t hem ad va nce t hei r ow n education.

KEYSPOTs provide educational trainings and opportunities for adult participants.

According to the Primes’ Annual Reports to the NTIA for 2012 , 5 percent of all PCC training hours focused on GED training and an additional 4 percent of participants focused on college preparation. Approximately 1 percent of SBA trainings and participants were for GED training.

Meanwhile, OTI's quantitative data revealed different lea rning impacts . On t he one ha nd , on ly 5 percent of W US re s p onde nt s s e le c t e d t h i s o p t i o n a s t h e i r m a i n reason for coming (the fourth most popular choice). On the other hand, a sizeable majority (6 8 p erc ent) of E x it S u r ve y t a ker s w ho c omple t e d SBA trainings for educational purposes said trainings helped connect them to educational opportunities.

Additiona l mentions of educationa l oppor tu nities surfaced in both Webguide focus groups and a majority of inter viewees at Managing Partners. For example, several Webg u ides sha red stor ies of yout h a nd adu lt lea rners who frequented K EYSPOTs to study or do research for homework assignments. One Webguide shared, “Parents come in and bring their kids, and then do homework. So I have a homework night.”

Few adult participants shared stories of using KEYSPOTs to help them advance their own education.

Qualitative data reveal some evidence that K EYSPOTs help participants with educational advancement. Across t he t h ree pa r t icipa nt focu s g roups, references to how d i g it a l l iter a c y sk i l l s f u nc t ion a s s teppi n g-s tone s i n advancing through educational programs occurred four times. One participant who came to a KEYSPOT through an addictions recovery program shared a personal story of how Webguides helped him overcome his initial fears of technolog y and encouraged him to enroll in universit y. I n it ia l l y, he wa s a nov ice u ser, s ay i n g t h at he “cou ld barely turn the computer on.” College was never part of his l ife plans, but af ter being in a K EYSPOT program,

t h a t c h a n g e d . “ W h e r e I ’m a t n o w i n m y l i f e t o d a y, I wasn’t pla nning on going to col le ge .   B u t f rom b ei n g i n t hat [d i g it a l l iterac y] cla ss , I t hou g ht about it a nd [t he

teachers] said, ‘You should go for it .’ So I applied.” Now, this K EYSPOT user attends a local universit y with the support of a Pell grant and has a lso been certif ied as a Recover y Specialist. The motivation and encouragement this participant received from supportive Webguides was critical to helping him advance his education.

What are the effects of KEYSPOT access and training on participants’ educational attainment?

Q:3b

WUS/SES GIS FG: Participants FG: Webguides Interviews: Managing Partners Interviews: Primes Interviews: Host Sites Interviews with Allied Orgs. Primes’ Report to NTIA

123456789

Sour

ces

Use

d

“Parents come in and bring their kids, and then do homework. So I have a homework night.” – Webguide

46

➜➜ Though participants do not explicitly state that they seek community through digital literacy and public Internet access, they find community, both on-site and online, at KEYSPOTs.

➜➜ KEYSPOTs function as a home away from home that often provides a safe space for participants.

➜➜ KEYSPOTs foster an environment of peer learning, giving participants the opportunity to connect with one another.

Though participants do not explicitly state that they seek community through digital literacy and public Internet access, they find community, both on-site and online, at KEYSPOTs.

Community-oriented KEYSPOT usage trailed workforce, education, a nd sk i l ls-ba sed pu rposes: only 4 .5 percent o f W U S r e s p o n d e n t s s e l e c t e d “s o c i a l n e t w o r k i n g ” a n d 2 .5 p e rc e n t s e l e c t e d c o m i n g “ t o p a r t i c i p a t e i n t hei r com mu n it y” a s t hei r ma i n rea son for at tend i n g a K E YS P OT. A c r o s s a l l p a r t i c i p a n t f o c u s g r o u p s , however, according to several impassioned participants, communit y helped motivate attendance at K EYSPOTs. One participant said, “I had some good experiences with the class, bonding with people that had the same thing in common with me. So, I didn't feel out of place, because other people were struggling with the same things I was struggling with.” Many participants spoke about returning to K EYSPOTs on mu ltiple occasions due to the sense of com mu n it y fostered at computer centers . In some inst a nces pa r ticipa nt s compla ined about nois y public computer centers with too much socializing.

Though to a lesser degree, similar stories surfaced in Webguide focus groups. A Webguide explained, “I have

a lot of people coming in just for the communit y. They make friends there.”

Across a l l t h ree pa r t icipa nt focu s g roups , severa l p a r t i c i p a n t s o f f e r e d p o w e r f u l s t o r i e s o f v i r t u a l l y c o n n e c t i n g w i t h f a m i l y, f r i e n d s , a n d c o l l e a g u e s , par ticu larly through using Facebook . One par ticipant shared a stor y of another KEYSPOT student's emotional Fa c eb o ok e x p er ienc e , " We were s it t i n g t here doi n g a le s s on , a nd she got re a l d i s t u r bi n g ne w s f rom her Fa c e b o o k . By h e r b e i n g f r i e n d s w i t h s o m e b o d y o n Facebook, she was able to find out that her stepchildren’s mom had passed away. It was emotional because she was rea l ly close to t hat lady by t a k ing ca re of her k id s . . . I wouldn’t want to learn none of that off the computer. But it’s better to k now than not to k now." One participant rec a l led how Facebook a l lowed her to keep i n touch w ith her chu rch group who had traveled to Ma law i to provide healthcare to local residents. Another participant described a more active use of Facebook during the 2012 election, “I was using my Facebook as sort of a platform, not to tell just people in Philly.. . but people all over the countr y, ‘Go and vote. Do your thing.’”

B ot h Web g u id e fo c u s g rou ps a nd a m a jor it y of inter views with staff members at Managing Partners also revealed a number of detailed accounts of how participants c o n n e c t e d w i t h o n l i n e v i r t u a l c o m m u n i t i e s . O n e Webguide spoke about a Spanish-speak ing participant who applied newly minted computer sk ills to use Skype a nd reach out for t he f irst t i me in yea rs to her son in another countr y. A staff person at a Managing Partner said, “ We had [a stor y] of a father getting on Facebook and communicating with his children for the first time in t went y years.”

What are the effects of KEYSPOT access and training on community engagement?

Q:4

WUS/SES GIS FG: Participants FG: Webguides Interviews: Managing Partners Interviews: Primes Interviews: Host Sites Interviews with Allied Orgs. Primes’ Report to NTIA

123456789

Sour

ces

Use

d

47

KEYSPOTs function as a home away from home that often provides a safe space for participants.

I n i n t e r v i e w s a n d f o c u s g r o u p s , t h e c r e a t i o n o f a comfor table space for learning su rfaced as a recu rrent theme. An inter viewee from a host site stated, “ We [tr y to be] a welcoming place [where] they can feel at home... I think that's a big part of learning—that you're comfortable learning and there's a sense of order and accomplishment.” In focus groups, participants shared six prominent stories that revealed that the KEYSPOT “was a ver y relaxed and supportive environment.”

K E Y S P O Ts a l s o o c c a s i o n a l l y f u n c t i o n e d a s sanctuaries that sheltered individuals from the problems or dangers in their daily lives. Focus group conversation yielded several accounts of this, both from the perspective o f p a r t i c i p a n t s a n d We b g u i d e s . F o r e x a m p l e , o n e pa r t ic ipa nt i n t r a n sit ion a l hou si n g s a id , “ E sp ec ia l l y when I ’m st ressed out f rom work or somet h i ng , I u se [the K EYSPOT] as a respite.” One Webguide brought a video game console to a KEYSPOT as a reward for youth who completed their homework, saying , “Parents don’t want [their children] out too late and traveling around the area. So [many] times, you get a lot people that come through the [KEYSPOT]... for a total of five hours. They love the games.”

KEYSPOTs foster an environment of peer learning, giving participants the opportunity to connect with one another.

Qualitative data show the significance of peer learning—b o t h s p o n t a n e o u s l y g e n e r a t e d a n d i n t e n t i o n a l l y st r uc t u red—i n c re at i n g a sen se of belon g i n g a mon g participants. The theme of giving and receiving help from classmates throughout the learning process came up in all three participant focus groups. Participants spoke about finding “ buddies” who exchanged class material during absences and shared computer and Internet k nowledge with one another. “K EYSPOTs are people-friendly. You get to meet dif ferent people from a ll wa lks of l ife that want to help you,” one participant said.

In addition to spontaneous moments of peer support, five out of thirteen Webguides mentioned in focus group

conversation the use of peer lea rning exercises . Of ten they introduced peer learning as a way to levera ge the sk i l ls of adva nced st udent s for t he pu r pose of a id i n g le ss k now led ge able one s . One Webg u ide e st abl ished a “ b a r t e r i n g p r o g r a m ,” b a s e d o n h i s k n o w l e d g e o f participants, to deal with different sk ill levels. Younger par ticipa nts paired w ith older par ticipa nts to improve their digital skills. In return, the older participants taught non-digital lessons to the younger participants like playing the piano. According to the Webguide, this technique connected students, helping to bridge generational and digital divides simultaneously.

48

➜➜ The Partnership increases the staffing capacity of Managing Partners and host sites through planned professional development activities.

➜➜ K EYSPOT trainings increase the digita l sk il ls of org a n i z at ion s i nter na l a nd ex ter na l to t he Partnership.

➜➜ BTOP funding allowed partners to upgrade their IT, freeing up time and resources to better ser ve target populations.

➜➜ Collaboration in the form of participant referrals, resource sharing, and networking increased partners’ overall capacity to improve service delivery.

➜➜ Pa r tners u se K EYSPOTs to f u r ther their ow n organizational goals, specifically for employment, education, and community engagement.

The Partnership increases the staffing capacity of Managing Partners and host sites through planned professional development activities.

A c c o r d i n g t o f i v e i n t e r v i e w e e s a t t h e P r i m e s a n d M a n a g i n g P a r t n e r s , t h e P a r t n e r s h i p p r i o r i t i z e d opp or t u n it ie s for prof e s s ion a l de ve lopme nt a mon g frontline staf f. A couple Webgu ides a nd a majorit y of host site inter viewees praised the “ Trainer Roundtables” or the “Sustainability Forums,” which were opportunities for Webguides and staff at Managing Partners, host sites, and Allied Organizations to net work, share experiences and resources, and discuss the issues surrounding digital inclusion efforts and how to sustain the work.

However, across both focus groups, several Webguides d e s c r i b e d p r e s s u r e s t o t a ke o n m u l t i p l e ro l e s w i t h insufficient capacity to do so. One Webguide said, “[With] this grant, the onus [is] placed on the instructor to be a curriculum developer, a teacher, a program administrator, and an outreach specialist . That’s too much to do when you’re supposed to be at a site for t went y hours a week.” Another added that he juggles the role of “ human filter” for any inappropriate content participants might tr y to look up while simultaneously tr ying to manage an entire room of people and their needs.

KEYSPOT trainings increase the digital skills of organizations internal and external to the Partnership.

A vocal minorit y of inter viewees emphasized how they or t hei r orga n i z at ion s took adva nt a ge of K EYSPOT of fer i n gs to i mprove t he d ig it a l l iterac y of t hei r ow n staff, despite the fact that KEYSPOTs’ primar y audience is nov ice u sers . One st a f f i nter v ie wee at a Ma na g i n g Partner reported that one of the KEYSPOTs he manages h a s been a ble to t ra i n t hei r ow n st a f f a nd volu nteer board through the FR P and as a resu lt improved their capacity. Another inter viewee from a different Managing Pa r t n e r r e c o u n t e d a s i m i l a r s t o r y, e x p l a i n i n g t h a t her orga nization gives “ou r development tea m cla sses o n P h o t o s h o p b e c a u s e i t h e l p s w i t h s o m e o f t h e p u b l i c a t i o n s t h a t t h e y p r o d u c e . ” O n e A l l i e d O r g a n i z a t i o n i n t e r v i e w e e d i s c u s s e d h o w a c c e s s t o c o m p u t e r s a n d t r a i n i n g s t h r o u g h the KEYSPOT programs supported the members’ work to a m e l i o r a t e l i v i n g c o n d i t i o n s i n a h i g h - c r i m e ,

In what ways has the FRP increased partners’ capacity?

Q:5

WUS/SES GIS FG: Participants FG: Webguides Interviews: Managing Partners Interviews: Primes Interviews: Host Sites Interviews with Allied Orgs. Primes’ Report to NTIA

123456789

Sour

ces

Use

d

49

impoverished area of Philadelphia. She also described how K EYSPOT resources critically supplement her group’s programs.

BTOP funding allowed partners to upgrade their IT, freeing up time and resources to better serve target populations.

BTOP f u nd ing focu sed Ma na g ing Pa r t ners’ a nd host sites’ attention to technolog y improvements, a l low ing a l l sta keholders to better ser ve their constituents . A ll se ven i nter v ie w s w it h s t a f f a t hos t s ite s emph a si z ed how involvement in t he FR P a l lowed t hem to upd ate technolog y and enhance IT support. Such improvements reduced staff time spent on troubleshooting. “We’ve had about a dozen computers . We ba sica l ly have the sa me number. It’s just that we have ones that work now,” said one host site staff member. A few host site staff inter viewees recounted the ways in which upgrades made participant use of computers more productive. Staff inter viewees at two Managing Partners also mentioned the benefits of IT upgrades. However, during focus groups, some Webguides noted a lack of adequate space, computers, and printers to accommodate program participants.

Collaboration in the form of participant referrals, resource sharing, and networking increases partners’ overall capacity to improve service delivery.

The Partnership Visualization (next page) demonstrates more t ha n 4 0 0 l i n k a ge s bet ween 2 0 8 org a n i z at ion s . These linkages represent instances where organizations made referrals, promoted ser vices, shared resources, and conducted outreach events.

Referra ls bet ween K EYSPOTs helped participants access needed ser vices. As revealed in one focus group, a Webg u ide took t he i n it iat ive to est abl ish a refer ra l system, allowing a handful of K EYSPOTs to easily point participants to other sites’ ser vices, digital or other wise. Ad d i t i on a l l y, a m a j or i t y of We b g u i d e s a c ro s s b ot h focus groups talked about mak ing and benefiting from referrals. Five staff members at Managing Partners also shared stories of referring participants to other sites for

ser v ices. A Managing Par tner inter v iewee stated, “ We have a number of individuals who are tr ying to get their GED, so we refer them into [a partner’s] GED program. [Me a nw h i le] we're doi n g a computer t ra i n i n g [at t he pa r tner’s site], [so t he col laboration] helps u s , it helps [t he pa r t ner], a nd it helps t he i nd iv idu a l .” In a not her interview, a staff member at a Managing Partner described how collaboration with a nearby host site helped bolster attenda nce. However, no systematic, par tnership-w ide referral system grew out of these efforts.

More generally, managing Partners shared physical, human, and technical resources to improve their ser vices. From the informal sur veys used to create the Partnership Visu a l iz at ion, stor ies reg u la rly su r faced about sha red spaces for computer trainings a nd specia l events, such as sites for regular SBA training and pop-up K EYSPOT labs . One st a f f member at a Ma na g i ng Pa r t ner spoke about loaning equipment due to a shipment delay at the beginning stages of the Partnership. Staff at Managing Pa r t ners a lso referred tra iners a nd volu nteers to each o t h e r ’s p r o g r a m s a s n e e d e d ; t h i s r e s u l t e d i n s o m e Webguides finding multiple employment opportunities bet ween several Managing Partners. Technical resources that partners shared included English language learning and GED curricu la , youth policies for public labs, and software licenses for career readiness computer programs.

Ex terna l to t he FR P, Ma na g ing Pa r tners enga ged in collaborations w ith other communit y organizations t o i n c r e a s e s e r v i c e d e l i v e r y. O n e s t a f f p e r s o n a t a Managing Partner said that the BTOP grants helped his organization expand its services throughout Pennsylvania after collaborating with a local partner with connections t h rou ghout t he state, “ We've been able to rea l ly grow what we do. We've been able to do a lot with more people and more organizations.” The Partnership Visualization also maps the less formal collaborations that took place bet ween partners and a wide array of non-BTOP funded groups—including nonprofits, Community Development Corporations, Neighborhood Adv isor y Cou ncils , cit y agencies and commissions, state and federal government-run social services, and a technology-focused media outlet. Work w ith these A llied Organizations introduced the

50

DrexelMMP

NCCF

OICYOACAP

FIGHT

PEC

ODAAT

PPRFLP

CCP

MCOL

UCD

PHA

Mighty Engine

Zivtech

Titan Advertising

Wilco

Mobile Citizen

PYN

ARN

Tech Depot

Knight Foundation

WURD

Generocity

Government in the Lab

Two.One.Five Magazine

Constitution World

the Philadelphia Sunday Sun

Technically Philly

El Sol

Code for America

Temple University

The Philadelphia Tribune

TMCNet.com

Health & Libraries Broadband Coalition

American Library Association

the Online Computer Library Center's Web Junction

Clear Channel Philadelphia

The Republic

The Philadelphia Inquirer

SEPTA

Philly-In-Focus

RISE

Universal Charter School

Philadelphia Youth Network

Philly Rising

Veterans Group Home

St. Josephs University Arcadia

University

St. Josephs SigEp Philadelphia

University

Pennsylvania Prison Society

SWCDC

Upli� Solutions

The Veterans Group

PPS

City Lights

CDC

Global Citizen

JEVS

District Attorneys Community Action Center

UAC

Acer

District Attorneys O�ice

Philadelphia School District

NPower

Colorado BTOPPaLA

PCC

OIT

Food Fit

HPC

OCLC/Webjunction

Black Male Engagement Initiative

FCC

Ford Foundation

Spitfire Strategies

Roxborough HS

Overbrook High School

Sayre High

other high schools

Local 22 (Firefighter's Union)

Prometheus Radio

Broadband meets Broadcast

Head Start Learning Tree

Feltonville Early Childhood Center

Iraq Veterans Against the War

South African ShackDwellers Movement in South Africa

SEIU Hospital and Healthcare Workers PA

West Philadelphia Skills Initiative

Point Breeze Organizing

Action United

Juntos: Fuerza

Fight for Drivers Licenses

Asian Americans United

One Love

Put People First! PA

United Workers

Vermont Workers Center

People's History Project ofElkhardt IN

Cambodian Association

NEPA Organizing Center

Philly Cam

Reels for Rights

Senior centers

Children's Hospital 2

Deliverance Baptist Church

CareerLink

Local daycare

Churches

Mid-City Apartments

Children's Hospital

St. Christopher Hospital

Philadelphia Prison System

GALAEI (focused on LGBT Latin@s)

ActionAIDSFaith Advisory Board

Recovery houses

PA Leadership Charter School

AMHA sta�

Bright Hope Baptist Church

Congreso

Consumer Satisfaction Team

Crisis Management Services

Depaul House

Detox Mercy Hospice

Dignity House (combined)

Fresh Start V

Good Sheppard

Hannah's House Inc

Harris School of Business

Impact Earn

Joy of Living (men's group)

Joy of Living (women's group)

Kate's Place (Project HOME)

Kirkbride/Traveler's Aid

Liberty Resources

McAuley House

Mercy Hospice

Mercy Neighborhood Ministries

New Directions for Women

New Start FIR

New Start II

NOVA II (RHD)

OIC Pregnancy Prevention

People R Us

Positive Committee

Prevention Point

Prevention Summit Workshop

Rainbow Room

Ray Homes (Project HOME)

Reed House

Re-Enter

Self Safe Haven

Station House

Stop and Surrender

Straight Inc.

The Next Step

Trevor Campaign for Homeless Women and Children

Waterview Rec Center

West Philly YMCA

Wynnfield Branch, Free Library

Calvary BaptistChurch

African Cultural Alliance of North America (ACANA)

African and Caribbean Business Council

Partnership CDC

Mayors Commission on African and Caribbean Immigrant A�airs

Simpson House Senior Living Center

Child Care Center

Mural Arts Program

IHM Center for Literacy

Welcoming Center

Convention Center

1199c

AFAHO (African Family Health Organization) Asian Arts

Initiative

Brewerytown CDC

Frankford CDC

Golden Star Baptist ChurchLafiya

Family Services

Leeway Foundation

Lutheran Settlement House

PhillyCAMSlought Foundation

University City High School

SW Community Advisory Groups

Nicetown Community Advisory Groups

North Philadelphia Community Advisory Groups

SW Global Times Newspaper

Scoop Newspaper

University of Penn

Li�

NHS

Earn Center

Mayor's Commission on Aging

local schools

Local churches

Local halfway houses

My Brother's Keepers

New Directions

James Logan School

Organ Reed School

Parent U School

Athletic Teen Council

Philly Goes2College

KEYSPOT Managing Partner

Collaborations

Other

SBA SBA+

PCC

PCC

Funding

60+

1-10

10-59

Colla

bora

tions

Project HOME St Columba sta� meeting

DrexelMMP

NCCF

OICYOACAP

FIGHT

PEC

ODAAT

PPRFLP

CCP

MCOL

UCD

PHA

Mighty Engine

Zivtech

Titan Advertising

Wilco

Mobile Citizen

PYN

ARN

Tech Depot

Knight Foundation

WURD

Generocity

Government in the Lab

Two.One.Five Magazine

Constitution World

the Philadelphia Sunday Sun

Technically Philly

El Sol

Code for America

Temple University

The Philadelphia Tribune

TMCNet.com

Health & Libraries Broadband Coalition

American Library Association

the Online Computer Library Center's Web Junction

Clear Channel Philadelphia

The Republic

The Philadelphia Inquirer

SEPTA

Philly-In-Focus

RISE

Universal Charter School

Philadelphia Youth Network

Philly Rising

Veterans Group Home

St. Josephs University Arcadia

University

St. Josephs SigEp Philadelphia

University

Pennsylvania Prison Society

SWCDC

Upli� Solutions

The Veterans Group

PPS

City Lights

CDC

Global Citizen

JEVS

District Attorneys Community Action Center

UAC

Acer

District Attorneys O�ice

Philadelphia School District

NPower

Colorado BTOPPaLA

PCC

OIT

Food Fit

HPC

OCLC/Webjunction

Black Male Engagement Initiative

FCC

Ford Foundation

Spitfire Strategies

Roxborough HS

Overbrook High School

Sayre High

other high schools

Local 22 (Firefighter's Union)

Prometheus Radio

Broadband meets Broadcast

Head Start Learning Tree

Feltonville Early Childhood Center

Iraq Veterans Against the War

South African ShackDwellers Movement in South Africa

SEIU Hospital and Healthcare Workers PA

West Philadelphia Skills Initiative

Point Breeze Organizing

Action United

Juntos: Fuerza

Fight for Drivers Licenses

Asian Americans United

One Love

Put People First! PA

United Workers

Vermont Workers Center

People's History Project ofElkhardt IN

Cambodian Association

NEPA Organizing Center

Philly Cam

Reels for Rights

Senior centers

Children's Hospital 2

Deliverance Baptist Church

CareerLink

Local daycare

Churches

Mid-City Apartments

Children's Hospital

St. Christopher Hospital

Philadelphia Prison System

GALAEI (focused on LGBT Latin@s)

ActionAIDSFaith Advisory Board

Recovery houses

PA Leadership Charter School

AMHA sta�

Bright Hope Baptist Church

Congreso

Consumer Satisfaction Team

Crisis Management Services

Depaul House

Detox Mercy Hospice

Dignity House (combined)

Fresh Start V

Good Sheppard

Hannah's House Inc

Harris School of Business

Impact Earn

Joy of Living (men's group)

Joy of Living (women's group)

Kate's Place (Project HOME)

Kirkbride/Traveler's Aid

Liberty Resources

McAuley House

Mercy Hospice

Mercy Neighborhood Ministries

New Directions for Women

New Start FIR

New Start II

NOVA II (RHD)

OIC Pregnancy Prevention

People R Us

Positive Committee

Prevention Point

Prevention Summit Workshop

Rainbow Room

Ray Homes (Project HOME)

Reed House

Re-Enter

Self Safe Haven

Station House

Stop and Surrender

Straight Inc.

The Next Step

Trevor Campaign for Homeless Women and Children

Waterview Rec Center

West Philly YMCA

Wynnfield Branch, Free Library

Calvary BaptistChurch

African Cultural Alliance of North America (ACANA)

African and Caribbean Business Council

Partnership CDC

Mayors Commission on African and Caribbean Immigrant A�airs

Simpson House Senior Living Center

Child Care Center

Mural Arts Program

IHM Center for Literacy

Welcoming Center

Convention Center

1199c

AFAHO (African Family Health Organization) Asian Arts

Initiative

Brewerytown CDC

Frankford CDC

Golden Star Baptist ChurchLafiya

Family Services

Leeway Foundation

Lutheran Settlement House

PhillyCAMSlought Foundation

University City High School

SW Community Advisory Groups

Nicetown Community Advisory Groups

North Philadelphia Community Advisory Groups

SW Global Times Newspaper

Scoop Newspaper

University of Penn

Li�

NHS

Earn Center

Mayor's Commission on Aging

local schools

Local churches

Local halfway houses

My Brother's Keepers

New Directions

James Logan School

Organ Reed School

Parent U School

Athletic Teen Council

Philly Goes2College

KEYSPOT Managing Partner

Collaborations

Other

SBA SBA+

PCC

PCC

Funding

60+

1-10

10-59

Colla

bora

tions

Project HOME St Columba sta� meeting

51

DrexelMMP

NCCF

OICYOACAP

FIGHT

PEC

ODAAT

PPRFLP

CCP

MCOL

UCD

PHA

Mighty Engine

Zivtech

Titan Advertising

Wilco

Mobile Citizen

PYN

ARN

Tech Depot

Knight Foundation

WURD

Generocity

Government in the Lab

Two.One.Five Magazine

Constitution World

the Philadelphia Sunday Sun

Technically Philly

El Sol

Code for America

Temple University

The Philadelphia Tribune

TMCNet.com

Health & Libraries Broadband Coalition

American Library Association

the Online Computer Library Center's Web Junction

Clear Channel Philadelphia

The Republic

The Philadelphia Inquirer

SEPTA

Philly-In-Focus

RISE

Universal Charter School

Philadelphia Youth Network

Philly Rising

Veterans Group Home

St. Josephs University Arcadia

University

St. Josephs SigEp Philadelphia

University

Pennsylvania Prison Society

SWCDC

Upli� Solutions

The Veterans Group

PPS

City Lights

CDC

Global Citizen

JEVS

District Attorneys Community Action Center

UAC

Acer

District Attorneys O�ice

Philadelphia School District

NPower

Colorado BTOPPaLA

PCC

OIT

Food Fit

HPC

OCLC/Webjunction

Black Male Engagement Initiative

FCC

Ford Foundation

Spitfire Strategies

Roxborough HS

Overbrook High School

Sayre High

other high schools

Local 22 (Firefighter's Union)

Prometheus Radio

Broadband meets Broadcast

Head Start Learning Tree

Feltonville Early Childhood Center

Iraq Veterans Against the War

South African ShackDwellers Movement in South Africa

SEIU Hospital and Healthcare Workers PA

West Philadelphia Skills Initiative

Point Breeze Organizing

Action United

Juntos: Fuerza

Fight for Drivers Licenses

Asian Americans United

One Love

Put People First! PA

United Workers

Vermont Workers Center

People's History Project ofElkhardt IN

Cambodian Association

NEPA Organizing Center

Philly Cam

Reels for Rights

Senior centers

Children's Hospital 2

Deliverance Baptist Church

CareerLink

Local daycare

Churches

Mid-City Apartments

Children's Hospital

St. Christopher Hospital

Philadelphia Prison System

GALAEI (focused on LGBT Latin@s)

ActionAIDSFaith Advisory Board

Recovery houses

PA Leadership Charter School

AMHA sta�

Bright Hope Baptist Church

Congreso

Consumer Satisfaction Team

Crisis Management Services

Depaul House

Detox Mercy Hospice

Dignity House (combined)

Fresh Start V

Good Sheppard

Hannah's House Inc

Harris School of Business

Impact Earn

Joy of Living (men's group)

Joy of Living (women's group)

Kate's Place (Project HOME)

Kirkbride/Traveler's Aid

Liberty Resources

McAuley House

Mercy Hospice

Mercy Neighborhood Ministries

New Directions for Women

New Start FIR

New Start II

NOVA II (RHD)

OIC Pregnancy Prevention

People R Us

Positive Committee

Prevention Point

Prevention Summit Workshop

Rainbow Room

Ray Homes (Project HOME)

Reed House

Re-Enter

Self Safe Haven

Station House

Stop and Surrender

Straight Inc.

The Next Step

Trevor Campaign for Homeless Women and Children

Waterview Rec Center

West Philly YMCA

Wynnfield Branch, Free Library

Calvary BaptistChurch

African Cultural Alliance of North America (ACANA)

African and Caribbean Business Council

Partnership CDC

Mayors Commission on African and Caribbean Immigrant A�airs

Simpson House Senior Living Center

Child Care Center

Mural Arts Program

IHM Center for Literacy

Welcoming Center

Convention Center

1199c

AFAHO (African Family Health Organization) Asian Arts

Initiative

Brewerytown CDC

Frankford CDC

Golden Star Baptist ChurchLafiya

Family Services

Leeway Foundation

Lutheran Settlement House

PhillyCAMSlought Foundation

University City High School

SW Community Advisory Groups

Nicetown Community Advisory Groups

North Philadelphia Community Advisory Groups

SW Global Times Newspaper

Scoop Newspaper

University of Penn

Li�

NHS

Earn Center

Mayor's Commission on Aging

local schools

Local churches

Local halfway houses

My Brother's Keepers

New Directions

James Logan School

Organ Reed School

Parent U School

Athletic Teen Council

Philly Goes2College

KEYSPOT Managing Partner

Collaborations

Other

SBA SBA+

PCC

PCC

Funding

60+

1-10

10-59

Colla

bora

tions

Project HOME St Columba sta� meeting

DrexelMMP

NCCF

OICYOACAP

FIGHT

PEC

ODAAT

PPRFLP

CCP

MCOL

UCD

PHA

Mighty Engine

Zivtech

Titan Advertising

Wilco

Mobile Citizen

PYN

ARN

Tech Depot

Knight Foundation

WURD

Generocity

Government in the Lab

Two.One.Five Magazine

Constitution World

the Philadelphia Sunday Sun

Technically Philly

El Sol

Code for America

Temple University

The Philadelphia Tribune

TMCNet.com

Health & Libraries Broadband Coalition

American Library Association

the Online Computer Library Center's Web Junction

Clear Channel Philadelphia

The Republic

The Philadelphia Inquirer

SEPTA

Philly-In-Focus

RISE

Universal Charter School

Philadelphia Youth Network

Philly Rising

Veterans Group Home

St. Josephs University Arcadia

University

St. Josephs SigEp Philadelphia

University

Pennsylvania Prison Society

SWCDC

Upli� Solutions

The Veterans Group

PPS

City Lights

CDC

Global Citizen

JEVS

District Attorneys Community Action Center

UAC

Acer

District Attorneys O�ice

Philadelphia School District

NPower

Colorado BTOPPaLA

PCC

OIT

Food Fit

HPC

OCLC/Webjunction

Black Male Engagement Initiative

FCC

Ford Foundation

Spitfire Strategies

Roxborough HS

Overbrook High School

Sayre High

other high schools

Local 22 (Firefighter's Union)

Prometheus Radio

Broadband meets Broadcast

Head Start Learning Tree

Feltonville Early Childhood Center

Iraq Veterans Against the War

South African ShackDwellers Movement in South Africa

SEIU Hospital and Healthcare Workers PA

West Philadelphia Skills Initiative

Point Breeze Organizing

Action United

Juntos: Fuerza

Fight for Drivers Licenses

Asian Americans United

One Love

Put People First! PA

United Workers

Vermont Workers Center

People's History Project ofElkhardt IN

Cambodian Association

NEPA Organizing Center

Philly Cam

Reels for Rights

Senior centers

Children's Hospital 2

Deliverance Baptist Church

CareerLink

Local daycare

Churches

Mid-City Apartments

Children's Hospital

St. Christopher Hospital

Philadelphia Prison System

GALAEI (focused on LGBT Latin@s)

ActionAIDSFaith Advisory Board

Recovery houses

PA Leadership Charter School

AMHA sta�

Bright Hope Baptist Church

Congreso

Consumer Satisfaction Team

Crisis Management Services

Depaul House

Detox Mercy Hospice

Dignity House (combined)

Fresh Start V

Good Sheppard

Hannah's House Inc

Harris School of Business

Impact Earn

Joy of Living (men's group)

Joy of Living (women's group)

Kate's Place (Project HOME)

Kirkbride/Traveler's Aid

Liberty Resources

McAuley House

Mercy Hospice

Mercy Neighborhood Ministries

New Directions for Women

New Start FIR

New Start II

NOVA II (RHD)

OIC Pregnancy Prevention

People R Us

Positive Committee

Prevention Point

Prevention Summit Workshop

Rainbow Room

Ray Homes (Project HOME)

Reed House

Re-Enter

Self Safe Haven

Station House

Stop and Surrender

Straight Inc.

The Next Step

Trevor Campaign for Homeless Women and Children

Waterview Rec Center

West Philly YMCA

Wynnfield Branch, Free Library

Calvary BaptistChurch

African Cultural Alliance of North America (ACANA)

African and Caribbean Business Council

Partnership CDC

Mayors Commission on African and Caribbean Immigrant A�airs

Simpson House Senior Living Center

Child Care Center

Mural Arts Program

IHM Center for Literacy

Welcoming Center

Convention Center

1199c

AFAHO (African Family Health Organization) Asian Arts

Initiative

Brewerytown CDC

Frankford CDC

Golden Star Baptist ChurchLafiya

Family Services

Leeway Foundation

Lutheran Settlement House

PhillyCAMSlought Foundation

University City High School

SW Community Advisory Groups

Nicetown Community Advisory Groups

North Philadelphia Community Advisory Groups

SW Global Times Newspaper

Scoop Newspaper

University of Penn

Li�

NHS

Earn Center

Mayor's Commission on Aging

local schools

Local churches

Local halfway houses

My Brother's Keepers

New Directions

James Logan School

Organ Reed School

Parent U School

Athletic Teen Council

Philly Goes2College

KEYSPOT Managing Partner

Collaborations

Other

SBA SBA+

PCC

PCC

Funding

60+

1-10

10-59

Colla

bora

tions

Project HOME St Columba sta� meeting

Partnership to new communities and additional networks of social ser vice organizations. As a result , the FR P was able to reach more target populations.

Partners used KEYSPOTs to further their own organizational goals, specifically for employment, education, and community engagement.

A majorit y of staff inter viewees at Managing Partners, hos t s ite s , a nd A l l ied Or g a n i z a t ion s con nec ted t he FR P’s progra mmatic work to orga nizationa l missions. F o r e x a m p l e , s t a f f a t t w o e m p l o y m e n t - f o c u s e d A l l ied Orga n iz at ions sa id refer ra ls to FR P prog ra ms enhanced their own operations. One inter viewee, whose organization has partnered with more than thirt y other hos t s ite s , s a id , “ We m a ke su re t h a t t he K E YSPOT information is in our career resource center. Our staf f has the information. So when they’re working one-on-one with job seekers that are in need, we let them k now that the K EYSPOTs are in their communit y.” Pa r t ner or g a n i z at ion s t h at work on educ at iona l o u t c o m e s a l s o c r e d i t e d i n v o l v e m e n t i n t h e F R P w it h f u r t her i n g t hei r orga n i z at ion’s m ission . S e vera l inter viewees mentioned the ways in which K EYSPOTs help youth with homework, as well as adults with GED training. An inter viewee at a Managing Partner credited BTOP funding with allowing him to implement a GED t ra i n ing prog ra m t hat m i xed in-cla ss inst r uct ion a nd online training for adult learners. Additionally, this staff person reported finding new funding for collaboration w ith a nother Ma na ging Pa r tner a nd host site to of fer GED training and better ser ve the educational needs of adu lt learners. Several other inter viewees at Managing Partners also discussed clients’ use of sites to pursue the GED.

A s f o r o r g a n i z a t i o n s w i t h m i s s i o n s f o c u s e d o n communit y engagement, inter views with staff revealed t hat d igita l l iterac y sk i l ls a nd public computer access helped t hem to col lectively enga ge t heir constituents and tackle local problems. A staff member at a Managing Pa r t n e r d i s c u s s e d h i s o r g a n i z a t i o n’s e f f o r t t o u s e K EYSPOTs to connect their Spanish-speaking clientele with other communities in the neighborhood, “ There's

52

been a number of folks from the neighborhood [slowly] coming in and getting to k now the [Spanish-speak ing] com mu n it y t h rou g h t he com mu n it y center a nd ne w r e l a t i o n s h i p s b e i n g b u i l t t h ro u g h t h a t .” T h i s s a m e orga n iz ation tra ined com mu n it y leaders to cha l lenge local anti-immigration sentiment; K EYSPOT students produced an immigrant rights radio show and podcast. At a host site, a staf f inter viewee discussed the goa l of connecting digital literacy with social issues. He said, “We want a place where you learn how to use the machine. We want a place where people can interact. We also encourage people to get involved in advocacy on broader issues. [The KEYSPOT] brings people in, so we’re tr ying to get those people to fight for themselves around whatever issues are going on at the time.”

53

Overall Assessment

➜➜ W h i le a majorit y of Ma na g ing Pa r tners had a positive assessment of the Partnership, many also felt that given more time, the Partnership would have had improved functioning and a greater impact.

Challenges and Successes

➜➜ The Pa r t nersh ip faced operat iona l cha l lenges and lacked unified direction and goals during the implementation stage.

➜➜ S o m e M a n a g i n g P a r t n e r s f e l t c h a l l e n g e d by h ier a rc h ic a l de c i s ion-m a k i n g t ie d to one programmatic side of the FRP, while praising the more supportive management on the other.

➜➜ Overa ll , host sites are satisf ied w ith Managing Partners’ oversight.

Sustainability

➜➜ At all staff levels of the FRP, people strongly support the continuation of digital literacy training and public computer centers.

➜➜ Partners face formidable challenges in planning and securing the resources to sustain these programs.

Overall Assessment While a majority of Managing Partners had a positive assessment of the Partnership, many also felt that given more time, the Partnership would have had improved functioning and a greater impact.

In se ven of ele ven i nter v ie ws w it h st a f f at Ma na g i n g Partners and Primes, inter viewees agreed that the FR P

functioned effectively as a Partnership. Interviewees cited examples of feeling respected and heard, referring clients to one another, working through challenging discussions, a nd u n it i n g u nder one sh a red goa l of “ br id g i n g t he digital divide” through a diversit y of programming and organizations. One staff member said, “I really appreciated t he Pa r tnership; I st i l l do. Ever ybody is enga ging a nd ever ybody gets to speak their mind, and nobody blows up at each other, even though there are disagreements. The group really works more to find a way to make things work and be inclusive versus just pushing ideas out the way and doing what they want to do. I feel l ike, in that way, it’s really made us a strong partnership. It’s given me a new way to look at partnership.” To many, this Partnership represented the first time that organizations across the cit y collaborated on digital divide issues.

A voc a l m i nor it y of st a f f members at Ma na g i n g Partners felt that their side of the grant was an effective partnership, but they could not say the same for the FR P, given the more top-down management st yle of the other side of the grant (see Finding 6C). For one staff member at a Managing Partner, “It’s a lways in my mind that the Freedom R ings Partnership has t wo ha lves. A nd that’s always been the distinction. I would say [that one grant] I feel is the idea l definition of ‘par tnership.’ I ca n’t say the same on [the other grant]. I think we have too many strong entities. As much as we try to paint a picture of one Freedom R ings Partnership, it’s not one.” Three others echoed this sentiment.

Ma ny i nter v ie wee s a l so felt t h at w it h add it iona l time, the Partnership cou ld have had a greater impact. In terms of meeting the Partnership goals, inter viewees at Ma na g i n g Pa r t ners a nd Pr i me s celebrated stor ie s

How well has the FRP functioned?Q:

6WUS/SES GIS FG: Participants FG: Webguides Interviews: Managing Partners Interviews: Primes Interviews: Host Sites Interviews with Allied Orgs. Primes’ Report to NTIA

123456789

Sour

ces

Use

d

54

“You can’t fix the digital divide overnight when it took years and decades to develop. I'm not saying it's gonna take a decade. I'm just saying it needs more time than the time that's been allotted.”

“It's only going to get worse as more job applications become online. As the GED goes digital in 2014, everybody agrees that this incredible opportunity to have this equipment and infrastructure in place... [and] that the problem is not just going to disappear after March 2013.”

“The concept of this Partnership and delivering [digital literacy training] is needed. Absolutely needed: it’s the way

technology is moving... It’s just the way of the world from here.” Select Managing

Partner Quotes about Post-BTOP Support(Interviews)

of K EYSPOTs’ shor t-ter m educat iona l a nd econom ic impacts on participants, such as getting people connected to GED trainings, learning workforce skills, finding jobs, as well as learning basic computer skills. All staff persons inter viewed at Managing Partners attributed this success to the FR P’s coupling of digital literacy and social services a nd t he d iversit y of t he Pa r tnersh ip. A sta f f person at a Ma na g i ng Pa r t ner sa id , “ Becau se of t he plet hora of resources, and the diversit y of all the organizations that are involved, we’re able to make connections and really ser ve the population well… Even though the end goal is core Internet basics [training], we also address the other needs for the population.”

However, the issue of time came up in six inter views w it h M a n a g i n g Pa r t ne r a nd Pr i me s s t a f f a s b e i n g a challenge to Partnership health and impact. In terms of

Partnership health, t wo inter viewees commented on how over time, roles and Partnership-wide activities became more clearly defined, but it was a challenge to deal with in the early stages of planning. Similarly, one inter viewee commented, “Not all partnerships have to be long-lived, but I think this one would 've benefited from another year or t wo.. .With this much start up required, we've barely had time to form and get used to each other and begin to learn how to work together.” As the timeline of planning and implementation activities demonstrated (see p. 29), the Partnership took on ambitious goals within a two-year time frame. Staff interviewees at three Managing Partners commented that the Partnership truly succeeded in laying a foundation for future success. Much more work remains. One inter v iewee su mmed up, “I believe we are doing a good job for a young program.  But it really takes a long

55

time for it all to come into focus. There is a ton yet to do.”

Challenges and SuccessesThe Partnership faced operational challenges and lacked unified direction and goals during the implementation stage.

Staf f inter v iewees representing six Managing Partners described several challenges that negatively impacted the implementation of their programs. These included delays in initial payments, lack of clear direction and shared goals ea rly on, a nd a slow sta r t to bra nding a nd adver tising. One staf f person at a Managing Partner shared, “Af ter the initial contracts were signed, it was almost six months before we were actually able to get our money for laptops and equipment.” The Primes, for their part, have suggested that staff interviewees may interpret these issues as delays, but pay ment proce sse s a re e st a bl i shed a nd ex pec ted routines and should be acknowledged as such.58 Over time, partners created more structure and direction in their FR P work. Many staff members discussed the challenge of tr ying to get their own programs implemented to meet the NTIA deliverables, so felt l imited in their abilit y to set up Partnership shared goals in the beginning.

Some Managing Partners felt challenged by hierarchical decision making tied to one programmatic side of the FRP, while praising the more supportive management on the other.

Nea rly ha lf of t he inter v iewees at Ma na g ing Pa r tners described discomfor t w ith the top-dow n ma nagement s t y l e o f o n e g r a n t p r o g r a m i n t h e e a r l i e r s t a g e s o f i m p l e m e n t a t i o n , i n d i c a t i n g a l a c k o f t r a n s p a r e n c y i n d e c i s i o n - m a k i n g , l a c k o f m u t u a l r e s p e c t , o r micromanagement. For example, one staff member at a Managing Partner organization felt frustrated, claiming a Prime mandated an earlier budgetar y timeline without c o n s i d e r i n g t h e n e e d s o r p e r s p e c t i ve s o f t h e o t h e r organizations. The Primes, for their part, have suggested that these criticisms represent only one side of t ypically complex partnership dynamics.59

Despite management struggles, three inter viewees

a t M a n a g i n g P a r t n e r s r e m a r k e d o n t h e p o s i t i v e encou ra gement of t he Prime a nd t he improvement of Primes–Managing Partner relations throughout the grant implementation. One Managing Partner staff person said partners worked through many challenges and attributed the tensions he encountered with the Prime to the lack of clarity from the NTIA. He said, “Those tensions may not have been there if some of the guidelines and f lexibilit y of the grant were more clearly defined in the beginning."

Six out of eleven inter views with staff at Managing Pa r t ne r s l a u d e d t he w ay s i n w h ic h t he ot he r Pr i me interacted with partners and worked in a f lexible manner. Th ree of t he si x Ma na g ing Pa r t ner st a f f inter v iewees concurred that the Prime gave them the support to adjust prog ra m del iverable s , such a s t ra i n i n g requ i rement s , and program deadlines. As one person said, rather than work ing with “ heav y handed insistence that that money has to be spent by this date or else,” the Prime prioritized the implementation of qualit y programs within f lexible deadlines. Another staff person at a different Managing Par tner added , “ Whatever their request is , it’s done in such a professional way, and it’s just an inclusive feeling. They are pretty good with what they do. It’s a good thing, because you can focus your mind on other things. I feel comfortable that they are handling those things.”

Overall, host sites are satisfied with Managing Partners’ oversight.

Four staff persons positively assessed Managing Partners’ oversight, with one offering a mixed opinion. As explained by one host site st a f f member, it s Ma na g i n g Pa r t ner “str uct u red t he relat ionsh ips w it h t he subcontractors in a way that enables us to f lex our muscles in what we do. At the sa me time, [we are] a lways in dia logue w ith them and [they are] really open in terms of things that needed to be modified or revised or just considered, as we're moving for ward with other plans." Some host site inter viewees referred to decision-making early on in the Partnership, but most said that they did not think it was necessa r y for t hem to be deeply involved . A s a not her staf f person said, “[We] didn’t feel included, didn’t feel excluded.” A couple interviewees indicated feeling trusted

56

BTOP has “expanded our capacity, in a way that a lot of grants do to a lot of organizations. Once the grant is over, you sort of scale back close to where you were originally.”

“As we evaluate different sites...the sites in place [now] might not need to continue together, and that's not necessarily a bad thing.”

“[A] lot of these agencies that have KEYSPOTS now will be able to absorb them or continue them in some capacity or other. They have this infrastructure. They have this

equipment. They're going to continue to use it.”

“The partnership should actually get bigger once it ends.”

“The Managing Partners on the PCC side...already laid the groundwork for a professional development association for a digital literacy providers that we’re calling the ‘Technology Learning Collaborative.’ It’s going to include staff from non-profit organizations across the city that have computer labs... they could be KEYSPOTS or non-KEYSPOTS...We’re going to

try to continue...professional development opportunities, like mini conferences, meetups... It’s not anything that’s funded right now.”

“[We’re] trying to figure out different uses for KEYSPOTS... like

fee for service—which is what might start happening with some of the KEYSPOTS turning into GED testing sites. You do get a stipend per student that takes the GED. So it’s like an income generator.”

“[The] City government and the Mayor would be amiss to not see the value that this has offered for

the City. I think...with the importance of technology access and broadband Internet to what makes a healthy economy and a healthy civic life today, I think there's an important role for City government in best sustaining things like these computer centers and these training programs."

Scale back BTOP services

Absorb BTOP services into organization

Expand the FRP

Organize an unfunded, informal

partnership with the intent to grow

in the future

Generate income through sites

House program within

the municipal government

Select Managing Partner Suggestions for the Future of the Partnership (Interviews)

57

to ma ke t heir ow n decisions about t heir K EYSPOTs . The one mixed review centered on claims of inadequate support from a Managing Partner for development of a new digital l iteracy class.

SustainabilityAt all staff levels of the FRP, people strongly support the continuation of digital literacy training and public computer centers.

In all of the inter views with staff at Managing Partners, host sites, and Allied Organizations, staff persons talked about the importance of continuing programs.

Si m i la rl y, i n foc u s g roups , Webg u ide s ex pre ssed strong concerns about discontinuing programs. “ We’re a l re a d y sc ratch i n g t he su r f ace of c losi n g t he d i g it a l divide. So what are you going to do? Pull that away now? That makes no sense at all,” said one Webguide. Another Webg u ide cha racter i zed FR P prog ra ms a s put t i ng “a glimmer of light in an impoverished communit y.” These f ront l i ne worker s de sc r ib ed how t he prog r a m s h ave changed the lives of people in the communities by helping them f ind work or educationa l oppor tu nities throu gh digital learning, escape homelessness, deal with addiction, or provide a safe space for communit y.

Partners face formidable challenges in planning and securing the resources to sustain these programs.

St a f f members at t he Pr i mes a nd Ma na g i n g Pa r t ners organized sustainabilit y forums to guide sustainabilit y planning conversations, bringing together staf f at host sites, Managing Partners, and A llied Organizations to discuss potentia l strategies af ter BTOP f u nding ends. However, in inter views w ith Managing Partners, there w a s a l a c k of c on s e n s u s over p os sible d i re c t ion s , a s demonstrated by a range of suggested directions for the future of the program. As one staff person at a Managing Partner said, “We’ve created this wonderful infrastructure. What parts of it can we and should we preser ve, so that the hard work that’s been happening for the past two years doesn’t go down the tube?”

Thou g h a w ide ra nge of opin ions were expressed , i nt er v ie we e s a t fou r M a n a g i n g Pa r t ner s s p e c u l a t e d about future ways to link digital literacy and educational at t a in ment , d iscu ssing ef for t s to ma ke GED tra ining available to participants and preparing for online-only proc tor i n g of GED te st s . One of t he se i nter v ie wee s said, “One of the things that a lot of K EYSPOTs have expressed great interest in is tr y ing to f ig u re out how to become a GED testing site and how to train their lab managers in proctoring the test , and tr ying to build in preparator y training for people to take the test .” From staffing to participant training to adapting the physical spaces, staf f at K EYSPOTs are readying their sites for the GED transition.

Some partners remarked on the difficult y of looking for additional funding. One challenge may be the prospect of competition bet ween pa r tners, mentioned by three Managing Partner inter viewees. As one inter viewee said, “We’re very fragmented, because everybody is out for their own sur vival. Ever ybody’s out to seek funding that will keep their program running."

However, seven staff members described an interest in col laborat ing on g ra nt s a nd/or seek ing f u nd ing to cont i nue t he ent i re pa r t nersh ip. A n i nter v ie wee at a M a n a g i n g Pa r t ne r e x pl a i ne d , " I n t he s u s t a i n a bi l it y meet ings t hat we're hav ing , we're t a l k ing about g ra nt l a n g u a g e t o g e t h e r, w e 'r e t a l k i n g a b o u t g r a s s r o o t s fundraising programs together. We're talking about, how do you build earned income revenue streams together." In the absence of continued or adequate funding, many pa r t ners a lso voiced t he opinion t hat t he Pa r t nersh ip and the communit y connections made across the diverse organizations will continue. One staff person at a partner orga n ization sa id , “[The Pa r t nersh ip] w i l l ca rr y on in posit i vely spl i ntered ways .” A not her i nter v ie wee at a Managing Partner said, “[We’re] tr ying to preser ve the profe ssiona l t ie s t hat we have e st a bl i shed w it h i n t he Partnership.” A third inter viewee at a Managing Partner noted, “Regardless of what happens with the Partnership, we’re always going to have good and strong relationships

with ever yone that’s around the table right now.”

58

59

These next sections attempt to explain the significance of the findings in relation to the FR P’s progress towards NTIA deliverables and shared goals. Additionally, this section aims to point out both expected and unexpected f i nd i n g s a nd t hei r con nec t ion s to ot her re se a rch on u nderser ved com mu n it ies in Ph i ladelph ia a nd d ig it a l inclusion efforts nationwide.

Mixed Success in Reaching NTIA Deliverables

Overall, the FR P demonstrated partial success in meeting its NTIA deliverables. It succeeded in establishing public computer center s , e xceed i n g awa rene s s t a r get s , a nd ser ving training participants, but fell short of broadband subscription targets and public computer center usage.

Once partners overcame planning and implementation challenges, achievement of several deliverables occurred q u ic k l y. E ve n b e fore t he F R P l a u nc he d t he of f ic i a l KEYSPOT awareness campaign in January 2012, grantees on the SBA side of the Partnership already met their goal of reaching 75,000 participants with outreach materials. By Januar y 2013, the SBA program engaged more than 4 60 , 0 0 0 pa r t icipa nt s t h rou g h a n a g g re ssive cit y w ide advertising and outreach campaign. In a single year, the K EYSPOT program generated nearly ninet y thousand visits, more than a quarter of which constituted new visits. As for its participation goals, the SBA program met the deliverable of ser ving 15,000 training participants in June 2012 , one year ahead of schedule. On the PCC side of the Partnership, by July 2012 , grantees had established 76 of the 77 planned public computer centers.

T h e s e t r e n d s i n m o n t h l y d a t a s u g g e s t t h a t t h e Pa r t n e r s h i p w i l l a c h i e v e t h e S B A t r a i n i n g h o u r s

deliverable before the end of the grant. The Partnership reached its SBA participation goal at a faster rate than its training hours goal, meaning that many participants did not complete fourteen hours of structured training (the original estimated length, see p. 18), and most likely opted for one-on-one ta i lored instr uction rather t ha n structured classes. As the evaluation findings demonstrate, the FR P stil l improved participants’ digital l iteracy and community engagement through peer learning techniques a nd one- on - one t a i lore d i n s t r u c t ion . T he mo d e l of providing individualized training with trusted instructors (in this case, K EYSPOT Webguides) effectively moves participants from non-users of technolog y to confident active learners, particularly when working with vulnerable p opu l a t ion s w ho m ay not b e a ble to c om m it to t he traditional classroom model of training, given transient work schedules and competing life needs.

The SBA progra m demonstrated mi xed success in meeting its broadband subscriber targets, which consists of distributing netbooks and increasing both home and business broadba nd subscriptions. Month ly data show a consistent increase in netbook distribution over time. R ate s for home a nd bu si ne ss su bscr ipt ion i ncre a sed , though in a less predictable pattern than rates of netbook distribution. A marked leap in home subscription occurred in Januar y 2013, just a few months after the Partnership finalized the KEYSPOT Discount Broadband Plan. This i ncre a se , a lon g w it h t rend s i n f u l f i l l i n g ot her N T I A deliverables, suggests that had the program developed and implemented the plan earlier in the grant, it may have been on track to fu lf i l l ing its subscriber nu mbers. Since the original subscriber metrics were intended to be achieved

Discussion

60

in conjunction with the unfunded CCI grant, this delayed development of a subscription plan, and subsequent delay in generating subscriptions, is unsurprising.

Other than operationa l issues, participant barriers to home subscriptions, such as lack of hardware and cost of Internet, presented challenges to the FR P in meeting grant subscription goals. While the FR P developed the netbook d istr ibution a nd low-cost broadba nd pla n to address these participant barriers, the netbook program is l imited to Philadelphia Housing Authorit y residents and the K EYSPOT Discount Broadband plan may stil l be too cost ly for v u lnerable popu lations . In add it ion, a n o t h e r c h a l l e n g e m a y s t e m f r o m N T I A’s f l a w e d programmatic expectations of generating new broadband subscriptions through awareness and training efforts (see Discovering Common Goals, p. 22 for more discussion of subscription definition limitations). Given the complexity of lea r n ing a nd u se pat ter ns of new u sers t a rgeted by FR P progra ms (see Adoption Discussion, p. 61), when broadba nd adoption encompasses a much w ider set of activ ities a nd mea ning f u l outcomes, home broadba nd subscriber nu mbers represent too na rrow a n indicator of program success.

The PCC progra m’s lack of success in meeting its participation targets may stem from a variety of participant, Partnership, and externally related factors. The expected weekly rate of 15,000 participants may simply be too high of an estimate; partners recorded 5,500 actual visitors per week as the highest rate. Partners never articulated a precise timetable for when they expected to have all public computer centers operational, and delays on establishing contracts and opening centers may have led to lower than expected usage numbers. Also, survey data and focus group material demonstrate that KEYSPOT users are often repeat visitors, in some cases developing attachment to particular terminals at public computer centers. The case of repeat use implies that rates may plateau until repeat users recommend and/or awareness campaigns draw in a new set of users. Overall, the participation rates reported by the PCC Prime do not provide a fine-grained look at different KEYSPOTs and which ones were consistently frequented—or packed, as many focus

group participants reported—and which ones were empty. Fi n a l l y, t he m i xed suc c e s s of t he Pa r t ner sh ip i n

meet i n g it s N T I A del iverable s su g ge st s t he need for additional research to explain why some indicators were easily met and others were not. For example, the launch and execution of marketing and outreach may correlate w ith the early accomplishment of the SBA program in reaching participation targets, but does not explain why the PCC program was unable to reach its usage targets. Fu r t her resea rch cou ld include a site-level a na lysis to understand the usage patterns in order to see which sites w ith lower usage than anticipated perhaps drove dow n the overall attendance average. While this Report did not assess the direct impacts of the outreach campaign, the differences in participation/usage rates raises the question of awa rene ss c a mpa i g n’s ef fec t s , bot h i n i n f luenci n g program usage as well as its overall impact on stimulating broadband interest and demand.

Success of an Embedded Approach in Reaching Target Populations

T h e s u c c e s s i n r e a c h i n g t a r g e t p o p u l a t i o n s i s d e m o n s t r a t e d b o t h i n t e r m s o f d e m o g r a p h i c s a n d p a r t i c i p a n t s ’ “ b r o a d b a n d a d o p t e r p r o f i l e s ”— b a s i c cha rac ter i st ic s related to u se (or non-u se a s t he c a se m a y b e) o f b r o a d b a n d t e c h n o l o g i e s . A s i m p l e G I S a na lysis prov ides t he most compel l i ng resu lt s . W h i le su r vey d at a pa int a na rrow socioeconom ic por tra it of participants based only on gender, age, and race, and not educationa l atta inment status or income information, the geographic data from the W US helped to map users. Many participants came to the computer centers from neighborhoods known for high levels of povert y and low educational attainment.

As for adoption characteristics, WUS and Exit Survey data reveal rates of non-adoption of home Internet that exceed both national and local statistics—only one third of adu lt K E YSPOT re sident s have I nter net at home (W US =33 percent , Ex it Su r vey=36 percent). This rate is fairly consistent among demographic groups such as gender, a ge, a nd race. In compa r ison, resea rch by t he FCC and Pew Internet & American Life Project shows

61

that approximately 65 percent of American adults have h i g h - s p e e d I nt er ne t c on ne c t ion s a t home , a nd t h a t seniors, minorities, and low-income populations are less l i kely to have home broadba nd compa red to ot hers . 6 0 A lso, 2010 FCC data demonstrates that non-adoption rates in select Philadelphia census tracts are as high as 80 percent, w ith an estimated non-adoption rate of 55 percent throughout the cit y. 61 This Repor t echoes the more pronou nced broadba nd disparities identif ied for Philadelphia and further underscores the success of FR P programs in connecting to underser ved populations with much more challenging obstacles to in-home broadband use than previously estimated.

The success of placing digital l iteracy in established a nd t r u sted com mu n it y g roups a nd ser v ice prov iders a f f i r ms prev iou s nat iona l resea rch on broadba nd a nd low-income commu nities, which identif ied the role of communit y anchor institutions whose supportive staf f help underserved populations use technologies. The work in Philadelphia speaks to a broad set of institutions able to connect with historically marginalized communities.

The Power of Social Support in Helping Philadelphians Adopt Broadband

The FR P ’s successf u l approach to embedd i n g d ig it a l l i t e r a c y a n d p u b l i c c o m p u t e r a c c e s s i n e x i s t i n g community-based and social ser vice networks also ties to the evaluation’s f indings related to increasing adoption. With regards to this FR P goal, the major success stor y is about the importance of social processes and context for stimulating broadband use: by tak ing an active role in welcoming new users to online worlds, FR P programs are mak ing a difference in participants’ l ives.

Individuals start off with personal interest in digital technologies, even thou gh they may a lso be fea r f u l of technolog y or face learning barriers. The supportiveness of staff encourages new users, and the comfortable, safe, a nd pu r posef u l K E YSPOT env i ron ment cont r ibute t o p a r t i c i p a n t s ’ a b i l i t y t o p r o g r e s s a n d b e c o m e k now led ge able about computers a nd t he Inter net . A sen se of com mu n it y develops a lon gside pa r t icipa nt s’ involvement at K EYSPOTs through connections made

on site as well as contacts made virtually. An abundance of digital learning takes place at KEYSPOTs, motivating participants to learn and use computers and the Internet.

The findings in this Report directly speak to existing broadba nd adopt ion re sea rch . Wit h rega rd s to home broadba nd subscriptions, wh ich t he FR P considers a s one a mong severa l for ms of broadba nd adopt ion, t he findings here portray a different profile of non-use, not found in one heavily cited national study on broadband u se. Th is resea rch, for exa mple, st ated t hat non-u sers (of home broadba nd Inter net) repor ted “ lack of need or interest as their primar y reason for not having home broadband Internet access.”62 However, both qualitative and quantitative data presented here speak to the primary issue of cost—not just of Internet connectivit y but also hardware, and echo other research focused on cost issues.63 In addition, the fact that no W US respondents selected “waste of time” as a reason for not getting home broadband a l so cha l len ge s cla i m s t hat non-u sers do not see t he relevancy of technology. High rates of daily use of Internet for both KEYSPOT participants with and without home Internet also support the assertion that participants find broadband relevant to their lives. Meanwhile, qualitative data suggest that interest is high: K EYSPOT users start off with a base interest and recognition of the value and releva nce of d ig it a l com mu n ic at ions to d a i ly l i fe . A s ex pre ssed i n a st udy on low-i ncome popu lat ion s a nd broadband, “ broadband access is a pre-requisite of social and economic inclusion, and low-income communities know it” (emphasis added). 6 4

The data in this Repor t over whelmingly points to the role of social support in helping novice users adopt broadband technologies. A stor y of broadband adoption emerged t h rou g h t he se f i nd i n g s , w here pa r t icipa nt s overcome their fear of technolog y w ith the support of a helpf u l Webg u ide , i ncrea se t hei r i ntere st i n d ig it a l learning, and develop feelings of satisfaction and comfort in KEYSPOT settings. Participants become part of a deeply social process through gaining familiarity with and using broadband technologies in ways that make sense to them. Adoption relies on social support structures, so if new users have unequal access to dedicated staff for digital literacy

62

Individuals existin a state of non-use

(either due tocost, lack of know-how or both).

Motivated by social circumstance, individuals spark an interest in being digitally literate.Intermediary organizations (such as

Community Anchor Institutions)

couple digital learning and computer access

with other social service o�erings.

Individual interest, coupled with intermediary organizations’

engagement strategies draw individuals to digital literacy.

Individuals move to

a state of meaningful broadband use and

elect to use broadband.

Individuals use broadband technologies in settings

that they find most appropriate to their personal

needs and contexts.

Sta� at these organizations help individuals overcome fear of technology and give individuals space to

further develop personally relevant goals for digital learning.

KEYSPOT Model of Broadband Adoption

63

programs, then digital inequalities are unlikely to change.65

T h e Pa r t n e r s h i p h a s a n e q u a l a p p r e c i a t i o n o f t he d iverse contex t s i n w h ich new u sers enter d ig it a l worlds, meaning that it does not value technolog y use in public or private spaces differently. The initial work of Managing Partners to expand the definition of broadband adopt ion cha l lenges t he a ssu mpt ion t hat home u se is more impor ta nt t ha n d igita l lea rning at , for exa mple, a K EYSPOT. Though other research on social support s y s te m s a nd broa d ba nd u s e h a s tende d to de l i ne a te bet ween home broadband adoption and computer and Inter net u se at t r u sted i n st it ut ion s l i ke l ibra r ie s a nd com mu n it y-ba sed or g a n i z at ion s , t h i s R epor t avoid s dichotomies between adoption in and outside of the home.

The FR P’s success with regards to its first and second goals suggests a model of broadband adoption implicit in the expanded definition chosen by the Partnership at the outset of the evaluation planning process. This model is premised on the interaction between personal motivations and social support net works.

T h i s m o d e l e c h o e s w h a t o t h e r r e s e a r c h e r s a n d eva luators have said about broadband adoption and the importance of social setting, including the unique benefits that community anchor institutions bring to an individual ’s encounter with broadband. With that said, the FRP’s story of broadband adoption success is incomplete. First , the indicators used for this e va lu a t ion i nc luded a broa d de f i n it ion of a dopt ion , ra nging f rom a l l t y pes of d igita l lea rning to a l l forms of Inter net access . E ach piece of t h is model is key to understanding broadband adoption. However, it may be helpful in the future to identif y specific t ypes of digital learning to differentiate those that are more effective at stimulating relevance and interest in home broadband use specifically. Second, the evaluation plan did not establish a way to track how FR P programs directly affected home subscr ipt ions , w h ich wou ld have requ i red sig n i f ic a nt resources and external collaboration with Internet Service Providers. This Report does not capture the secondar y effects of the programs in increasing broadband awareness and potentially increasing subscription rates for mobile or home broadba nd , st at ist ics t hat a re genera l ly kept

propr iet a r y by Inter net Ser v ice Prov iders , a s wa s t he c a se w it h t he Comc a st I nter net E s sent ia l s prog ra m . Accord i n g ly, t he Pa r t nersh ip sou g ht to approx i mate such information by assessing interest a nd pa r ticipa nt self-assessment of the likelihood of subscription (see also Discovering Common Goals, p. 22 , on the development of adoption metrics). Finally, and more broadly speak ing , the evaluation of t he K EYSPOT Model of Adopt ion w i l l need f ine-tu ning in the f utu re if other prov iders or par tnerships are interested in replicating Philadelphia’s ef forts. The pairing of quantitative and qualitative data for the analysis of (a n expa nded def init ion of ) broadba nd adoption is important but difficult to execute. As written about in one study of broadband use, a more nuanced understanding of broad ba nd adopt ion depend s on t he ex a m i nat ion of qu a nt it at ive met r ic s , such a s t he nu mber of home subscriptions, but also real-world ’ data and feedback. 6 6 This has been the norm for this evaluation: the analysis of focus group data and inter view material often speaks to experiences that are nuanced and difficult to capture.67 Ad d it ion a l re s e a rc h i s ne e d e d to s t a nd a rd i z e t he s e experientia l indicators and help locate reliable ways to match between quantitative metrics and qualitative ones. It may be helpful to explore other reliable methods beyond inter views or focus groups. For example, obser vational research forms a core part of people-centered studies of computers and information systems and can provide non-invasive ways of capturing the practices and experiences of new users when trying technologies for the first time. 68

Unanticipated KEYSPOT Success in Building Communities

Given the FRP’s innovative approach to embedding digital literacy ser vices in existing communit y-based and social ser vice net works, it is not surprising that the FR P had clearer results in spurring communit y engagement than in st i mu lat ing employ ment or i mprov ing educat iona l at t a in ment for pa r ticipa nt s . A s a l luded to prev iou sly, publ ic computer centers a nd cla ssroom set t i n gs help t o c u l t i v a t e a c o l l e c t i ve s e n s e o f b e l o n g i n g a m o n g K E YSP OT p a r t ic i p a nt s . T h i s c u lt u re of b e lon g i n g refers to both what happens on-site, such as participants

64

bonding as they discover broadband together, and what occu rs v i r t u a l ly, l i ke when pa r t icipa nt s con nect w it h social net works online. Though sur vey data indicate that communit y reasons were not participants’ main reasons for K EYSPOT u sa ge, qua litative data clea rly ind icate that the comfortable and safe environments for digital learning keep participants coming back to sites.

The degree of community connection revealed in this eva luation complements other qua litative studies that discuss effects of digital inclusion efforts on communit y health. One study of a digital inclusion project for a poor u rba n commu nit y fou nd that residents’ discussions of the commu nit y resou rces website ref lected impor ta nt indicators of social cohesion and civic-mindedness; the online project itself cultivated and fostered community as KEYSPOTs have done with digital inclusion trainings and computer labs. Another study found a strong connection b e t w e e n b u i l d i n g d i g i t a l c a p a c i t y a n d i n c r e a s i n g com mu nit y control of it s col lective wel l-being—what the authors refer to as digital human capital. 69 Previous re s e a rc h on c y b erc a fé s a nd com mu n it y i n for m a t ic s in developing cou ntries a lso high light the qua litative, communit y-building effects of providing digital access, in both off line and online communities.70

Interim Impacts of KEYSPOTs on Employment and Educational Attainment

A s for p a r t ner sh ip go a l s re l a te d to bro a d b a nd ’s positive socioeconomic effects, more time is needed to ascertain K EYSPOT effects on educational attainment or employment. Nonetheless, the evaluation shows some promise that the FR P is on that path.

In terms of employment impacts, K EYSPOTs laid a foundation for future improvements in employment status of par ticipants. Both quantitative and qua litative data demonstrate how participants mainly used K EYSPOTs for job search and preparation. Given the larger backdrop of Philadelphia’s high unemployment rate in 2010, it is unsurprising that workforce development reasons drive participant use of KEYSPOTs.71 Additionally, KEYSPOTs provided professional development opportunities and led to direct employment effects, though only for a handful of

individuals, who started as students and were then hired as Webguides. This suggests that while digital l iteracy may not immediately lead to the creation of a new class of information workers ready for Philadelphia's digita l economy, the Partnership nonetheless may be developing t he cond it ion s for event u a l employ ment i n t he cit y's digital future.72

T h e c o l l e c t i v e P a r t n e r s h i p - w i d e i m p a c t o n e d u c a t i o n a l a t t a i n m e n t i s s i m i l a r t o i m p a c t s o n employ ment: K EYSPOTs prov ided pat hways towa rd s educational attainment. Partners with a specific focus on educational attainment used BTOP funds to expand their educational offerings, though dedicated training hours for GED and other formal learning opportunities were modest . Mea nwhile, a high percenta ge of pa r ticipa nts who were seeking educational opportunities reported that trainings did connect them to educational opportunities. O n t h e E x i t S u r ve y, p a r t i c i p a nt s s e l f - d e f i n e d w h a t “educationa l oppor tu nities” mea nt to t hem, so ta k ing a b a s i c c o m p u t e r t r a i n i n g c l a s s m a y h a v e b e e n a n educational opportunit y on its own, even though it was not necessarily a class on college financial aid. (This is also an artifact of the sur vey only targeting adults.) Similarly, partners, participants, and Webguides shared numerous stories about learning (digital and otherwise), though they all were virtually silent on the matter of formal educational success stories. While the time frame of the evaluation limits the ability of the FRP to document improvements in formal educational attainment, it is clear that participants improved their k nowledge of digital sk il ls and felt that KEYSPOTs helped them improve their general education.

Assessing the Partnership, Its Successes, and Challenges

The assessment of Partnership-level goals tells a complex stor y. The findings most clearly demonstrate evidence of increased capacity as a result of FR P programs. However, i n relat ion to t he he a lt h of t he Pa r t nersh ip, f i nd i n gs point more to t he need for add it iona l resea rch rat her t ha n def i n it ive conclu sions . Wit hout a clea r ba sel i ne of expectations of what a well-functioning partnership looks l ike, progress is dif f icu lt to interpret , even when

65

change has ta ken place. Nevertheless, the evolution of the Partnership since 2010 has had positive impacts, most clearly on the abilit y of the FR P to ser ve Philadelphia’s most u nderser ved a nd help u sher broad ba nd a nd it s benefits to these communities.

Increased Capacity to Do More

On the whole, partners saw their organizational capacities improve. For host sites, BTOP funding provided much needed staffing support and new training opportunities. Historically, these organizations have had many different programs and have been unable to fully staff computer labs. With the funding to hire Webguides and access to Partnership-wide resources for digital l iteracy training, updated hardware, and technical support, organizations were free to focu s on orga nizationa l mission a nd core p r o g r a m m i n g . T h e Pa r t n e r s h i p a l s o i n c r e a s e d t h e capacit y of staf f at Managing Partners and Webguides to implement d igita l inclu sion a nd tra ining progra ms through resource sharing and professional development o p p o r t u n i t i e s , s u c h a s t h e Tr a i n i n g R o u n d t a b l e s a nd Su st a i nabi l it y For u m s . K EYSPOTs repre sent a n u nprecedented oppor t u n it y to expa nd operat ions for Ma na g ing Pa r tners a nd develop a new cla ss of d ig it a l l iterac y t ra i ners a nd computer a ssist a nt s for t he 21st centur y.

A clea r innovation of the Pa r tnership ha s been to use technolog y as a linking force bet ween organizations. W here a s ot her d i g it a l i nc lu sion pa r t ner sh ips m i g ht revolve around a traditional issue area such as healthcare, workforce development, community advocacy, or literacy, FR P partners connected to one another on the issue of broadband access, and the Partnership elevated the role and centrality of technolog y in the work that partners do. Partners, who wou ld have other w ise remained in their separate communities and fields of practice, connected to one a not her a nd to ot her orga n iz at ions a s a resu lt of their com mon mission to brid ge the d igita l d iv ide. Through these connections, partner organization were better equipped to meet their target populations.

Initial and Ongoing Impediments to the Partnership

Like ma ny pa r tnerships that a re sta r ting of f, the FR P ex per ienced com mon cha l len ge s to succe s sf u l g ra nt implementation.73 A significant amount of time, energ y, and resources went into planning and sorting out how t he Pa r t nersh ip cou ld work toget her. Operat iona l ly, the Partnership did not a lways f unction smoothly. For ex a mple , delays i n cont rac t s t ied up t he d ist r ibut ion of f u nding , which in tu rn impacted the lau nch of new trainings, computer labs, and outreach efforts. Partners a lso su f fered f rom com mu n ic at ion a nd coord i nat ion i s sue s . Gi ven t h at some pa r t ners h ad not pre v iou sl y worked together, the init ia l sta ges of the gra nt mea nt g e t t i n g t o k n ow o n e a n o t h e r a n d c u l t i v a t e t r u s t . 74 Particularly on one side of the Partnership, management style of one Prime was perceived as top down (as opposed to shared or consensus based, which was more the case with the other grant). One potential motivating cause of the initial diff icu lties involves the Partnership’s lack of explicitly shared goa ls or a unified vision, which cou ld have f u nctioned to cla rif y orga nizationa l roles . Given t he nu mber of pa r t ners a nd t he scope a nd breadt h of the Partnership goa ls, these operationa l cha llenges are unsurprising and to be expected.

Many of the initial challenges subsided as partners developed shared goals and fell into routines for designing, implementing, and eventually improving FR P programs. However, some challenges persisted. Among these, the most notable ones were coordination and communication. For exa mple , some st a f f est abl ished a refer ra l s ystem (which was not an NTIA deliverable). But while Managing Partners and host sites evidenced spontaneous or organic forms of knowledge sharing and referral systems, partners lacked an intentionally designed and efficiently run system of collaboration that cou ld have improved interactions with participants.

These cha l lenges wou ld be ea sier to interpret had the FR P identif ied clea r expectations at the outset of what a wel l-f u nctioning pa r tnership enta i ls . However, i n t he a bsence of such bench m a rk s , it i s d i f f ic u lt to

66

specu late on the signif ica nce of the tra nsformation of the Partnership over time. What the challenges broadly sp e a k to a re t he comple x it ie s of a l a r ge a nd d i ver se partnership. The wide variet y of programs and ser vices offered by more than eighty partner organizations is hard to visualize and utilize to the fullest extent. While some PCC goals are easy to measure (e.g., how many computer centers were opened), sustainable broadband adoption is a much more nuanced and difficult concept to grasp or ach ieve a s a mea su rable goa l for com mu nit y-ba sed organizations. Having consensus around shared goals in such a large Partnership is difficult to acquire under the best of circumstances, and in this case, the difficult y was compounded by the t wo-year time frame of the BTOP grants, which staff inter viewees found too restrictive for Partnership formation as well as program impact.

Cha l len ge s t hat be a r on t he Pa r t nersh ip’s he a lt h or f u nc t iona l it y a re a lso d i f f icu lt to i nter pret due to e x t er n a l f a c tor s , p a r t ic u l a r l y t he f u nd i n g s t r u c t u re and program timeline as determined by NTIA . As one staff member explained, the FR P inherited some of its prog ra m de si g n a nd i mplement at ion problem s f rom less-than-clear directives from the NTIA. Within a t wo-year gra nt , par tners spent signif ica nt a mou nts of t ime on planning and developing Partnership structures and work ing protocols, leaving less time for actual program implementation and therefore, less time to see long-term impacts on outcomes. The evaluation planning process

mirrored the overall process of program implementation: needs assessment and evaluation planning took one year a nd severa l mont hs, wh i le actu a l d at a col lection on ly spa n ned seven mont h s . Issues related to f u nd i n g a nd sustainabilit y a lso depend on externa l factors. How to f u nd FR P progra ms may have less to do w ith interna l dy na m ic s a nd more to do w it h econom ic t rend s a nd their impact on federa l, state, loca l, and philanthropic budgets. The variation in staff sentiments about the future of the FR P likely results from the complex interplay of specific inter-partner struggles, the effects of a large and diverse partnership, and the absence of identified future funding prospects.

B et ween t he cha l len ge s of ma na g i n g a la rge a nd diverse pa r tnership a nd externa l factors related to the g ra nt a d m i n i st rat ion , t he Pa r t ner sh ip demon st rate s t he over a l l d i f f ic u lt y of b ei n g "shove l re a d y" i n t h i s l i n e o f w o r k . A s o n e m o d e l o f t e a m d e v e l o p m e n t i l lustrates, group collaboration t ypica l ly goes through t h r e e s t a g e s — “ f o r m i n g ” t h e g r o u p , “ s t o r m i n g ” o r wo r k i n g t h ro u g h g ro u p a r g u m e nt s , a n d “n o r m i n g ” o r c o m i n g t o s h a r e d g o a l s — b e f o r e t h e g r o u p a c t u a l i z e s p r o j e c t g o a l s ( “ p e r f o r m i n g ” ) . 7 5 A s s e e n with the FRP, allocated time for the early stages of project pla nning is a n impor ta nt factor when u nder ta k ing a n a mbitiou s project whose impacts a re ha rd to pinpoint and disaggregate.

67

Paving a Path for Future Research

Though this Report has attempted to provide a rich and complex understanding of the Freedom R ings Partnership and its progress towards shared goals, several additional gaps (ot her t ha n t hose a l ready mentioned prev iou sly) cont r ibute to a n i ncomplete por t ra it here . T he most obvious path for future research pertains to broadening t he scope of t he e va lu at ion: a f u l l a sse ssment of t he FR P’s progress toward shared goals should take place at t he conclu sion of t he project in Au g u st 2013 . Due to external factors, this Report only represents a portion of the FR P’s lifecycle.

Ot her a re a s for f ut u re re se a rch t hat c a n add re ss t he l i m it e d s c o p e of t h i s e v a lu a t ion c onc e r n you t h populations and awareness or outreach efforts. Having excluded youth from the pool of participants that sur vey instruments could quer y for this evaluation, this Report cou ld not a sse ss FR P prog ra m s’ educ at iona l i mpac t s on you n g people , e specia l ly t hose related to pa r t ner organizations with youth- and education-related missions. Futu re resea rch into t he activ it ies a nd experiences of you n g people at K E YSPOTs wou ld help to broaden a n u nderst a nd i n g not on ly of i n for mat ion related to e d u c a t ion a l a t t a i n me nt , bu t m ay a l s o she d l i g ht on issues in digital literacy (e.g., intergenerational learning), t he nat u re of pu bl ic computer centers a nd t hei r u se (e.g. , dif ferences in computer center experiences when participants are a mixture of young and old), and processes of bringing broadband into the home (e.g., whether and how young people inf luence technological purchases in the home). 76

R e se a rch on t he i mpac t of spec i f ic prog ra m s on broadband subscription status would demonstrate which

programs are more effective at stimulating demand for broadband, including home broadband, public broadband, and mobile broadband. Mobile broadband use is growing i n u rba n popu lat ion s a nd a s more people ow n sma r t phones and tablets, it is an important mode of broadband to examine. The future of any digital inclusion initiative will depend on continued track ing of publicly available datasets on home broadband subscriptions, and on the opening up of proprietar y data collected by Internet and mobile ser vice providers.

A f ut u re eva lu at ion wou ld do wel l to specif ica l ly a n a l y z e t h e e f f i c a c y o f a w a r e n e s s c a m p a i g n s a n d t hei r i mpac t on prog ra m at tend a nce a nd broad ba nd subscription rates for residents and businesses. Though some inter views and focus group material speak to the inf luence of outreach campaigns on increasing attendance, this Report does not systematically examine impacts of the awareness campaign, program specific or other wise. Of particular interest would be differences in the effects of outreach for the K EYSPOT low-cost Internet pla n and the Comcast Internet Essentials program given the different awareness strategies and reach bet ween the t wo programs. For example, a comparative content analysis of t he ca mpa ig ns’ out reach mater ia ls wou ld est abl ish the messaging contained in each program. Subsequently, researchers could conduct an impact analysis of awareness m a t e r i a l s o n p r o g r a m u s a g e a n d c i t y w i d e r a t e s o f subscription. A retrospective study of outreach materials and media hits can also be conducted to examine the topic of “digital divide” and its salience in the news media as an agenda-setting topic for policy discussion.

T he f i nd i n g s reg a rd i n g broadba nd adopt ion a nd broadba nd ’s posit ive socioeconom ic ef fec t s spe a k to

Conclusions

68

the need for deeper research about the interconnection bet ween adoption goa ls and other basic needs. As this evaluation has shown, participants view digital literacy as integral to participating economically and socially in the modern world. At the same time, digital, educational, and economic needs are co-dependent: the learning challenges that confront new digital users interact with the financial barriers to access. Outside the United States, a body of research and progra m eva luation has a lready begu n to look at the intersection of digital and other human needs. This same body of research also points out the difficult y of separating out broadband adoption goals from those related to broadband ’s positive socioeconomic ef fects: individuals must have access to education, shelter, food, and other basic needs in order to take advantage of digital resources, which in turn can have an impact on the delivery and quality of basic needs. 77 Applied to the FRP, a holistic fra mework l ike the one used in this emergent body of work can help explain the complexity of digital inclusion efforts that target underser ved populations.

L a s t l y, to f u r t her u nder s t a nd t he i mp a c t of t he Pa r t ner sh ip on e a c h pa r t ic ipa t i n g pa r t ner, it wou ld b e i nt e re s t i n g t o f o l low- u p w i t h e a c h o r g a n i z a t i on involved with the FR P to see where the digital inclusion work st a nd s post BTOP f u nd i n g a nd w het her or not the partners sti l l work together in some capacit y. This evaluation documented the planning and implementation phases during BTOP funding, but due to external factors, is u nable to study the close-out procedures and f uture he a lt h of t he org a n i z at iona l relat ion sh ips de veloped during this time frame. Following up with the Managing Partners from the FR P as well as host sites of KEYSPOTs wou ld be t he most def init ive stor y on whet her or not t he Pa r t nersh ip succeeded i n i ncrea si ng capacit y a nd advancing a common agenda.

Recommendations and Lessons LearnedThe experiences of the FR P and the findings discussed here generate several recommendations and lessons for both the FR P and other digital inclusion efforts.

Based on the findings presented in this Report, OTI offers the following recommendations to the members of the Partnership:

➜➜ Create a forma l (e.g. , FR P-w ide) u ser-f r iend ly resource sharing system that assists partners in referring participants to KEYSPOTs and exchanging best practices around curriculum, peer learning, and mixed skills classroom strategies for trainers.

➜➜ Develop a Neighborhood Working Group to focus on connecting and promoting KEYSPOTs’, host sites’, and Allied Organizations’ ser vices (digital and non-digital) at the neighborhood level, through, for example, the creation of neighborhood-specific listservs or the hosting of local gatherings.

➜➜ Create a Community Engagement Working Group to focus on ways to increase and improve community experiences at KEYSPOTs, including a roundtable on community engagement models that connect participants to on-site and off line community.

➜➜ Develop a strategic outreach plan for continued c o l l a b o r a t i o n s w i t h c i t y w i d e s e r v i c e s a n d community hubs to better coordinate the creation of new partnerships .

➜➜ Ser ve as a capacity-building net work to support and consult with other community organizations on how to embed digital literacy into traditional programming. Seek external funding or develop an earned-income model to provide these resources and services.

➜➜ Conduc t a n a nony mou s su r ve y of Ma na g i n g P a r t n e r s a n d K E Y S P O T s i t e s t o c a p t u r e o v e r a l l a t t i t u d e s a n d f e e l i n g s a b o u t f u t u r e p a r t n e r s h i p s u s t a i n a b i l i t y a n d o p e r a t i o n s .

69

Philadelphia’s K EYSPOT experience ca n a lso inform future partnerships focused on broadband adoption and inclusion with these lessons:

➜➜ Digital literacy trainers and public computer lab assistants must be knowledgeable, friendly, and above all, patient, particularly for the population of novice users who are fearful of new technologies.

➜➜ P r o g r a m s s h o u l d i n c o r p o r a t e p r o f e s s i o n a l development opportunities for their digital literacy trainers, including trainings that address the rapidly changing pace of technology.

➜➜ For large collaborations, partnership-wide tasks, such as evaluation, awareness, and technical support should be established at the very beginning stages of planning and revisited periodically throughout the grant period.

➜➜ For grants that require a partnership structure, funders should realistically allocate more time for partnership building.

Looking ForwardT he Ph i l a de lph ia Freedom R i n g s Pa r t ner sh ip i s a n innovative, large-sca le, and ambitious set of programs that address deeply rooted socioeconomic problems as well as opportunities related to broadband access and use. Within the short time frame of the BTOP grants, partners developed a massive net work of commu nit y-ba sed a nd soc ia l ser v ice or g a n i z at ion s w h ic h l a id a tech no-socia l i n f ra st r uct u re for br id g i n g t he d ig it a l d iv ide a nd ha rnessing broadba nd ’s posit ive inf luence on workforce development, educational attainment, and community engagement. The Partnership accomplished this in spite of periodic decision-making, communication, and coordination challenges.

The atta in ment of these goa ls is neit her ea s y nor immediate, and the FR P’s approach to digital inclusion i l lu strates u nique ways in wh ich fou nd ationa l ef for ts can take place. The practice of embedding broadband adoption ef for t s w it h in com mu nit y-ba sed a nd socia l

s e r v i c e o r g a n i z a t i o n s e f f e c t i v e l y c o n n e c t s t a r g e t populations to programs. The focus on social support has also inf luenced participants to positively view and engage w it h new d igita l tech nologies . These factors combine to tel l t he stor y of a K EYSPOT Model of Broadba nd Ad o p t i o n t h a t s h ow s o n e p a t h f o r h ow i n d i v i d u a l s historically on the w rong side of the digital divide can increase their broadband use. Connecting digital literacy to other pressing needs of target populations and doing so in a welcoming, supportive way is an effective means to cultivating personal and community interest in broadband.

Whether KEYSPOT broadband adoption will result in desired long-term outcomes is difficult to predict based on the evidence in this Report. For one, the effects of FRP programs on economic and educational betterment and communit y engagement will be increasingly difficult to disentangle, especially as time passes. These goals are also inter t w ined w ith one a nother. Second , as the f indings around sustainabilit y suggest, the duration of the FR P is under question, as BTOP funding sunsets and partners discuss and debate the need to continue the Partnership in its current form.

It is unlikely, though not impossible, that a funding o p p o r t u n i t y s u c h a s B TO P w i l l c o m e a l o n g a g a i n to suppor t a s d i verse a nd la rge a n ef for t a s t he F R P. A s Ph i ladelph ia wa it s for t h i s nex t oppor t u n it y, t he findings from our evaluation demonstrate that with an interdisciplinar y approach to digital literacy, community p r o g r a m s c a n h a v e a g r e a t e r i m p a c t o n v u l n e r a b l e popu lations’ l ives and holistica l ly address severa l areas of need at once. The K EYSPOT experience shows that sustained, tailored support for new adopters is necessar y to ensure that broadband technologies engage people in ways that are most relevant to their lives.

70

1 IBM & City of Philadelphia, Smarter Cities Challenge Philadelphia Report, December 2011, http://digitalonramps.com/files/IBM_Smarter_Cities_Challenge-Philadelphia_Report.pdf; Brian James Kirk, “By 2030, 600K Philadelphians Won’t Have Basic Skills for Work, Report Says,” Technically Philly, February 22, 2012, http://technicallyphilly.com/2012/02/22/by-2030-600k-philadelphians-wont-have-basic-skills-for-work-report-says-fight-for-digital-literacy.

2 Pew Charitable Trusts, Philadelphia 2011: The State of the City, April 2011, 4, http://www.pewtrusts.org/uploadedFiles/wwwpewtrustsorg/Reports/Philadelphia_Research_Initiative/Philadelphia-City-Data-Population-Demographics.pdf.

3 Pew Charitable Trusts, A City Transformed: The Racial and Ethnic Changes in Philadelphia Over the Last 20 Years, June 2011, 1, http://www.pewtrusts.org/our_work_report_detail.aspx?id=85899360312.

4 U.S. Bureau of the Census, 2010 Census, Profile of General Population and Housing Characteristics, Philadelphia County, Pennsylvania, http://factfinder2.census.gov/. Due to high rates of incarceration in Philadelphia,these numbers may fail to capture the full picture. See: Becky Pettit, “Black Progress? Not When You Include the Incarcerated,” The Root DC Live blog, entry posted on November 13, 2012, http://www.washingtonpost.com/blogs/therootdc/post/black-progress-not-when-you-include-the-incarcerated/2012/11/13/1412b6b2-2da0-11e2-9ac2-1c61452669c3_blog.html (accessed June 24, 2013).

5 U.S. Bureau of the Census, 2010 Census.

6 Pew, Philadelphia 2011, 9.

7 Ibid., 52.

8 Ibid., 5. U.S. Bureau of the Census, 2010 American Community Survey, One-year Estimate Selected Economic Characteristics, Philadelphia, Pennsylvania, http://factfinder2.census.gov/.

9 Pew, Philadelphia 2011, 16; Pew Charitable Trusts, Philadelphia’s Workforce Development Challenge, January 2012, http://www.pewtrusts.org/uploadedFiles/wwwpewtrustsorg/Reports/Philadelphia_Research_Initiative/Pew-Philadelphia-Workforce-Development-Jobs.pdf.

10 Pew, Philadelphia 2011, 15; Mary Umberger, “‘Eds’ and ‘Meds’ Soothe Real Estate Pain in Philadelphia,” Inman News, Septermber 7, 2011, http://www.inman.com/news/2011/09/7/eds-and-meds-soothe-real-estate-pains-in-philadelphia.

11 Jane M. Von Bergen, “Low Literacy Limits Half of Phila. Workforce, Study Finds,” Philadelphia Inquirer, June 28, 2009, http://articles.philly.com/2009-06-28/business/25285375_1_low-literacy-adult-literacy-study.

12 Pew, Philadelphia 2011, 16.

13 Ibid., 19-20.

14 U.S. Bureau of the Census, 2010 American Community Survey, Tenure by Housing Cost as a Percentage of Income in the Past Twelve Months, Philadelphia, Pennsylvania, http://factfinder2.census.gov/. Paying more than 30 percent of income is generally accepted as the benchmark for unaffordable rent.

15 Project HOME, Saving Homes, Saving Lives, 2010, 4, http://www.projecthome.org/pdf/PRHOMESavingLivesFINAL080210.pdf; Philadelphia Housing Authority, Agency Overview Factsheet, December 6, 2012, http://www.pha.phila.gov/media/89488/pha_fact_sheet_final_12_6_12.pdf.

16 The Pennsylvania Budget and Policy Center, “Details on $212 Million in 2010-2011 Budget Cuts,” http://pennbpc.org/details-212-million-2010-11-budget-cuts; The Foundation Center, Foundation Growth and Giving Estimates 2010 Edition, 2010, 1, http://foundationcenter.org/gainknowledge/research/pdf/fgge10.pdf.

17 Pew, Philadelphia 2011, 30-31; Pedro Molina, “Philly School District’s Funding Under Control,” Philadelphia Inquirer, June 28, 2011, http://articles.philly.com/2011-06-28/news/29713101_1_budget-cuts-stimulus-funds-budget-gap. Although the School District did not face a budget shortfall in 2009 and 2010, officials were aware of the impending end of stimulus funding and pending cuts to state education moneys.

18 The School District of Philadelphia, “Enrollment: District Operated Schools, September 2011,” http://www.phila.k12.pa.us/about/#enrollment (accessed June 24, 2013).

19 Pew, Philadelphia 2011, 6. U.S. Bureau of the Census, 2010 American Community Survey, One-year Estimate, Selected Social Characteristics in the United States, Philadelphia, Pennsylvania, http://factfinder2.census.gov/.

20 John Dunbar, “Poverty Stretches the Digital Divide,” Investigative Reporting Workshop, March 23, 2012, http://investigativereportingworkshop.org/investigations/broadband-adoption/story/poverty-stretches-digital-divide/; Dharma Dailey et al., Broadband Adoption in Low-Income Communities, Social Science Research Council, 2010.

21 Brian James Kirk, “55 Percent of Philadelphia Households Lack Access to Internet,” Technically Philly, April 4, 2012, http://technicallyphilly.com/2012/04/04/55-percent-of-philadelphia-households-lack-access-to-internet-new-early-data-shows-rate-higher-than-previously-thought; Jacob Fenton, “Broadband Adoption Map,” Investigative Reporting Workshop, March 23, 2012, http://investigativereportingworkshop.org/investigations/broadband-adoption/htmlmulti/broadband-adoption-map/.

22 Greater Philadelphia Urban Affairs Coalition, Broadband Non-Infrastructure Application Submission to NTIA—Sustainable Broadband Adoption, August 2009, 10, http://www2.ntia.doc.gov/files/grantees/urbanaffairscoalition_sba_application_part1.pdf. These neighborhoods included Fairmont, Spring Garden, Northern Liberties, Frankford, Kensington, Nicetown, Logan, Germantown, Wynnefield, Paschall, Kingsessing, Cobbs Creek, and Point Breeze, among others.

23 Nicole Dreisbach, “Who’s not online? Internet Use and Older Adults in Southeast Pennsylvania,” Community Health Database, August 1, 2012, http://www.chdbdata.org/datafindings-details.asp?id=103 (accessed June 11, 2013).

24 Karen Mossberger, Caroline J. Tolbert, and Allison Hamilton, “Measuring Digital Citizenship: Mobile Access and Broadband,” in “Broadband Adoption,” ed. Seeta Peña Gangadharan and Greta Byrum, special section, International Journal of Communication 6 (October 2012); John B. Horrigan, “Broadband Adoption and Use in America,” Federal Communications Commission Omnibus Broadband Initiative Working Paper Series, 1, February, 2010; Welcoming Center for New Philadelphians, Digital Diaspora, November, 2012, 23, http://welcomingcenter.org/sites/default/files/digital_diaspora_final_report_-_nov_2012.pdf; Pew Internet and American Life Project, Library Services in the Digital Age, January 22, 2013, http://libraries.pewinternet.org/2013/01/22/library-services/.

25 Joshua Breitbart, “The Philadelphia Story,” New America Foundation, December 11, 2011.

26 Brian James Kirk, “The Digital Philadelphia Vision,” Technically Philly, August 12, 2009, http://web.archive.org/web/20120204135850/http://technicallyphilly.com/dp/wireless.html.

27 Ibid.

28 U.S. Department of Commerce, “A Look Ahead to 2012: NTIA by the Numbers,” The Commerce Blog, December 30, 2011, http://www.commerce.gov/blog/2011/12/30/look-ahead-2012-ntia-numbers (accessed June 24, 2013).

29 Connie Langland, “Coming Soon: A Harder GED,” The Notebook 17, no. 3 (February 2012), http://thenotebook.org/february-2012/124488/coming-soon-tougher-ged-exam.

30 Dailey et al., Broadband Adoption in Low-Income Communities, 4.

31 Ibid., 5.

32 Philadelphia Freedom Rings Partnership, “Philadelphia Freedom Rings: A Comprehensive Community Infrastructure Proposal,” Broadband USA Applications Database, http://ssl.ntia.doc.gov/broadbandgrants/applications/search.cfm.

33 National Telecommunications and Information Agency, “Notice of Funds Availability: Broadband Initiatives Program, Broadband Technology Opportunity Program,” Federal Register 75, no. 14, January 22, 2010, 3796.

34 National Telecommunications and Information Agency, “Notice of Funds Availability,” 3796. NTIA identified the primary indicator of broadband adoption as “the number of new broadband subscribers and other regular users the project will generate.” Funding cuts external to BTOP forced one SBA Managing Partner to cease operations in April 2012.

35 The National Comprehensive Center for Fathers (NCCF) closed April 2012.

36 Freedom Rings Partnership, “KEYSPOT: About Us,” http://www.phillykeyspots.org (accessed June 24, 2013). The “KEYSPOT” branding for FRP programs was rolled out in January 2012.

37 A copy of the report, “Analysis of Siting and Spatial Distribution of Public Computer Centers in Philadelphia,” is on file with the authors of this report.

Endnotes

71

38 Comcast Internet Essentials is a national program which started Summer 2011, offering $9.99 per month home Internet plans for families with children who qualify for free lunch in school. Since January 2012, Comcast has expanded eligibility to include private and parochial schools, as well as children who qualify for reduced lunch. See http://www.internetessentials.com for more details.

39 OTI obtained details regarding the development of the KEYSPOT Discount Broadband Plan through personal and e-mail correspondence with UAC and Drexel. Copies of the correspondence are on file with the authors of the report.

40 Greater Philadelphia Urban Affairs Coalition, Broadband Non-Infrastructure Application Submission to NTIA.

41 National Telecommunications and Information Administration, “Broadband USA: Grants Awarded,” under “All Recipients,” http://www2.ntia.doc.gov/all-recipients (accessed June 11, 2013); Steven J. Jackson and Andrew Gordon, “Building Community Broadband: Barriers and Opportunities for Community-Based Organizations in the Federal BTOP and BIP Broadband Development Programs,” Proceedings of the American Society for Information Science and Technology 48, 1 (2012): 1-11.

42 Jackson and Gordon, “Building Community Broadband. This article provides further information about the history of the grant application process.

43 See also Breitbart, “The Philadelphia Story.”

44 Other academic partners have also worked with the Partnership on research and evaluation projects, including Rutgers University (ethnographic research) and Temple University (GIS study).

45 Lisandra R. Carmichael et al., “Practical Approaches and Proposed Strategies for Measuring Selected Aspects of Community-Based Broadband Deployment and Use,” in “Broadband Adoption,” ed. Seeta Peña Gangadharan and Greta Byrum, special section, International Journal of Communication 6 (October 2012): 16-17.

46 In the original BTOP Round 2, Notices of Funding Availability (issued January 2010), the NTIA stated that SBA programs must report “the increase in the number of households, businesses, and community anchor institutions subscribing to broadband service, and the methodology used to measure the increase.” No definitive methodology was provided.

47 At the Community Broadband Adoption Impact and Sustainability Conference, in June 2011, a representative of the NTIA indicated the agency’s interest in understanding a continuum of broadband adoption, which suggests NTIA openness to expanded broadband adoption definitions. See http://cmediachange.net/2011/06/28/btop-user-v-subscriber-v-adopter/. However, the NTIA has never formally announced revisions to its definitions of users and subscribers.

48 Though the program data reported to the NTIA do include youth, the data is less robust and rich in detail compared to other data collection efforts.

49 At the time of this writing, UAC issued an email to the Evaluation Working Group, describing discussions with the NTIA to revise its subscription methodology, “Based on the feedback from NTIA, [UAC is] exploring ways to revise [the] methodology to more accurately capture the impact of [the KEYSPOT] program, including attributing a reasonable percentage of Comcast Internet Essentials subscribers in Philadelphia to the work of [the Freedom Rings Partnership], looking to potential publicly available data sources, and/or even doing a survey.” A copy of the correspondence is available with the authors of the report.

50 U.S. Department of Commerce, “Protection of Human Subjects,” Code of Federal Regulations 15, §27.101(b).

51 Originally, the evaluation plan included cohort focus groups with participants and Webguides, as well as two interviews with staff at different time periods. However, in light of the delays in getting the NTIA certification, OTI revised its methods to fit a compressed timeline and changed the cohort focus groups to be one-time groups in order to reach more participants and Webguides and decided on one interview with staff instead of two. The decision to modify the cohort design of the focus groups was also made due to the fact that we would have participant attrition over time.

52 Federal Communications Commission, “Internet Access Services: Status as of December 31, 2010,” October 2011, http://hraunfoss.fcc.gov/edocs_public/attachmatch/DOC-310261A1.pdf; Fenton, “Broadband Adoption Map.”

53 To plan the original grant service areas, the FRP utilized a 2008 home broadband subscription dataset from the Knight Center for Digital Excellence. However, the methodology for data collection was never clear. As the FCC released its 2010 data in 2012, OTI uses this dataset as a more accurate representation of subscription rates during the KEYSPOT program time period.

54 Prior to receiving final NTIA certification to implement primary data collection, OTI analyzed these Partner Monthly Reports in developing formative Quarterly Evaluation Reports for the Partnership. Starting in Q3 2012, OTI focused its resources on implementing the primary data collection methods and analysis. For this

Report, OTI did not systematically analyze the monthly reports as similar questions were asked using the primary data collection instruments.

55 In the first year and a half of the grant, Philadelphia Parks and Recreation (PPR) used a burdensome paper and Excel sheet system to monitor basic usage statistics. Starting in August 2012, PPR used Google Forms and Excel spreadsheets to track open access lab participants and training sessions. Data collected include participant gender, age, and length of lab use. More detail about PPR is on file with the authors of the report.

56 Kathleen M. Mazor et al, “A Demonstration of the Impact of Response Bias on the Results of Patient Satisfaction Surveys,” Health Services Research 37,5 (2002): 1403–1417.

57 As of May 2013, Drexel and partners completed the creation of the e-learning system to serve as a resource sharing and participant tracking tool for trainers.

58 Email correspondence dated May 23, 2013 is on file with the authors.

59 Email correspondence dated May 23, 2013 is on file with the authors.

60 Horrigan, “Broadband Adoption and Use in America”; Pew, Philadelphia 2010.

61 Federal Communications Commission, “Internet Access Services: Status as of December 31, 2010.”

62 Exploring a Digital Nation, NTIA (p. vi).

63 See also Dailey et al, Broadband Adoption in Low-Income Communities.

64 Ibid, 14.

65 Paul DiMaggio and Ezster Hargittai, “From the ‘Digital Divide’ to ‘Digital Inequality’: Studying Internet Use as Penetration Increases.” (Princeton, NJ: Princeton University, 2001), 11; Colin Rhinesmith, “Free Library Hot Spots: Supporting Broadband Adoption in Philadelphia’s Low-Income Communities,” in “Broadband Adoption,” ed. Seeta Peña Gangadharan and Greta Byrum, special section, International Journal of Communication 6 (October 2012); Ricardo Gomez and Elizabeth Gould, “The “cool factor” of public access to ICT: Users’ perceptions of trust in libraries, telecentres and cybercafés in developing countries,” Information Technology & People, 23, 3 (2010): 247-264.

66 Charles M. Davidson, Michael J. Santorelli, and Thomas Kamber, “Toward an inclusive measure of broadband adoption,” International Journal of Communication 6, (2012): 1-20.

67 Ricardo Gomez and Shaun Pather, “ICT evaluation: Are we asking the right questions?” Electronic Journal of Information Systems in Developing Countries 50, 1 (2012). Gomez and Pather discuss the importance and challenges of evaluating the “intangible outcomes” of ICT projects in developing countries.

68 Batya Friedman, Human values and the design of computer technology. (Stanford: CSLI Publications, 1997).

69 Keith Hampton, “Internet Use and the Concentration of Disadvantage: Glocalization and the Urban Underclass”. American Behavioral Scientist 53, 8 (2010): 1111-1132; Amy Bach, Gwen Shaffer, and Todd Wolfson, “Digital Human Capital: Developing a Framework for Understanding the Economic Impact of Digital Exclusion in Low-Income Communities,” Forthcoming in Journal of Information Policy.

70 Caroline Haythornthwaite and Lori Kendall, “Internet and Community,” American Behavioral Scientist 53, 8 (2010): 1083-1094.

71 U.S. Bureau of the Census, 2009-2011 American Community Survey 3-Year Estimates, http://factfinder2.census.gov.

72 Michael Nutter, “The 21st Century-Ready Workforce,” Citizen IBM blog, entry posted on March 19, 2012, http://citizenibm.com/2012/03/the-21st-century-ready-workforce.html (accessed June 11, 2013).

73 Arno Bergstrom et al., “Collaboration Framework: Addressing Community Capacity,” National Network for Collaboration, 1995, http://www.uvm.edu/extension/community/nnco/collab/framework.html, (accessed June 11, 2013).

74 Jackson and Gordon, “Building Community Broadband.” See also the History and Methodology section of this report.

75 Bruce Tuckman, “Developmental sequence in small groups,” Psychological Bulletin 63, 6(1965): 384–99.

76 S. A. Brown, V. Venkatesh, and H. Bala, “Household Technology Use: Integrating Household Life Cycle and the Model of Adoption of Technology in Households,” Information Society 22, 4 (2006): 205-218; Neil Selwyn, “The digital native myth and reality,” Aslib Proceedings, 61, 4 (2009): 364-379.

77 Dorothea Kleine, Technologies of choice?: ICTs, development, and the capabilities approach (Cambridge,: MIT Press, 2013).

72

APPENDIX A: Quantitative Methodology: Long-Form Workstation User Survey (WUS)

METHOD

OTI implemented the WUS in two different ways during t he e va lu at ion . T he Shor t Work st at ion User Su r ve y w a s a n on l i ne s u r ve y w h ic h c ol le c t e d d e mo g r a ph ic information on K EYSPOT participants. The Managing Partners collectively determined the survey questions and method of administration for the W US. Most notably, some Managing Partners had concerns around collecting information on participant education and income levels. These demographic characteristics were excluded from the sur vey. Additionally, partners agreed that no unique identifiers would be collected to track participants over time. Partners wanted the sur vey to be an ongoing data collection tool, so the Partnership worked with Cognis, the IT support for the partnership, to have all KEYSPOTS terminals automatically launch the survey upon participant log-in. For SBA partners using the sur vey, OTI shared custom sur vey l inks for each Managing Partner to use during their trainings.

Three K EYSPOTS piloted the W US and provided feedback to OTI in July 2012 , and additional refinements to t he su r ve y appe a ra nce a nd word i n g were made i n S e pt e m b er 2 012 . By t he e nd of D e c e m b er 2 012 , 59 K EYSPOT sites sent in some data ind icating that the survey was used at their sites, representing 8 of 9 managing pa r t ners . Ph i ladelph ia Pa rk s a nd R ecre at ion d id not deploy the survey at their sites as their sites serve primarily youth and youth under 18 did not participate in this data collection. Other youth-only KEYSPOTS did not deploy the sur vey either.

Fo r t h e L o n g W U S , f ro m O c t o b e r 17 - 2 3 a n d December 5 - 11, 2012 , OTI added questions to the Short W US to assess other participant characteristics, such as

mode of transportation to the K EYSPOT, main reasons for coming to a K EYSPOT, home Internet subscription, and frequency of Internet use. For the final report, OTI only ana lyzed data from the Long W US as it contains these additiona l questions to assess both demographic and adoption characteristics of KEYSPOT characteristics. A lso, it represents a t ime period in which most of the sites already had the sur vey running; one of the last sites implemented the sur vey in October. Participants in this lon g W US d at a set repre sent 43 site s across t he cit y, representing 8 of 9 ma na g ing pa r t ners . There may be different explanations for why fewer sites are represented in the long W US than the Short W US . One reason is sur vey fatigue: for sites that ran the short W US from the su mmer of 2012 , their participants may have routinely closed out of the sur vey by the time of the long W US deployments.

ANALYSIS

OTI used PSPP (an open-source software version of SPSS) and Excel to create frequency tables and run statistical tests of signif icance bet ween demographic groups and home subscription rates. ArcGIS soft ware was also used to map the W US data by ZIP code.

Long WUS Response Rate:

A . 538 respondents who are over 18 and consent / 1055 respondents over 18 = 51.0% .B. 538 respondents who are over 18 and consent / 1472 total respondents = 36.5%

LIMITATIONS

While a significant percentage of host sites ser ving adult participants ran the W US during the long-form W US deploy ment , t here a re st i l l l im it ations in inter preting the findings, such as self-selection and social desirabilit y bias. Given that the WUS is a sur vey administered via the computer, only participants who are confident in their mouse and keyboard skills are able to fil l the sur vey out. This means that the W US findings may not accurately re pre s e nt nov ic e u s er s . A l s o , b e c a u s e t h i s s u r ve y i s

73

volu nta r y, it is more l ikely t hat pa r ticipa nts who have strong opinions (positive or negative) about KEYSPOTS filled it out. For participants without strong opinions, they may not want to take the time to fil l out a sur vey only to report on average experiences. Socia l desirabilit y plays a role in how participants f i l l out a sur vey as well; they may be more l ikely to prov ide answers that they think program staff want to hear rather than their true opinions. For example, on the long form W US, there is a question around main reason for coming to the K EYSPOT. If a participant’s main reason for going to the K EYSPOT is to check out Facebook, he or she may not feel comfortable selecting that answer given that it isn’t as “meaningful ” as searching for a job online.

With regards to sur vey design and implementation, the WUS does not track individuals over time and it does not represent anyone under 18. A lso, since there are no u nique identif iers for su r vey responses , t hese d at a do not ref lect individual persons. Lastly, these sur veys are voluntary and participants can skip questions if they want to, so some questions have missing data.

LONG-FORM WORKSTATION USER SURVEY QUESTIONNAIRE

1. How did you get to this computer center today?❏➜Using public transportation❏ In a car, truck, or other vehicle❏ By walk ing or bik ing❏ Other

2 . What is the main reason you are here today?❏ To take a class❏ To send or read e-mail❏➜To look (or apply) for a job online❏➜To get news online❏ To shop online❏ To watch videos, play games, or listen to music❏ To study or lea rn (exa mples: to prepa re for t he

GED or do homework)❏➜➜To visit a government website (examples: to apply

for benefits or pay a park ing ticket)❏ To u se a soc ia l net work i n g site l i ke MySpace ,

Facebook, or Linked❏ To create or work on your own online journal, blog,

video, or audio❏ To pa r ticipate in you r com mu nit y (exa mple: to

organize with neighbors)❏ To do any bank ing online❏ Other: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

3. About how often do you use the Internet or email?❏ Several times a day❏ About once a day❏➜3 -5 days a week❏ 1-2 days a week❏ Ever y few weeks❏ Less often❏ Never❏ Don't k now

4. Do you have Internet service at home? ❏ Yes ❏ No

5. If you answered “No” to Question #4, what is the main reason you do not have Internet at home?❏ Just don’t k now how❏ Don’t have a computer❏➜Can get online somewhere else❏ Cost (it's too expensive)❏ It’s a waste of time (don’t need it)❏ It's too difficult/frustrating❏ Worried about privacy or securit y❏ Other _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

6. Do you have any comments (about this computer center, the classes a nd progra ms of fered here, this sur vey, etc)?

7. Have you been to this computer lab before? ❏ Yes ❏ No

8. What is your zip code? _ _ _ _ _ _ _ _ _

9. What is your gender? ❏➜M ❏ F ❏ Other or self-defined

74

10. Are you of Hispanic, Latino, or Spanish origin?❏ Yes ❏ No

11. What is your race? Mark one or more boxes.❏ White❏ Black or African American❏ American Indian or A laska Native❏ Asian❏ Native Hawaiian or Other Pacific Islander❏ Other race: _ _ _ _ _ _ _ _ _ _ _ _ _ _

12 . What is your age?_ _ _ Years

APPENDIX B: Quantitative Methodology: Standardized Exit Survey (SES)METHOD

T h i s on l i ne su r ve y wa s ad m i n i stered at t he end of a c l a s sro om-ba s e d , i n s t r uc tor- le d t r a i n i n g to c a pt u re information on participant outcomes and satisfaction. Questions f rom this su r vey related to the Pa r tnership sha red goa ls of i ncrea si n g adopt ion, educ at iona l a nd workforce opportunities, and communit y engagement. The process of developing the exit sur vey was similar to the W US, in that partners led the question development w it h t he g u id a nce of OT I . W h i le ma ny pa r t ners felt that for novice training participants, a paper and pencil sur vey would be better suited for their populations, they all agreed that an online sur vey administration would be less burdensome on the trainers and program managers. OTI developed the Google Form to share with managing partners to enable easy sharing of the participant data on the backend.

The Standardized Exit Sur vey was finalized in Spring 2012 and OTI shared custom link s for each managing

pa r tner to u se w ith their tra inings. From Ju ly 2012 to December 2012 , seven managing partners used the exit su r vey w it h some of t hei r t ra i n i n g pa r t icipa nt s . OT I empha si zed t hat pa r t ners shou ld not u se t h i s su r ve y with youth, as youth were excluded from the evaluation. Trainers were encouraged to administer this sur vey at the last session of their class.

ANALYSIS

OTI used PSPP (an open-source software version of SPSS) and Excel to create frequency tables and graphs.

Standardized Exit Survey Response Rate: 105 respondents who are over 18 and consent / 113 total respondents = 92 .9%z

LIMITATIONS

The main limitation of this instrument is low usage by t ra i ners a nd t herefore , l i m ited repre sent at i vene s s of t he Pa r tnership. This instr u ment wa s not w idely u sed across all trainings, perhaps due to its late development and difficult y in sharing the sur vey protocol with all of the trainers. A lso, ma ny par tners have their ow n post-test eva lu ation su r veys, a nd may have felt t hat add ing another exit sur vey would have been too much for their pa r ticipa nt s . W h i le t he on line ad m in istration of t he for m wa s suppor ted by pa r t ners , t h i s su r ve y d id not automatically launch upon logging into a computer like the W US did for most participants, which may have led to decreased use. Ultimately, implementing this sur vey at the later stage of the evaluation planning process relied on too many layers of communication: OTI shared links with managing partners, who then shared them with trainers, who then had to direct their class to an online sur vey.

L i k e t h e W U S , t h e e x i t s u r v e y d i d n o t t r a c k i nd i v idu a l s over t i me; t h i s me a nt t h a t w it h re g a rd s to home I nt er ne t s u bs c r ipt ion , OT I c ou ld on l y a sk about participant interest and intent rather than actual behaviors. Also like the W US, the exit sur vey deals with problems of self-selection a nd socia l desirabi l it y bia s . Only those who are interested in providing their opinions (typically very positive or negative) will, and given that the

75

exit sur vey asks for participants’ assessment of a training, they are more likely to provide positive feedback.

STANDARDIZED EXIT SURVEY QUESTIONNAIRE

1. What k ind of training did you take today? Please choose one.

a . Basic computer/Internet sk il lsb. Job search/readinessc. Multimedia classd. GEDe. Microsoft Office sk illsf. Certified training programs (like ICDL) g. Other (please describe: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _)

2 . How d id you le a r n a bout t h i s t ra i n i n g? Ple a se choose one.

a . Friend or family memberb. R adioc. Newspaperd. Flyere. SEPTA adf. This centerg. An eventh. Other way

3. Have you completed a computer training or class before this one?

a. Yesb. No

4. Did you learn what you wanted from this training?a. Yesb. No

5. Did this training help you learn job sk ills?a. Yesb. No

6. Did this training help you find a job?a. Yesb. No

c. I ’m not look ing for a job.

7. Did this training help you connect to educational opportunities (like finding online classes, applying to college, or learning about financial aid)?

a. Yesb. Noc. I ’m not look ing for those opportunities.

8. Where do you use Internet? Check all that apply.a . At a coffeeshop or restaurantb. At homec. At t h is lab, a not her publ ic computer center, or

librar yd. At work or schoole. Any where, using a smart phonef. I do not access the Internet

9. D i d t h i s t r a i n i n g m a ke yo u w a n t a n I n t e r n e t connection at home?

a. Yesb. Noc. I a lready have Internet at home.

10. If you do not have an Internet connection at home, how likely are you [5] to sign up for one in the

next six months?a. Ver y likelyb. Somewhat likelyc. Not likely at alld. I a lready have Internet at home.

11. Now that you have finished this training, will you come back to this center in the next six months?

a. Yesb. Noc. Not sure

12 . Wou ld you recommend this center to a fa mily member or friend?

a. Yesb. No

76

c. Not sure

13. What is your age? _ _ _ _ _ _

APPENDIX C: NTIA Quarterly and Annual ReportsA ll Ma na ging Par tners track key outputs for the FR P programs in Partner Monthly Reports, which the Primes t hen re v ie w a nd compi le i nto Pr i me s’ Qu a r terly a nd Annual Reports to the NTIA. All NTIA Quarterly and Annual Reports are publicly available on the NTIA BTOP website: http://w w w2 .ntia .doc.gov/.

ANALYSIS

OTI relied on Primes’ Qua r terly Repor ts to track t wo key outputs for each grant: SBA training hours and participants, and PCC sites and average users per week. OTI also examined the Primes’ 2011 Annual Report to the NTIA (the most recent report available at the time of the report's release) to determine the range of training hours and topics offered by PCC and SBA programs. At the time of the report writing, the Primes’ 2012 Report to the NTIA annual report was not available.

APPENDIX D: Qualitative Methodology: Focus Groups with ParticipantsMETHODS

OT I de veloped a n i nter v ie w que st ion na i re for foc u s groups with participants who use the computer centers and/or training programs. In February 2012 , OTI worked with the Primes to develop and finalize these scripts. OTI sought participants from sites affiliated with all eleven of the Managing Partners through f liers and/or in-person recruitment. To minimize the invasiveness of research, OTI d id not col lect pa r ticipa nt demographics du ring

the focus groups. OTI conducted three participant focus groups:

Fo u r p a r t i c i p a n t s w h o u s e d t h r e e K E YS P OT S operated by three different Managing Partners attended the focus group at the Independence Branch Librar y on July 16, 2012 .

F o u r t e e n i n d i v i d u a l s f r o m We s t P h i l a d e l p h i a who u sed f ive K EYSPOTS operated by fou r d if ferent Ma n a g i n g Pa r t ner s at tended t he foc u s g rou p at t he People’s Emergency Center’s Rowan House on September 25, 2012 .

Ten participants who used six K EYSPOTS operated by four different Managing Partners attended the focus group at the Philadelphia OIC on November 9, 2012 .

ANALYSIS

OTI used the following steps to manage and analyze the focus group information: Two coders created a participant codebook to categorize the focus group information by key topical themes, adding new codes as needed. Using a coding sheet, coders categorized statements/quotes under specif ic codes . Coders t hen a na ly zed a nd a g g regated codes. The coders discussed discrepancies, deliberated, and u ltimately harmonized dif ferences to arrive at the same codes.

LIMITATIONS

F o c u s g r o u p m e t h o d o l o g y i s l i m i t e d i n i t s representativeness, as only a small number of people can pa r t icipate i n each d iscu ssion . However, focu s g roup d at a prov ide s more i n-dept h accou nt s of pa r t icipa nt experiences and progra m phenomenon than statistica l snapshots of behavior, values, or beliefs. Given that OTI does not follow the same participants across participant focus groups, we are unable to track long-term outcomes such as jobs attainment or improved educational status.

R e c r u i t m e nt p r i m a r i l y t o o k p l a c e o n s i t e , w i t h OTI staff or partner staff (on behalf of OTI) ask ing for volunteers. As is the case with focus groups, the process of self-selection for pa r ticipation mea ns thou ghts a nd opinions often ref lect enthusiastic or positive assessments (especia l ly for a n eva lu ation t hat is f ra med a s helping

77

to su s t a i n K E YSPOTS a nd prog r a m s l i ke it). S ome pa r ticipa nt s ca me w it h a f r iend or a group of f r iend s, potentia l ly compou nd ing a problem in herent in focu s groups—na mely, t hat t hey inspire consensu s or group t h i n k , rat her t ha n d isa g reement or t he expre ssion of divergent views. In addition, the fact that the sur vey data indicate only a majority of African American participants and roughly equal representation across gender, whereas a l l three focu s groups d rew ma in ly A f rica n-A merica n women, ra ises quest ions about represent at iveness . To respect concerns of invasiveness (a concern raised in needs assessments and in meetings of the Evaluation Work ing Group), the specific demographic profile of participants was not recorded; OTI identified race and gender by sight, not by ask ing participants to articulate race and gender themselves. This lack of precision introduces additional problems of context.

S i n c e o u r f o c u s g r o u p s a r e s e m i - s t r u c t u r e d , p a r t i c i p a n t s h a ve t h e f r e e d o m t o d e v i a t e f r o m t h e ma i n l i ne of que st ion i n g. W h i le t y pic a l ly a st ren g t h of qu a l it at ive re se a rch, t he abi l it y to ex plore t heme s conf licts with the hour and a half length of a focus group, a nd t i me con st ra i nt s l i m it ou r abi l it y to d ive deeper into tangential conversations. Finally, each focus group represented unique circumstances that may have impacted the information shared, thus limiting the conclusions we can draw from the conversation. When the groups were held later in the day, we found that energ y levels were low and sometimes it was difficult to get rich discussions on certain topics.

PARTICIPANT FOCUS GROUP QUESTIONNAIRE

1. What computer center(s) do you go to? What types of trainings did you attend?

a. When did you first start going to a computer center/K EYSPOT

b. Why do you go there?2 . What did you hope to get out of coming to the

computer center(s)? (Probes: specif ic sk il ls , job, education, health, benefits, sense of communit y)

a. Do you feel like you are getting what you wanted?b. Is there more that you wou ld like to learn? If so,

what and why?3. How easy or hard was it for you to get to centers?

(Probe for stories: What made it easy? what made it hard? transportation, childcare, job scheduling)

4. How e a s y o r h a rd w a s i t f o r yo u t o c o m p l e t e tra inings? (Probes for stories: schedu ling , don’t have enough time/equipment to practice, forgot what I learned)

a. What makes it hard to finish a training you started?5. How did you feel about using a computer and the

Internet before the tra ining /coming to the lab? (probes: ner vous, excited, k new a little bit , k new a lot).

a . T h i n k a b o u t t h e f i r s t t i m e y o u c a m e t o a K EYSPOT, or the first day or training. How did you feel?

6. Now that you’ve gone through this training/come to the lab, how do you feel about using a computer and the Internet?

7. How are you using the skills you learned? (Probes: email, social media, job applications, online classes, look ing for health information)

8a. Now we want to learn a little bit about whether or not you have Internet at home or on your phone. Please raise your hand if this is true for you.

Poll #1: Who has a computer at home? This can be a laptop or a desktop.

Poll #2: H o w m a n y p e o p l e h a v e a n I n t e r n e t su bscr ipt ion for t hei r home computer or laptop?

Poll #3: How many people have cell phones? Poll #4: How many people have phones that can access

the Internet? a “smart phone”? Poll #5: How many people can get Internet on your

phone?8b: For those of you who have Internet at home or on

your computers, did you sign up for it after you attended a training or a visited a computer center?

a. For people who don’t have Internet at home, do you think you will sign up? Why or why not?

b. I k now some of you a re look i n g for jobs , w hat wou ld it t a ke for you to get t he Inter net u nder

78

these conditions?9. Do you feel comfortable/at home/safe at trainings/

at this computer center? 10a. We’re going to a sk a few questions a ga in, a nd

please raise your hand if it applies to you. Poll #1: How many people come to the labs/trainings

for the communit y experience? Poll #2: How many of you come to a lab to just focus

on using the computer or getting a training? (explain: you just want to focus on your goal without distractions?)

10b: How have you u sed computers a nd/or d igita l skills to connect with or help your communit y? (Probe for stories: - neighbors, friends, church or school, family? Others who use this center?)

11. Th in k ing about ever y t h ing we’ve ta l ked about t o d a y, w o u l d yo u r e c o m m e n d t h e c o m p u t e r computer center you use to others? Why? (Probes- How well were these classes/labs designed for you? location, scheduling, facilit y, trainings offered).

12 . Does anyone have any final comments they want to share before we wrap up?

APPENDIX E: Qualitative Methodology: Focus Groups with WebguidesMETHOD

OT I de veloped a n i nter v ie w que st ion na i re for foc u s groups with Webguides who teach SBA digital l iteracy t r a i n i n g c l a s s e s o r w h o s t a f f t h e p u b l i c c omp u t i n g centers. OTI sought Webguides from sites representing a l l Managing Partners through f l iers and/or in-person recruitment. We did not survey Webguide’s demographics during focus groups. OTI conducted t wo focus groups w ith 13 individua l webguides represented bet ween the t wo groups:

E l e ve n We b g u i d e s a t t e n d e d t h e f o c u s g ro u p a t t he I ndep endenc e B r a nc h L ibr a r y on Ju l y 2 3 , 2 012 , r e p r e s e n t i n g s e v e n Pa r t n e r s a n d e l e v e n d i f f e r e n t

K EYSPOTS.Eig ht webg u ides at tended t he focu s g roup at t he

Independence Bra nch Libra r y on November 2 , 2012 , representing five Managing Partners and eight different KEYSPOTS. Six of the Webguides present had attended the prev ious focus group, w ith t wo new Webgu ides in attendance.

ANALYSIS

OTI used the following steps to manage and analyze the focus group information: Two coders created a Webguide codebook to categorize the focus group information by key topical themes, adding new codes as needed. Using a coding sheet, coders categorized statements/quotes under specif ic codes . Coders t hen a na ly zed a nd a g g regated codes. The coders discussed discrepancies, deliberated, and u ltimately harmonized dif ferences to arrive at the same codes.

LIMITATIONS

F o c u s g r o u p m e t h o d o l o g y i s l i m i t e d i n i t s representativeness, as only a small number of people can pa r t icipate i n each d iscu ssion . However, focu s g roup d at a prov ide s more i n-dept h accou nt s of pa r t icipa nt experiences and progra m phenomenon than statistica l snapshots of behavior, values, or beliefs.

Indicators which rely on focus group material from Webguides provide a relatively representative snapshot of f ront l ine sta f f, their experiences, a nd obser vations. Though focus groups tend to promote group think or consensus, recru itment of Webgu ides took place w ith the help of staf f at Managing Partners, some of whom told OT I t hat t hey i ntent iona l ly su g ge sted na me s of i nd i v idu a l s w it h va r ied opi n ion s of K E YSPOTS a nd FR P programs. Thus, Webguide focus groups were not as prone (as is typically the case) to agreement. With that said, the analysis of Webguide focus groups had its f laws: for reasons of cost and time, OTI did not connect the thoughts and opinions of Webguides who participated in both discussions, meaning it was impossible to triangulate bet ween statements made across conversation.

A lso, since ou r focu s g roups a re sem i-str uct u red ,

79

p a r t i c i p a n t s h a ve t h e f r e e d o m t o d e v i a t e f r o m t h e ma i n l i ne of que st ion i n g. W h i le t y pic a l ly a st ren g t h of qu a l it at ive re se a rch, t he abi l it y to ex plore t heme s conf licts with the hour and a half length of a focus group, and time constraints limit our abilit y to dive deeper into tangential conversations. Additionally, each focus group represented unique circumstances that may have impacted the information shared, thus limiting the conclusions we can draw from the conversation.

WEBGUIDE FOCUS GROUP QUESTIONNAIRE

1. Which site do you work at? When did you start? 2 . Describe where you work. (Probes: location, people

ser ved, classes/ser vices offered) 3. Before this position, what was your situation like?

(probes: life in general, prior employment histor y, school situation)

4. How did you hear about this job and how did you get hired?

5. Why did you accept the position?a. What did you want to get out of the job?6. Why do you think participants come to your site? 7. How do you see your students mak ing progress at your site? a . Adopt ion- i nc re a si n g u se of tec h nolog y/more

confident, b. J o b s /e d u - n e w e d u c a t i o n a l o p p o r t u n i t i e s o r

students getting jobs, c. Communit y engagement- connecting with others

online or on site 8. W hat do you see a s cha l len ge s to pa r t icipa nt s

a c h i e v i n g t h e i r g o a l s ? ( P r o b e s : s c h e d u l e s , transportation, childcare, competing life priorities)

9. What are some cha llenges of being a webgu ide? (probes: mixed-sk ills class, retention of students, issues working with students, technology problems, class setup, travel)

10. How have you addressed these challenges? 11. From you r perspec t i ve , how do you t h i n k t he

K EYSPOTS can be improved? 12 . W h a t ro l e d o e s t h e K e y s p o t /c o m p u t e r s i t e

play i n t he com mu n it y? W hat about cre at i n g

communit y? 13. What do you feel like you have achieved or learned

in this position so far? 14. How do you think this job w il l help you in the

future? (Probe: employment opportunities, new job sk ills, connecting with new people, etc.)

15. Is there anything else that you would like to share about your experience as a webguide?

APPENDIX F: Qualitative Methodology: Interviews with BTOP Staff (Host Sites, Managing Partners, and the Primes)METHOD

Altogether, OTI conducted 18 unique inter views with 20 interviewees representing 16 partners (five Host Sites, nine Managing Partners and the two Primes). With Managing Par tners that received both a PCC and SBA grant, we gave the option of separate inter views with a Managing Partner staff person focused on each grant. Two host sites’ asked that an additional staff person participate in their inter view, feeling that together they could better answer questions. In both instances, this meant a managerial-level inter viewee and the Webguide that worked most closely with participants.

Interviews with Host Sites: OTI conducted interviews with seven inter viewees at five host sites representing five Managing Partners. Three of the host sites are located in t wo of the three target geographic areas identified by the FR P (North and South). The remaining t wo Host Sites are located downtown and provide ser vices cit y wide. In add ition to t heir location, OTI chose Host Sites t hat ser ved some of the target populations of priorit y to the FR P. The K EYSPOTS are embedded within Host Sites who specialize and work directly with Spanish-speakers, seniors, youth, low-income communit y members and/or

80

individuals experiencing homelessness and unemployment. The questions OTI asked these Host Site staff revolved around the benefits, challenges and goals of the broader F R P a n d t h e i m p a c t t h e y s e e o f t h i s p r o g r a m o n participants.

Inter v iews w it h Ma na g ing Pa r t ners/Pr i mes: OT I conducted inter views with 13 inter viewees at 9 Managing Pa r t ners orga n i z at ions a nd w it h t wo st a f f of t he t wo Primes of the Sustainable Broadband Adoption (SBA) a n d P u b l i c C o m p u t i n g C e nt e r (P CC) g r a nt s . OT I inter viewed one SBA grantee who later closed operations in the middle of the grant. One partner did not respond to inter v iew requests a nd was not inter v iewed for this repor t . OTI a sked the Ma na ging Pa r tners a nd Primes the same questions, covering the benefits, challenges and goals of the FR P and the impact they see of this program on participants.

ANALYSIS

OTI wrote a one-page summar y document that describes a single inter viewee's answers to each of the questions in the interview scripts and that drew out illustrative quotes. Add it iona l ly, OT I rev iewed each i nter v iew for major points made, then analyzed/elicited themes across these inter views based on this initial analysis. OTI wrote brief summaries for each group of sources (Host Sites and the Managing Partners/Primes). Each summar y included a description of the inter v iews, a bu llet-point l ist of key findings, with relevant quotes. After wards, OTI wrote a summar y analysis across Host Sites, Managing Partners/Primes, and A llied Organizations.

LIMITATIONS

Indicators that rely on inter v iew materia l w ith staf f at Ma na ging Pa r tners a nd Primes prov ide a n adequ ately represent at ive snapshot of ma na ger ia l-level at t it udes , beliefs, and k nowledge. Only one organization did not respond to inter view requests.

Indicators that rely on inter view material with staff at Host Sites are less representative than material with staff at Primes a nd Ma na ging Pa r tners or w ith Webgu ides. OTI asked staff at Managing Partners for contact names

at Host Sites of interest and also created a list of potential inter viewees based on our k nowledge of the breadth of the Partnership. The snowball method of sampling tends to favor the selection of inter viewees who share similar views as those who suggested them, while excluding those with dissenting or disparate views. Additionally, the total nu mber of organizations represented in inter views is a fraction of the total number of Host Sites, which suggests the Final Report may exclude a wide swath of opinions and thoughts about the FR P.

BTOP STAFF INTERVIEW QUESTIONNAIRE

1. Tell me about your role in the Partnership.- What population do you ser ve? 2 . What benefits have you experienced as a result of

participating in the Partnership? (Probes: shared resources, shared knowledge, getting support from Partnership. Solicit actual examples)

3. W hat a re cha l len ge s f rom pa r t icipat i n g i n t he Pa r t nersh ip? (Probe s: repor t i n g requ i rement s , g o a l s n o t t h e s a m e , s t a f f /o r g . c a p a c i t y f o r implementation)

- What support did you want from the Partnership but did not receive?

4. Were you i ncluded i n deci sion-ma k i n g for t he Partnership? Please describe.

5. What are the Partnership's most important goals?- How have t he goa ls of t he Pa r t nersh ip cha n ged

over time? 6. How do your organization’s goals overlap or differ

from these shared goals? 7. How ha s being in the Pa r tnership a f fected you r

organization’s ability to achieve your organization's goals? Please share stories.

8. How has being in the Pa r tnership af fected you r organization’s abilit y to achieve the Partnership’s shared goals? (Probes: jobs, education, community engagement, adoption). Please share stories.

9. To what extent have the Partnership's goals been met? How so? Please share stories.

10. What does the term “partnership” mean to you? 11. Think ing about ever y thing we discussed today,

81

how well does the Freedom R ings Partnership fit that description?

12 . How e f f e c t i ve i s a p a r t ne r s h i p s t r u c t u re for managing the BTOP grant?

13. W h a t a r e c o r e c o m p e t e n c i e s n e e d e d f o r a P a r t n e r t o b e s u c c e s s f u l (e x p e r i e n c e w i t h budgets, management, training, scheduling, data management)?

14. How has your organization’s capacit y to manage the program changed over time?

15. What do you think will happen to the Partnership af ter the BTOP grant ends? What do you want to happen to t he Pa r t nersh ip? W hat c a n you r organization best contribute to realize that vision?

16. W h at wou ld you l i ke to see h a pp en w it h t he KEYSPOTS /computer site after the grant ends?

APPENDIX G: Qualitative Methodology: Interviews with Allied OrganizationsMETHOD

OT I conduc ted i nter v iews w it h 7 i nter v iewee s at si x non-BTOP funded Allied Organizations that primarily collaborated w ith six BTOP-f unded Host Sites and/or Ma na g ing Pa r t ners . Add it iona l ly, t h ree of t he A l l ied Orga n i z at ion s worked w it h t he FR P more genera l ly. O T I c h o s e A l l i e d O r g a n i z a t i o n s t h a t s e r v e d t h e target popu lations of priorit y to the FR P, represented d if ferent geographic a rea s of Philadelphia , a nd whose c o l l a b o r a t i o n s e q u a l l y r e p r e s e n t e d P C C a n d S BA Ma na ging Pa r tners . These A l l ied Orga nizations have exper t ise w it h popu lat ions st r u g gl ing w it h low-levels of education, high inca rceration rates, neighborhoods w it h h igh crime, vetera ns, t he u nemployed , or on t he wider impact of technolog y in Philadelphia. OTI asked allied organizations about K EYSPOT awareness as well as the benefits , cha llenges, goa ls and stories of impact of the FR P.

And one Allied Organization asked that an additional

staf f person participant in their inter v iew, feeling that together they could better answer questions.

ANALYSIS

OTI wrote a one-page summar y document that describes a single inter viewee's answers to each of the questions in the interview scripts and that drew out illustrative quotes. Add it iona l ly, OT I rev iewed each i nter v iew for major poi nt s made, t hen a na ly zed/el icited t hemes ba sed on this initial analysis. OTI wrote brief summaries for each group of sources (K EYSPOTS, BTOP staff, and A llied organizations). Each summar y included a description of the inter v iews, a bu llet-point l ist of key f indings, w ith relevant quotes. After wards, we wrote a summary analysis across all three groups of sources.

LIMITATIONS

I nd ic ators t h at rel y on i nter v ie w m ater ia l w it h st a f f at A l l ied Or g a n i z at ion s a re le s s repre sent at i ve t h a n material with staff at Primes and Managing Partners or with Webguides. To k now which organizations to reach out to for inter views, OTI asked Managing Partners and the Primes for recommendations. The snowball method of sampling tends to favor the selection of inter viewees who sha re simila r v iews as those who su ggested them, w h i l e e xc l u d i n g t h o s e w i t h d i s s e nt i n g o r d i s p a r a t e views. Additiona lly, the tota l nu mber of organizations represented in inter views is a fraction of the total number of Allied Organizations, which suggests the Final Report may exclude a wide swath of opinions and thoughts about the FR P.

ALLIED ORGANIZATION STAFF INTERVIEW QUESTIONNAIRE

1. Tell me about your organization and your role at your organization.

2 . How did you first learn about the Freedom R ings Partnership?

3. Do you think that the community you ser ve knows about the Partnership/computer centers/trainings? Why or why not?

4. What is your impression of the Partnership?5. From your perspective, what are its main goals?

82

a. Have you seen these goals change over time? 6. How have you or your organization contributed

to these goals? 7. Are there any particular stories that describe how

you helped the Partnership meet these goals? 8. How ha s you r orga n i z at ion benef ited f rom t he

Partnership? 9. How have prog ra m pa r ticipa nt s benef ited f rom

the Partnership? 10. How have u nderser ved commu nities benef ited

from the Partnership? 11. What, if any, disadvantages or challenges have you

obser ved of the Partnership? 12 . How do you see your site’s involvement with the

Partnership in the future?

APPENDIX H: Qualitative Methodology: Partnership VisualizationMETHOD

OTI developed a visual representation of the Partnership’s m a ny i nter n a l a nd e x ter n a l relat ion sh ips w it h ot her sta keholders . Th is prel imina r y a na lysis wa s prompted b y a n e x e r c i s e t h a t O T I l e d d u r i n g a M a r c h 2 01 2 Pa r t n e r s h i p - w i d e c e l e b r a t i o n o f a c c o m p l i s h m e n t s , where Ma na ging Pa r tners described dif ferent interna l and external collaborations and drew lines bet ween the partner agencies to indicate where connections had been made. To expa nd t h is init ia l l ist of pa r tnersh ips, OTI further examined Partner monthly reports and specifically solicited additiona l feedback from Partners in Januar y 2013. This data was organized using Excel spreadsheets.

For each relat ionsh ip l isted , OTI labeled t hem a s “Internal ” (within the same grant), “External ” (external to the BTOP funding), or “Cross-grant” (PCC - SBA). For partners which had both sources of funding, unless the specific grant source was specified, their BTOP grant collaborations were labeled “Internal ”. For multi-agency partnerships, each permutation of the partnership had

to be l i sted . For e x a mple , i f t here wa s a pa r t nersh ip bet ween Organization A, B, and C , the following list of partnerships was generated:

A-BA-CB-AB-CC-AC-B

ANALYSIS

O T I u s e d E x c e l t o c o l l e c t a n d m a n a g e t h e l i s t o f pa r t nersh ips . OT I t hen u sed Geph i , a n open-sou rce visualization soft ware, to create the Partnership map.

To i n d i c a t e t h e r e l a t i v e i m p o r t a n c e o f t h e col laborations, OTI weighted the col laborations w ith the following scores:

Internal collaborations with a score of “1” External collaborations with a score of “2”Cross-grant collaborations with a score of “3”

Cross-grant collaborations were weighted most heavily for t h is a na lysis , becau se when v iewed i n t he contex t of the BTOP grant proposa ls commitment to leverage t he SBA a nd PCC work to each ot hers' benef it , such col laborat ion s may have requ i red more coord i nat ion and/or represent a richer form of partnership over other t ypes of collaboration. OT I t hen i nput ted t h i s d at a sheet i nto Geph i w h ich created the net work map.

LIMITATIONS

There are several limitations to this preliminar y analysis. First , the l ist of partnerships is constantly changing as FR P is a dy na mic a nd ongoing pa r tnership. This map only captu res the Pa r tnership at one moment in t ime. S econd , Ma na g i n g Pa r t ners prov ided OT I w it h l i st s of col laborat ions, but OT I d id not s ystemat ica l ly get partnerships from host sites themselves. This map best represent s a macro-v ision of t he Pa r t nersh ip t h rou g h a M a n a g i n g Pa r t n e r l e n s . To b e t t e r r e p r e s e n t t h e

83

Partnership, a series of visualizations can be done with host site level partnerships. A nother l imitation to this pre l i m i n a r y m a p i s t h a t t he t y p e s of col l a b or a t ion s were never clea rly def i ned . For exa mple, host sites of K EYSPOT tra inings a nd labs a re by their ver y natu re collaborations with K EYSPOT Managing Partners, but they are not included on this map. Several different types of visualizations may be most effective for demonstrating different aspects of the Partnership. This map is just a first step in testing a new methodolog y.

Par tner 1 Grant Partner 2 GrantType of

CollaborationScore

Org. A PCC Org. B PCC Internal 1Org. A PCC Org. C SBA Cross-grant 3

Org. A PCC Org. Z External External 2

Sample partnership visualization spreadsheet

84

Evaluation Question Finding Statement Methodologies Used1A: Who participates at K EYSPOTs?

A . K EYSPOTs ser ve primarily African Americans, and age and gender are well represented across K EYSPOT users.

W USSES

B. K EYSPOTs draw from existing participant constituencies in their communities , and use their digita l l iteracy programming to attract new target populations.

Webguide Focus GroupsManaging Partner Inter viewsPrime inter viewsAllied Organization Inter views

C . Personal recommendation and reputation play a signif icant role in attracting target populations to K EYSPOTs.

SESParticipant Focus GroupsManaging Partner Inter views

D. K EYSPOT participants primarily come from the North, Southwest , and West Philadelphia neighborhoods that the Partnership initia l ly targeted.

W USGIS Analysis

E . Most K EYSPOT participants do not subscribe to broadband at home, primarily due to cost barriers.

W USSESParticipant Focus GroupsManaging Partner Inter views

F. For those able to afford home broadband, home subscriptions do not necessarily equate with use.

Participant Focus Groups

G. In addition to cost issues, participants just starting out at K EYSPOTs are fearful of technolog y.

Participant Focus Groups

H. Participants a lso start out at K EYSPOTs with significant (non-digital) learning challenges.

Managing Partner Inter viewsPrime Inter viewsAllied Organizations Inter views

Appendix I. Summary Table of Findings and Methods Used.

85

I . Despite fears and learning challenges related to technolog y, K EYSPOT participants feel the Internet and digital l iteracy are highly relevant to their daily l ives.

Participant Focus Groups

1B: What participant needs are met through K EYSPOTs?

A . While participants use K EYSPOTs primarily for training and workforce development, they a lso had many varied reasons for frequenting K EYSPOTs.

W USParticipant Focus Groups Webguide Focus Groups Managing Partner Inter views Host Sites Inter views

B. Apart from digital l iteracy and computer and Internet access , participants access a wide range of programs while visiting K EYSPOTS.

SESParticipant Focus GroupsHost Site Inter views Managing Partner Inter views Prime Inter views Allied Organization Inter views

C . K EYSPOT participants are ver y satisf ied with how programs meet their needs.

SES W US Participant Focus Groups

2: What are the effects of K EYSPOT access and training on broadband adoption?

A . K EYSPOTS provide an essentia l access point to the Internet.

W US SES

B. When learning new computer and Internet skil ls , participants rely on supportive K EYSPOT Webguides who use creative teaching strategies and al leviate participant anxieties.

Participant Focus Groups Webguide Focus Groups

C . K EYSPOTS increase the digita l l iteracy skil ls of participants and stimulate interest in continued learning.

Participant Focus Groups Managing Partner Inter views

D. Though Primes’ Quarterly Reports to the NTIA indicate the FR P as not yet having met its SBA broadband goals , quantitative and qualitative data suggest K EYSPOT participants credit trainings with inf luencing them to buy home subscriptions.

SESPrimes' reports to the NTIA Participant Focus Groups

86

E . For those participants who own home computers and have Internet subscriptions prior to attending a K EYSPOT, trainings help them use these items.

Participant Focus Groups

3A: What are the effects of K EYSPOT access and training on participants’ employment status?

A . Participants use K EYSPOTs for job seeking and job preparation.

W US Primes' reports to the NTIA Participant Focus GroupsWebguide Focus Groups Manager Partner Inter views

B. Within a short time span, K EYSPOTs can help participants f ind jobs.

SES Participant Focus Groups Host Site inter views Managing Partner Inter views

C . K EYSPOTs help participants keep their job skil ls current.

Participant Focus Groups

D. By virtue of hiring a corps of Webguides, the Partnership is itself an engine of job creation.

Primes’ reports to the NTIA/AR R A reports Webguide Focus Groups Managing Partner Inter views

3B: What are the effects of K EYSPOT access and training on participants’ educational attainment?

A . K EYSPOTs provide educational trainings and opportunities for participants.

SES Primes' reports to the NTIA Webguide Focus Groups Managing Partner Inter views

B. Few participants shared stories of using K EYSPOTs to help them advance their own education.

Participant Focus Groups

4: What are the effects of K EYSPOT access and training on communit y engagement?

A . Though participants do not state that they seek communit y through digital l iteracy and public Internet access , they f ind communit y, both onsite and online, at K EYSPOTs.

W US Participant Focus GroupsWebguide Focus GroupsManaging Partner Inter views

87

B. K EYSPOTs function as a home away from home that of ten provides a safe space for participants.

Participant Focus Groups Managing Partner Inter views

C . K EYSPOTs foster an environment of peer learning , giving participants the opportunit y to connect with one another.

Participant Focus Groups Webguide Focus Groups

5: In what ways has the FR P increased partners’ capacit y?

A . The Partnership increases the staff ing capacit y of Managing Partners and host sites through planned professional development activities.

Webguide Focus Groups Host Site Inter views Managing Partner Inter views Prime Inter views

B. K EYSPOT trainings increase the digita l skil ls of organizations internal and external to the Partnership.

Managing Partner Inter views Allied Organization Inter views

C . BTOP funding al lowed partners to upgrade their IT, freeing up time and resources to better ser ve target populations.

Webguide Focus Groups Host Site Inter views Managing Partner Inter views

D. Collaboration in the form of participant referrals , resource sharing , and net working increases partners’ overal l capacit y to improve ser vice deliver y.

Partnership Visualization Webguide Focus Groups Managing Partner Inter views

E . Partners used K EYSPOTS to further their own organizational goals , specif ical ly for employment, education, and communit y engagement.

Managing Partner Inter views Allied Organization Inter views Host Site Inter views

6: How well has the FR P functioned?

A . While a majorit y of Managing Partners had a positive assessment of the Partnership, many also felt that given more time, the Partnership would have had improved functioning and a greater impact.

Managing Partner Inter views Primes Inter views

B. The Partnership faced operational challenges and lacked unified direction and goals during the implementation stage.

Managing Partner Inter views

88

C . Some Managing Partners felt challenged by hierarchical decision making tied to one programmatic side of the FR P, while praising for supportive management on the other.

Managing Partner Inter views

D. Overall , host sites are satisf ied with Managing Partners’ oversight.

Host Site Inter views

E . At a l l staff levels of the FR P, people strongly support the continuation of digita l l iteracy training and public computer centers.

Webguide Focus Groups Host Site Inter views Managing Partner Inter views Allied Organization Inter views

F. Partners face formidable challenges in planning and securing the resources to sustain these programs.

Managing Partner Inter views

89

90