topic NiNE: Managing out in practice

411t o p i c N i N E : m a n a g i n g o u t i n p r a c t i c e

topic NiNE: Managing out in practice

overviewIn this topic, we pause to review the material and issues covered in the Managing Out unit so far. We explore the knowledge, skills and capacities that public sector middle managers need to meet the challenges of network governance. We examine new paradigms of management, including entrepreneurial leadership, communication, innovation, conflict resolution, negotiation, collaboration and strategic planning. We review the concept of ‘managing for results’ in the light of performance management tools and techniques such as performance information and performance indicators. We conclude by examining the importance of evaluation in public sector management.

Learning objectiveson successful completion of this topic you will be able to:

1. Summarise your understanding of the challenges involved in managing out.

2. review the skills, knowledge and capacities needed to manage out effectively.

3. understand different options for obtaining information on public sector performance and outcomes, including the strengths and limits of different models for identifying and applying performance indicators.

4. understand different models of evaluation and their application in the public sector context.

9.1 Managing out: a Brief Review

This second unit of the PSM Program addresses concepts and strategies involved in ‘managing out’, that is, managing relationships and interactions with individuals and organisations that may be external to the organisation, but are integral to the work of public sector managers. As earlier topics have shown, in a networked state, public sector managers have relationships with individuals or organisations outside of their own immediate sphere of authority that cannot be managed in traditional ways, as might, for example, be appropriate to managing staff. These relationships could include ones with clients, contractors, partner agencies, interest groups and members of the public, perhaps through the media.

In the first part of the Managing Out unit, we reviewed ways in which managers could become more effective in negotiating the environment of change within

412 p S M U N i t 2 : m a n a g i n g o u t : t h e p u b l i c S e c t o r i n t h e c o m m u n i t y

the public sector. This involves managing new relationships within and outside the agency. Effective public sector managers negotiate competing imperatives; they work collaboratively and creatively in an environment of contested ideas and interests. While seeking positive outcomes from their own contribution, they are also conscious of the scope and limits of what governments can achieve when faced with inherently difficult problems. They understand that one of their tasks is to manage the process of making hard decisions about the distribution of scarce resources, the determination of eligibility and the arbitration between competing claims. Little wonder that in this context expert commentators such as Edwards (2002b), emphasise the importance of interpersonal and relationship management skills.

As we discussed in Topics Four and Five, two important themes are driving institutional change and adaptation in the Australian public sector: participation and accountability and whole-of-government approaches. In each case, we tracked the origins of these reform ideas and their impact within particular agencies and regions. We found that though similar goals underpin these reform initiatives, there is substantial variation at the point of implementation and program delivery. We concluded that, since these aspirations for making the public sector more effective are likely to have an impact on agencies in all jurisdictions, public sector middle managers need to equip themselves to work in new and different ways. Knowledge management is an important aspect of this. Public sector managers need to be aware of developments relevant to their organisation and policy sector. Keeping in touch with colleagues in other branches, departments and agencies, as well as with outside organisations from which they can draw contacts and information, are two strategies for doing this. Being aware of guides and resources in an expanding knowledge environment, especially those that can be accessed online is another, as we saw in Topics Six and Seven. Added to this we moved to discuss the character of a variety of relationships that public sector managers are likely to be negotiating, with agents from bodies outside the public sector. We turned first to initiatives in community building, currently a major theme of whole-of-government initiatives. We explored ways of making sense of the different available models. This included clarifying normative and value-oriented conceptions of community, because, despite the ubiquity of the term ‘community’, what is actually meant by this is conceptually unclear. We noted that specialist vocabularies are being used to identify the social impact of economic and political change. ‘Social capital’ was offered as one model; the policy concepts of ‘social exclusion’ or ‘social inclusion’, ‘place management’ and ‘capacity building’ offer alternatives. Each of these is being used in different ways across national contexts, to address the complex problems of local areas. Each of these conceptual tools has been used as a means to adjust and coordinate the linking mechanisms between government agencies, so as to develop ‘joined-up solutions’.

Using these conceptual tools presents both an intellectual and a practical challenge, since the new social indicators have to be consistent and reliable if they are to form the basis of intra-agency and inter-agency processes of consultation, data-gathering, knowledge sharing, service provision and budgeting.

We identified some key skills:

• addressingproblems,issuesandpotentialsolutionsfromthepointofviewofcommunities, rather than from that of government agencies

• goodinterpersonalskills,includingeffectivecommunicationandanopennessto the views of others


• establishingnetworksacrossdepartmentaloragencyboundaries

• workingwithpeoplewithdifferentprofessionalandskillbackgrounds

• identifyingformalandinformalleaders,encouragingarangeofpeopletobecome involved and accepting that levels of involvement are likely to vary considerably

• developingspecificratherthan‘onesizefitsall’solutions

• movingbeyondprecedentandprocedure,workingacrossdepartments

• clarifyingexpectationsofallparties

• lookingbroadlyatoutcomesforlocalcommunities,andincludingtheviewsof service providers and community members.

Topic Six focused on the practical challenges of relationships between public sector departments and between public and private agencies linked by contractual and outsourcing arrangements. We summarised options for managing such relationships and the risk management tactics available to middle managers. This entailed a review of new models of service delivery, including purchaser–provider arrangements between government agencies, between tiers of government, between government and the private sector and between government and the not-for-profit sector. Again, these initiatives involve developing relationships with personnel and institutions that have different roles, responsibilities, values and expertise from those with which most public sector managers are familiar. Mostly, these individuals and bodies stand outside of conventional hierarchical relationships and rely instead on contractual arrangements (explicit and implicit) and on a combination of goodwill and information sharing.

Reviewing the strengths and weakness of initiatives based on such relationships, we explored some of the skills public sector managers need in order to negotiate this territory. They include deciding what to outsource, using indicative principles. Making such decisions depends on an understanding of the issues associated with accountability, privacy, security, consumer protection, and access and equity, as well as with the process of specifying outcomes and measuring performance. Risk management is a key skill, as is the capacity to assess costs and benefits. Reviewing the variety of available means of managing outsourcing, we discussed the difficulty of coming to a clear understanding of competing views about the advantages of outsourcing (in reducing cost and increasing flexibility, choice and variety) and its disadvantages (long-term cost estimates and impact on efficiency, quality and accountability).

Public sector managers also need to be equipped with the means to estimate the short-term and long-term costs and benefits of decisions about contracting out and outsourcing, bearing in mind that the debate on the social and economic impact of such strategies remains unresolved. There are tools though that can help managers to find their way through these thickets. These include safeguards drawn from administrative law, accountability requirements and managers’ own developing expertise in contract management. We explored ways to locate information on how to make use of such safeguards, as well as guides on risk management, complaints procedures and appeals.

As we turned, in Topic Eight, to the third area of managing out relationships, managing the media, we found that some of these skills were generic and


transferable. Here again it was important to understand the broad context of the changing environment: in this case, the political role of the media and of communications. It is important for public sector managers to have a strong sense of role and location, understanding their institutional place within the complex environment of publicity, news-seeing, public relations and political commentary. Comprehending the scope and limits of their roles and responsibilities can assist public servants in maintaining a distinction between providing information and responding to the media and the roles of the ministerial office. Within an information-hungry and speedy culture of news and discussion, public sector managers need to be prepared, risk conscious and forward thinking. They must provide appropriate information in response to the imperatives of accountability to the public, while resisting the temptation to put their own view or divulge sensitive information.

We explored some options for building the capacity of public sector managers in this regard. Public sector employees need to be aware of their agency’s procedures for handling unsolicited media inquiries, while taking a role, as required, in promotional activities. While some independent judgment is required, important resources to consult in this situation include guidelines on the appropriate chain of referral for media inquiries and references to media dealings in Codes of Conduct and in the provisions of Commonwealth legislation. Public sector managers also need to be familiar with the distinction between adhering to departmental procedures and to the law, rather than acting according to their personal concept of the public interest. Where such decisions are called for, public managers need to be conscious that the public interest is a contestable idea, defined by conflicting priorities. They need to be able to recognise when a decision involves the exercise of ethical judgment. When in doubt about their legal and institutional obligations, they need to be able to find information resources, advice and guidelines to assist decision-making. Preparation can help: both conceptual reflection and practical initiatives such as finding out how their work organisation has handled complex issues associated with the media.

9.2 Entrepreneurial Management in the public Sector

In a dynamic operating environment, a focus on creating public value would require public sector managers to innovate to ensure programs and services remain efficient, effective and responsive (Moore 1995). Moore (2005:43) argues that such innovations have the potential to increase public value in at least three different ways:

• bygeneratingbettermethodsforperformingcorefunctions,resultinginageneral productivity increase

• byexploitingtheperformanceadvantagesthatcouldbegainedbyencouraging adaptation and customisation of basic operational procedures to meet the demands of varied circumstances and clients

• bystrategicinnovationsthatrepositiontheorganisationandbuildcapabilityto achieve its mission.


9.2.1 Characteristics of the Entrepreneurial Manager

Innovation of the kind envisaged by Moore is driven by entrepreneurs – individuals that are open to and prepared to work for reform and change. Although the concept of the entrepreneur has been around for much longer, it was in the 1930s and 1940s that the economist Joseph Schumpeter linked entrepreneurship with innovation, hence contributing to today’s understanding of the term (for an elaboration of the emergence of the term ‘entrepreneur’, see Boyett 1996). This notion of the entrepreneur can be contrasted with the less complimentary 1980s Wall Street persona of the ‘curious mixture of greed, short-sightedness and megalomania’ (Godfrey 1996:49).

Although many have tried, a clear psychological profile of ‘the entrepreneur’ has not been established. What studies do agree on is that entrepreneurs reflect the characteristics of the time and the place in which they live. Hence, as he signals in the title of his article, Sadler (2000:2) finds that ‘entrepreneurship can thus be viewed as chameleon-like: a regional, temporal and strategic phenomenon that alters according to its operating environment’.

Sadler’s article comprehensively reviews the research and concludes, as did Drucker back in 1985, that entrepreneurship was as much a public sector phenomenon as a private sector phenomenon, but with many impediments peculiar to the environment. The most stifling of these impediments might be the multiplicity and ambiguity of goals, the lack of control over resources and risk aversion associated with the political consequences of ‘failure’ (Moore 2005). Some of the accountability and scrutiny considerations we examined in PSM Program Unit 1 are likely to also be significant. Moran (2004) takes issue with these perceptions, which he argues misunderstand the true nature of innovation in the public sector. Although he concedes that the scale and complexity of bureaucratic organisations constrains innovation, Moran (2004:4) argues ‘the public sector has the potential to be a hotbed of innovative activity’.

Godfrey (1996) asked what distinguished an entrepreneurial public servant from a traditional public servant, and identified three key characteristics:

1. They see themselves primarily as managers focused on outcomes (contrasted with administrators focused on process) using commercial models, frameworks and tests of effectiveness, rather than primarily public administrators or policy advisors.

2. Within the law, they focus on external customers but subordinate strict adherence to regulations to meeting their needs.

3. In practice, they place more emphasis on people as a key resource of the organisation’s strength, seeking to empower the ‘front line’ (contrasted with a command and control approach) and emphasising organisational learning as a key to sustainability.

Godfrey argues that an entrepreneurial public servant might distrust excessive analysis, value intuitive processes and tolerate errors as a form of learning. The role of the entrepreneurial manager is not to replace the conventional administrator, who is necessary to ensure legality, equity and consistency, but to provide an enriching counterpoint to decision-making, an alternative view and a conduit to change in the external environment (Godfrey 1996:53). Godfrey’s conclusion is that


all organisations need some entrepreneurial people near the top. However, according to Timmins (in Graham & Harker 1996:55), we should be wary of some of the stereotypes noted in Table 9.1 below:

table 9.1 Entrepreneurs and administrators

In the public sector context, the notion of entrepreneurship is often related tothepolicyprocess.Mackenzie(2004)explainshowtheliteratureonpolicyentrepreneurship has developed over time. It has been understood variously as:

• aformofpoliticalskill(see,forexample,Bardach1972)

• agentsofchangeimportanttotheagenda-settingprocessunderpinningthepolicy cycle (see, for example Kingdon 1995 or Haass 1999)

• intermsoftheinfluenceofkeyindividualsonpolicyinnovation,policylearning and policy diffusion (see, for example, Matlay & Mitra 2002).

It is important to be aware that policy entrepreneurs can be inside or outside of government. Diane Stone (1996), for example, provides a comprehensive account of how think tanks engage in policy entrepreneurship to connect with and influence the policy process. Within government, policy entrepreneurs have significant scope to pursue a preferred outcome by assisting or hindering problems or issues through different parts of the policy process.

Schumpeter’s exploratory work saw the innovator and the entrepreneur as one and the same. That is, the role of the entrepreneur was to innovate; the effect of innovation was economic activity that changed the production function. This change could include a new product, a new way of doing something, a new organisation and a new source of raw materials (McDaniels 2000).

McDaniels comments that some of the literature on innovation recognises four stages ‘from the laboratory to the market’. While this has a private sector flavour to it, there is no reason why the model cannot be adapted to public sector thinking. These stages are:

1. Research: characterised as basic or applied, this is the stage that identifies new activities or products and any potential practical use.

2. Development: this stage explores the potential in a laboratory setting to the

Entrepreneur public sector manager

ambiguity and uncertainty v. planning and rigour

creativity v. disciplined analysis

urgency v. patience and perseverance

flexibility v. organisation and management

innovation and responsiveness v. systemisation

risk management v. risk avoidance

current profits v. long-term equity


stage of producing a prototype.

3. Demonstration: sometimes called ‘the real world application’, this stage aims to prove operational success.

4. Commercialisation: this is the stage where the product or process is duplicated for sale in the marketplace.

9.2.2 Developing Innovation

Drucker argues that modern government needs innovations but asks what, in practical terms, it means to innovate. The question has been taken up by others, in a more applied mode. One instance is The Challenge of Innovating Government, a report commissioned by the Canadian government and prepared by Professor Sandford Borins (2001). The objective of this report was to find ways to change the traditional bias against innovation in the public sector. The research found that the five characteristics of successful innovation are:

1 The use of a systems approach.

2. The use of new information technology.

3. Process improvement.

4. The involvement of the private or voluntary sectors.

5. Empowermentofcommunities,citizensorstaff.

Related Canadian research suggests that that innovation ‘champions’ can be found at all organisational levels. Kernaghan, Marson and Borins (2000:269) found that ‘the most frequent initiators of public management innovations are ‘local heroes’, middle and front line public servants who have vision and are willing to take risks’. The authors reported that innovation is most likely to flourish in organisations where:

• seniorexecutivesarepredisposedtoentrepreneurship

• thepeopleinvolvedareentrepreneurial–results-oriented,notprocess-oriented

• goodpeopleareassignedtokeytasks,wheretheyaregivenresponsibility,accountability and are encouraged to take ownership

• thereisawillingnesstotakerisks,befirstandestablishprecedents

• workrelationshipsarecharacterisedbycollaborationandloyaltytoteamandproject

• thereisflexibilityandwillingnesstochangedirectionwhennecessary(adaptedfromKernaghan,Marson&Borins2000:270).

9.2.3 Impediments to Innovation

One of Borins’s less optimistic findings was that most barriers to innovation were internal to the bureaucracy. These barriers included hostile attitudes, turf fights, logistical problems, coordination difficulties between organisations, lack of enthusiasm, technology implementation problems, union opposition and public


sector opposition to entrepreneurial action. This list of barriers underscores the importance of developing organisational cultures that support and enable innovation. Promotion of such a culture can be difficult in an environment that is highly sensitive to criticism and in many ways intolerant of risk and failure (see Moore 2005;Moran2004).HoweverasKernaghan,MarsonandBorins(2000:207)argue,continuous improvement and innovation are closely associated with the ability to learn. We examine the concept of organisational learning and strategies for building learning cultures in detail in PSM Program Unit 4.

Required Reading 9.1albury, D 2005, ‘Fostering innovation in public services’, Public Money and Management, January,

pp. 51–56.

read the article by albury, which explores public sector innovation and the conditions that enable it. then answer the following questions:

1. What, according to albury, is driving the need for innovation in the public sector?

2. What are the two different types of innovations described by albury? What are their different characteristics?

3. think of an example of each of these types of innovation from your own organisation or jurisdiction.

4. What are the some of the conditions that would foster public sector innovation?

5. What would need to change to foster innovation in your organisation?

6. What practical steps could you take to reduce barriers to innovation and foster an environment that supports it in your own work unit or team?

9.3 Skills for Managing out

Building relationships of trust with networks of actors within and outside government requires entrepreneurial skills, and in particular an ability and willingness to work across boundaries and to experiment and innovate to find solutions to public policy problems (Bakvis & Juillet 2004:19). As this implies, and as we have emphasised throughout the PSM Program, new ways of working in the public sector require new skills and capabilities, in particular, leadership at all levels of the organisation. We explore leadership skills and behaviours at length in both PSM Program Unit 3 Managing In and PSM Program Unit 4 Managing Down. In the managing out context other important skills include conflict resolution, negotiation and bargaining, working collaboratively and strategic planning.

These are discussed in turn.


9.3.1 Conflict Resolution

Collaborative relationships depend on effective communication, in all its many forms (speaking, listening, writing and non-verbal signals) and trust. But the ability to manage and resolve conflict is also crucial.

One definition of conflict is ‘verbally or non-verbally expressed disagreement betweenindividualsandgroups’(Cole1998:517).Asecondisthatofa‘processwhich begins when one party perceives that another party has negatively affected, or is about to negatively affect, something that the first party cares about’ (Thomas 1992 in Robbins et al. 2001:489). Given the sorts of conflict that you or your staff might deal with in the context of managing out, such as with clients or contractors, which of these definitions do you think is the most useful or applicable? Each conflict is different but in general the stages of conflict may include discomfort, an incident, misunderstanding, tension and, if not resolved early, a crisis.

9.3.1.1 Conflict resolution styles

It may be that individuals routinely adopt a particular style of dealing with conflict without recognising that there are other more effective alternatives. Look at the matrix below (Figure 9.1). Have you seen this before? It is presented quite frequently in management training. You may have seen it with different labels on the axes, or in the shaded boxes which represent conflict resolution techniques. Not all sources present the middle shaded columns. However, these have been included here because these ‘middle of the road’ strategies for handling disagreement are quite common among professional groups outside the private sector (Marchant 2002).

Let’s start with the vertical axis, labelled ‘Importance of the outcome or issue’. In this case, the outcome is the issue at hand. That is, how important is it to you that the desired outcome of the client, contractor, media or other stakeholders be achieved? The range of options on this axis is from being very assertive about the outcome or issue, to being unassertive. Here you will need to consider such aspects as risk (political or otherwise), policy, law and procedure.

Thehorizontalaxisislabelled‘Importanceofrelationship’.Herethequestionistherelative importance of the relationship you have with the person (or people) you are having the conflict with. The range of options here is from being uncooperative, uncaring or disrespectful, for example, to being very cooperative, caring and respectful. This axis is sometimes also labelled ‘cooperativeness’, to signal how much one party is prepared to go along with the other (Robbins et al. 2001).

These two axes or dimensions form a matrix. The matrix has been filled in with seven different strategies or techniques for conflict resolution. Few people actually adopt the deliberate, systematic approach that this matrix implies. However, the matrix provides a useful framework for analysing the way people deal with conflict, and for identifying appropriate approaches in different situations.


Figure 9.1 conflict resolution styles

Sources: adapted from Kindler 1988; Robbins et al. 2001; Marchant 2002.

This matrix can be used to analyse any conflict situation. Locate one party to the conflict along the vertical axis (for example, yourself) and the other party on the horizontalaxis(forexample,acontractororclient).Adescriptionoftheelementsfollows:

1. Forcing, dominating or competing:

• Unilaterallyinduce,persuadeorordercompliance.

• Maybeusefulwhenquickdecisionsareneeded.

• Mayhavetobeusedasalastresortwhenvitalbutunpopularactionisneeded and the client or contractor refuses to cooperate.

• Isaggressiveandcanresultinantagonismandresentmentfromtheotherparty.

• Basedonpower(e.g.pullingrank)toimposeasolutionontheotherparty.

2. Bargaining or negotiating:

• Togetheryouseektosplitdifferences,settrade-offsortaketurns.

• Youexchangesomethingyouwantforsomethingtheotherpartywants.

• Thereismorechancethattheotherparty’sreactionswillbepositiveandthe desired outcome will be achieved, even if in modified form.

3. Collaborating or integrating:

• Establishsuperordinategoals;focusonwhatissharedincommon;fostera climate of collaboration by aiming at the new, higher goals which incorporate what both parties want.

• Appropriatewhentheissueissignificant,theotherparty’sconcernsareimportant and all parties are totally committed to the outcome.

• Inarelationship-focusedsituation,thisisapreferredstrategy.

• Thisstrategyseekstogiveadvantagetoallparties.

• Itisnoteasy;takestime,practice,skillandeffort;andrequiresgoodcommunication, problem-solving, empathy and assertiveness skills.

4. Compromising or co-existing:

• Intermediateincooperation,splittingthedifferencetoachievemiddleground.

importance

of the

outcome

or issue

High

Low

1 Forcing, dominating or competing

5 avoiding or withdrawing

importance of relationship

Uncooperative

Low

cooperative

High

6 using decision rules

2 bargaining or negotiating

3 collaborating or integrating

4 compromising or co-existing

7 accommodating or giving in

Assertive

Unassertive


• Arrivesatasolutionthatispartiallyacceptabletobothparties,butitmaybe that neither is fully satisfied.

• Usedwhentheissueisimportant,butnotworththetensioncausedbyinsisting on achieving the desired outcome.

• Mayworkasaback-upwhenotherstylessuchascollaborationorforcing have failed.

• Maybeusedasatemporaryshort-termsolutionwhileotherdiscussionscontinue.

• Maybethebestyoucanachieve.

5. Avoiding or withdrawing:

• Inthisstrategyyoujustdroptheissue,orputitofftilllater.

• Appropriateifneithertheissuenortheotherparty’sconcernsareimportant.

• Cantaketheformofdiplomaticside-steppingofasensitiveissue.

• Allowspeopletocooldownorgivesyouachancetobringinpeoplemore able to handle the conflict.

• Usefulwhentheissueistrivial,orotherissuesaremoreimportant.

• Usefulwhenyouseenochanceofachievingtherequiredoutcome.

• Conflictabouttheissuemaybesweptunderthecarpetandavoidedinthis strategy, but still ‘simmers’.

6. Using decision rules:

• Youandtheotherpartysetrulesabouthowtheconflictwillbehandled,or how differences in views will be dealt with.

• Examplesofthesedecisionrulesareashowofhands,majorityagreement, total consensus or even one party decides, with agreement.

• Useobjectivecriteriaorrules:‘Whatisafairwaytoevaluatethemeritsof our arguments?’.

• Thisstrategytendstobeover-used,tokeepthepeace.

7. Accommodatingorgivingin:

• Inthisstrategyyougoalongwithwhattheotherpartywants;itisacooperative strategy and is the opposite of forcing or dominating.

• Thisworkswhentheotherparty’sconcernsaremoreimportanttothemthan the issue is to you.

• Youmayusethisstrategytogaincreditsforlaterachievingotherrequiredoutcomes, or when harmony and stability are especially important.

• Thisstrategymayallowtheotherpartytodevelopbylearningfrommistakes, that is, let the other party go ahead with what they want and learn from the consequences.

• Thedangeristhattheotherpartymaynottakeyouseriouslyasyoudon’ttake a stand even when you may be right, for fear of losing their friendship or cooperation; therefore this strategy must be used in the right situation.

Sources: adapted from Kindler 1988; Robbins et al. 2001, Marchant 2002.


Activity 9.1 – conflict-handling quiz

now that we have examined conflict-handling styles in some detail, it may be appropriate to investigate how you normally approach conflict, and find out what your style is. most people tend to behave in a certain way most of the time. try this short quiz to identify your predominant conflict-handling intention. the quiz does not address the intermediate styles in the matrix, but rather focuses on the main, or extreme ones.

1. For each statement, write the appropriate number in the column that reflects your answer, for example if your answer to statement 1 is usually, put a 4 in the column in line with this statement.

2. When you have finished, refer to the scoring key below.

Scoring key for conflict resolution quiz

the scoring key below shows five conflict-handling styles. under each style there are three numbers which correspond to statement numbers in the table above, For example, Statement 1 shows a forcing or competing style if answered in the affirmative.

Statement

i argue my case with others to get across the merits of my view

i negotiate with others to achieve a compromise

i will agree to anything to keep the peace

i like to meet others’ expectations

i try to find mutually acceptable solutions to problems

i am firm in putting forward my side of an issue

i keep any conflict i feel to myself

i keep quiet about my ideas on a problem

i like to give and take to find an answer

i will give accurate information to others to facilitate joint problem-solving

i enjoy a chance to argue my case

i accommodate other people’s wishes

i like to have everybody’s concerns ‘on the table’ for discussion

i come up with suggestions for a middle ground in a deadlock

i find it easier just to go along with what others want

1Never

2Sometimes

3Don’t know

4Usually

5Always

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15


1. in the table below record your response score from the above table next to each statement number. For example, if for Statement 1 you ticked always, then put a 5 in the competing column next to the number 1.

2. Do this for all fifteen statements and then add up the column totals.

the column with the highest score indicates your primary conflict-handling style, and the next highest score is your ‘back-up’ style.

Source: adapted from Robbins et al. 1998:601.

9.3.2 Negotiation or Bargaining

If you revisit the matrix (Figure 9.1), you will see that bargaining or negotiating is a relatively assertive strategy – that is, one to be deployed when you need to gain the outcomes that you desire or require. But it is also less confrontational or destructive of the relationship than simply forcing the other party to do what you want. Given that we are discussing negotiation in the context of managing relationships with external individuals and organisations – such as clients, contractors or the media – this is an appropriate strategy for maintaining the relationship while still ensuring that requirements are met. Such requirements might relate to compliance, or to fulfilling a contract. A negotiating or bargaining strategy can still result in a win–win outcome (although not necessarily the best possible outcome) through a collaboration strategy. Although collaboration is ‘ideal’, it may be unachievable in some circumstances, hence our focus on negotiating or bargaining.

Further, negotiation skills are critical in situations where individuals have to work with others over whom they have no direct authority, even if a situation of conflict does not exist. One formal definition of negotiation is that it is a situation where two or more parties make an exchange and attempt to agree on a rate for the exchange (Robbins et al. 2001). Another definition is that negotiation is ‘any attempt by two or more conflicting parties to resolve their divergent goals by redefining the terms of their interdependence’ (McShane & Von Glinow 2000:418). Again, given the nature of any negotiating you do in the context of managing out, which of these definitions is most useful?

9.3.3 Working Collaboratively

As we have seen in earlier topics of this unit, public sector reform initiatives based on marketisation have emphasised competition which in many ways is an anathema to collaborative ways of working. But network governance requires cooperation

Forcing or competing

1

6

11

total

collaborating

5

10

13

total

Avoiding

3

7

8

total

Accommodating

4

12

15

total

compromising

2

9

14

total


andcollaborationbetweenindividualsandorganisations.Thesehorizontalapproaches demand new and different skills, particularly for public sector managers who some would say are more accustomed to hierarchy and control than to building relationships of trust that will inspire or motivate prospective partners to collaborate towards a shared vision and goals (Bakvis & Juillet 2004).

The literature on collaboration, partnerships and network governance emphasises the importance of trust (see Perri 6 et al. 2002). According to Bakvis and Juillet (2004: 9)trustis‘theall-importantlubricantthatmakeshorizontalarrangementswork’.But how do we build trust? We know from personal experience that trust develops – over time and through experience – it cannot be mandated. Importantly trust is fragile – it can be easily undermined or lost. Perri 6 et al. (2002: 119–120) identify three reasons why trust might develop:

• Onthebasisofpastexperienceofdealingwithapersonororganisation– they have proved themselves reliable in the past.

• Onthebasisofreputation–thepersonororganisationisreputedtobetrustworthy and reliable, and we assume they will behave in ways that will protect and enhance rather than undermine that reputation.

• Onthebasisofsharedcircumstancesoridentity–forexample,nationality,ethnicity, religion, gender or shared beliefs.

There are many obstacles to developing trust, not least of which are the political risks of sharing information and decision-making power. Building relationships also requires the investment of time, and this can be a challenge given the emphasis on demonstrating outcomes and achieving results. There are also important questions over whether collaborative approaches are compatible with the legal, ethical and accountability obligations of public sector agencies. For example, while informality, trust and the quality of personal relationships are crucial to working collaboratively, Lowndes and Skelcher (1998:322) found these relationships could be viewed negatively by other network actors, some of whom felt marginalised and excluded. These players were suspicious of ‘those in the know’, who they were concerned had privileged access to information and resources. Other obstacles derive from stereotypes about people and organisations, who may, for example, be regarded as ‘political’, ‘provocative’ or ‘left-wing’.

Perri 6 et al. (2002:124) are sceptical about obstacles to trust, noting that ‘many of the reasons that make holistic government hard to achieve stem from fear, lack of ambition, risk aversion and the power of incentives to maintain the status quo’. They encourage those interested in promoting collaborative governance to share information, to invest in relationships and to pursue strategies to build legitimacy and a culture that supports collaboration.

9.3.4 Strategic Planning

Strategic thinking is an important skill for managing in the public sector, particularly in the managing out context. Stewart (2004:16) notes that the business literature recognises the fundamental importance of strategy and that it is the essence of success since it seeks to anticipate and deal with challenges from competitors. Strategic thinking involves the proactive identification of issues and challenges


that require organisational change, and the harnessing of people, skills and ideas in pursuit of that change. An understanding of the potential of strategic thinking and planning is part of every middle manager’s role, even if they have minimal opportunity for direct input into corporate strategy processes.

Strategy provides a strong sense of priority and purpose (Smith, Anderson & Teicher 2004:115). It encourages agency leaders and staff to see their organisation in a wider, longer term and helps focus attention on the challenges and opportunities of the external operating environment. For public sector managers, a strategic focus provides:

• astatementofpurposethatwillhelpyoustayfocused

• astatementofpurposethathighlightsparticularpiecesofmanagerialworkthat are considered strategically important

• anarticulationofthebroadpurposesoftheorganisation,whichoftenstimulates and will guide the contributions of others, as well as yourself.

Strategy does not however provide detailed government policy outcomes and agency activities and indicators (Smith, Anderson & Teicher 2004:15). Agencies themselves must flesh out the detail of the various components that will help achieve the strategy. The well-managed public organisation will have a hierarchy of documents recording the objectives, strategies, expected outputs and resources of the agency. At the top of the hierarchy will be the corporate plan. There are likely to be divisional plans or regional office plans and maybe a section plan in addition to your own performance agreement. All of this is expected to make a coherent whole. Many corporate documents are also prepared to provide information, advice or guidance to your external audience and these can be a foundation on which you build these relationships.

9.4 Managing for Results

In the current reform environment, public sector skills and energies are directed towards managing to achieve the outputs and outcomes of the organisation. There is a strong emphasis on ‘results-oriented management’ – ‘the purposeful use of resources and information in efforts to achieve and demonstrate measurable progress towards outcome-related agency and program goals’ (Wholey 2001:344). This is accomplished using a three-step process:

1. Developing a reasonable level of agreement among key stakeholders on missions and outcomes-oriented goals – and strategies to achieve these goals.

2. Measuring performance (in particular, outcomes achieved) on a regular basis.

3. Using performance information in efforts to improve program effectiveness and strengthen accountability to key stakeholders and the public.

The remainder of this section will concentrate on a discussion of performance information and performance indicators.


9.4.1 Performance InformationAulich (in Aulich et al. 2001) argues that almost all the activities of public management are concerned with the management of data and information – how it is collected, stored and applied.

Performance information has become critical to management at all levels and the reform agenda has shifted the focus of data and information collection from inputs and processes to outputs and outcomes. Not surprisingly, the Australian National Audit Office (ANAO) is foremost among producers of information about performance measurement, which has a language of its own:

• Performance is how well a service meets its objectives, recognising the influence of external factors.

• Performance information is evidence about performance that is collected and used systematically.

• Performance indicators provide a guide on performance where causal links are not obvious.

• Effectiveness indicates the extent to which program outcomes are achieving program objectives.

• Efficiency relates to minimising program inputs for a given level of program outputs.

• Evaluation is the systematic, objective assessment of the appropriateness, effectiveness and/or efficiency of a program or part of a program that is undertaken periodically. Major programs in the APS are evaluated at three to five year intervals.

• Quality relates to the characteristics by which a product or delivery is judged by customers and stakeholders.

• Inputs are resources, both human and other, used to produce program outputs.

• Outputs are the product or services which are produced and delivered by a program.

• Outcome refers to all the impacts or consequences of the program beyond its direct outputs.

• Strategies describe the means by which objectives are achieved.

• Targets are quantifiable performance levels to be attained at some specified future date. They are not a performance minimum (ANAO 1996).

According to the ANAO (1996) for a program, organisation or work unit, the key elements of the program or work structure include the resources used (inputs), what is done (processes), what is produced (outputs), and what impacts are achieved (outcomes). Performance information addresses the relationships between these elements. It should provide sufficient information to answer the questions about key aspects of performance such as:

• Howeffectiveistheprograminachievingthedesiredoutcome?

• Howefficientisitinusinginputstoproducerequiredoutputs?

• Whatisthequalityoftheprogram’soutputsandoutcomes–thatis,areclients receiving a satisfactory level of service?

• Istheprogrammeetingaccessandequityrequirements?


Sufficient information will in many instances be a matter of judgment. While it is possible in some instances to apply statistical techniques to information collection (e.g. random samples or a given percentage of a given population), it is just as likely that you will trawl your organisation’s information system for available data. The major disadvantage of this is that you can only get out what has already been put in. The lesson is that suitable data should be identified early in the planning process and collection should be facilitated in a timely and orderly manner. Obstacles to performance assessment can be the failure to identify useful information and therefore the failure to collect it, collection of irrelevant information, or information that is not timely. The cost–benefit of having to go back and collect data then has to be weighed up.

For an overview of how the ANAO conducts performance audits, see ANAO 2008, Performance Auditing in the ANAO accessible at: <http://cms.anao.gov.au/uploads/documents/Performance_Auditing.pdf>(viewedon17September2008).

9.4.2 Performance Indicators

The term ‘performance indicator’ has become part of the management vocabulary of the public sector. Such indicators are used extensively in public documents, such as annual reports, and widely quoted as measures of success (or indeed failure). At the simplest level performance indicators are management tools, and the implementation debate is more often about what indicators are appropriate than about opposition to measurement itself. Nonetheless, performance indicators are useful tools in managing performance in:

• drivingthespecificationofservicestandards

• ensuringthatobjectivesareclearandoperational

• ensuringthatperformanceagainstobjectivescanbequantified.

When well developed and implemented, performance indicators help departments assess their performance in a rigorous and systematic way and provide a guide to future improvement. They are most effective when they are allied with an integrated financial management process which ties targets to resources. For stakeholders and customers, performance indicators can be a useful means of ensuring that performance in key areas of concern to them is measured and reported on.

For organisations and their members to understand what standards they are to achieve, criteria for performance must be specified. Too often the experience with performance measurement is the collection of sometimes large amounts of data but little analysis of what the information might mean, let alone the application of this learning to improve performance. Performance measures should be a critical element of the strategy management loop linking strategy to operations and outcomes.

Successful performance measurement has the following key features:

• Thereisaclearconceptualframeworkforperformancemanagement–theorganisation clearly understands how its strategy will relate to the way it will measure performance.

• Theorganisationusesindicatorswhereithasmostcontroloveritsownperformance.


• Onlyafewkeymeasuresareprovidedatthestrategiclevelofperformancemanagement (ten to twelve is often stated as the preferred number of high level indicators).

• Indicatorsarebrokendownbyappropriateorganisationallevels:goodperformance indicators are a tool in improving good performance at all levels in the agency.

• Indicatorsarelinkedtothefinancialmanagementprocessatalllevelsoftheorganisation.

• Performancemeasurementproducesintelligence,notjustdata.

• Theframeworkandsubsequentdevelopmentoftheperformancesystemiseffectively communicated to staff and stakeholders as part of a performance culture.

• Performancemeasurementispositive,notpunitive–theinformationobtained helps the organisation learn and develop its effectiveness.

• Resultsofperformancemeasurementareopenlysharedwithemployees,clients and stakeholders.

Performance indicators may also be used to compare the performance of departments with each other, and there is a growing body of literature on the value and mechanics of benchmarking in the public sector. Annually, the Productivity Commission prepares the Report on government services. This has the two functions of comparing the performance of government’s services and reporting on reforms, either implemented or under consideration.

There are three distinct types of performance indicator you are most likely to use and these are explained.

table 9.2 types of performance indicator

9.4.2.1 Limitations of performance indicators

In the current climate, which promotes models of performance management designed to facilitate the reform agenda of governments through the introduction of competition, it is easy to lose sight of an overall perspective on what performance

type of performance indicator

Economy

Efficiency

Effectiveness

Example

costs related to resources – is the organisation obtaining its resources at the lowest possible cost consistent with what it wants to achieve?

outputs related to costs – are outputs produced at the lowest cost consistent with what the organisation wants to achieve?

outcomes related to objectives – is the organisation achieving what it set out to achieve? is it providing the most appropriate services to meet its objectives?

What it is about

concerned with inputs

concerned with throughputs

concerned with outputs and objectives


indicators can and cannot do. You need to be aware of the limitations of their use. Some of these are listed below:

• Performanceindicatorsonlyindicate:theyarestatisticalmeasuresthatindicate what is happening in practice; performance indicators do not show the whole picture.

• Notallactivitiesarereadilyamenabletostatisticalmeasurement:someactivities are difficult to quantify.

• Performancemanagement,includingtheuseofperformanceindicators,is relatively short-term in its approach and is no substitute for long-term planning.

• Manycontextualfactorsarenotwithinthecontroloftheorganisationandcannot be measured as being a result of its performance.

• Indicatorsmaynottakeintoaccountthelegacyofpastdecisionsandperformance.

• Therearedifficultiesinweightingindicatorsagainsteachother,soinmostsituations indicators are not weighted. This means that indicators may not help with prioritising.

• Beawareofthepoliticsofperformanceindicators.Aswithanydataset,performance indicators can be manipulated or abused for political purposes. For example, chief executive officers on performance bonuses may want to play up the good news, and the section or unit wanting more resources can manipulate or interpret indicators to suit its own ends.

Doherty and Horne (2002:341) argue that managers need to propose ‘practical and creative measures of performance that are appropriate to their local conditions yet which avoid the worst consequences of misleading indicators’. They summarise some of the problems with performance measurement including:

• Definingtheobjectivesofcomplexserviceswheremultipleobjectivesconflict.

• Lackofcorrelationbetweenorganisationalobjectivesandspecificobjectives.

• Inexperienceofstaffindevelopingandusingperformanceindicators.

• Lackofrelevantandmeasurabletargetsforfinaloutputsandoutcomes.

• Lackofresourcestobuildupdata(e.g.clientsurveys).

• Resistancetodatarecordingbystaff.

• Costofperformancemeasurement.

• Lackofinterest.


Want to know more?For a critique of performance indicators in the social housing context, see burke and hayward 2001, Performance Indicators and Social Housing in Australia, available from the link for this topic on the other learning resources section of the pSm program national website.

See also:

australian national audit office 2004, Performance Management in the Australian Public Service, accessible at <http://www.anao.gov.au/uploads/documents/2004-05_audit_report_6.pdf> (viewed on 7 September 2008).

heath g & radcliffe J 2007, ‘performance measurement and the english ambulance Service’ Public Money & Management Vol. 27 no. 3 pp. 223–228.

9.5 Evaluation

9.5.1 Introduction to Evaluation

Evaluation is a research process, involving the gathering of information to find out whether a policy is working as planned, whether it is working well, whether it is achieving what it was supposed to achieve and how it may work better. Evaluation helps to address questions about whether the program is achieving expected results, or even whether the problems were correctly identified when the program was developed.

Evaluation also helps to assess how well a program may have been implemented – the program design may have been appropriate, but changes or delays in implementation may have produced unexpected or inadequate results. Althaus, BridgmanandDavis(2007:179)suggestthatevaluationistheendofthepolicycycle, and also the beginning – as information from evaluation triggers a new understanding of the original problem and further problem identification, analysis and decisions.

Evaluation is also about accountability, about the effective use of resources for public outcomes. But evaluation potentially goes a lot further than measuring activity or progress – it can provide the basis for fundamental reassessments of policy and potential changes of direction.

Evaluations may be planned and conducted as part of formal review cycles for programs, or as a review stage for a pilot program. Evaluations may also occur on an ad hoc basis, triggered or motivated by concerns about the program, major changes in the operating environment, or perhaps a desire to close a program down.


9.5.2 Types of Evaluation

Evaluations can have different aims and purposes. These may occur at different stages of the life of a policy or program and have a focus on different aspects of a program. It is important to be clear about what an evaluation process is aiming to achieve – and design the research activities appropriate to this aim.

There are thus several different types of evaluation, including:

• Formative evaluation, which involves analysing of a program during the implementation process, providing feedback that may improve implementation.

• Summative evaluation, which focuses on the impact of a program on the problems the program was meant to address.

• Process evaluation, which focuses on the effectiveness of organisational methods for delivering a program.

• Outcome evaluation, which focuses on the benefits of the program in meeting broad community outcomes.

9.5.3 Getting Evaluations Done

Evaluations are essentially policy research projects, and require appropriate expertise and adequate time and resources to be undertaken effectively. Evaluation activities may be undertaken ‘in-house’, but more typically they will be outsourced, to provide an independent perspective on the program. In recognition of this, the American Evaluation Association (Newman et al. n.d.) has published guiding principles, listed below, which are an equally useful reminder for in-house evaluators.

1. Systematic inquiry – evaluators conduct systematic, data-based inquiries about what is being evaluated.

2. Competence – evaluators provide competent performance to stakeholders.

3. Integrity/honesty – evaluators ensure the honesty and integrity of the entire evaluation process.

4. Respect for people – evaluators respect the security, dignity and self-worth of the respondents, program participants, clients and other stakeholders with whom they interact.

5. Responsibilities for general and public welfare – evaluators articulate and take into account the diversity of interests and values that may be related to the general and public welfare.

These five principles are elaborated in the report which is located on the PSM Program National website.

An evaluation may have a greater success in generating useful information if it has been planned at the outset of program implementation. Thus for a new program an evaluation plan should be developed in conjunction with the implementation plan. At the very least this enables important program information, essential for a later


evaluation to be captured in the establishment stage, e.g. data on gender, ethnicity, age and service usage for a health service program.

Some core starting questions in developing an evaluation plan are:

• Whatisgoingtobeevaluated?(scope).

• Whatdoyou(andothers)wanttoknowaboutthepolicyorprogrambeingevaluated? (terms of reference).

• Whatwillyoudowiththeinformation?(purpose).

Scoping the evaluation involves the consideration of fundamental questions about the context of the evaluation. These questions may be straightforward in the case of a cyclical program evaluation or the planned evaluation of a pilot program. But scoping may raise complex issues associated with identifying the program’s intent; will the evaluation focus on explicit program objectives, or will it also explore implicit program goals? There may be inconsistencies in stated objectives and program intent. In the case of an ‘old’ program, changes over time might mean the original stated objectives are less important than they once were. In the Required Reading 9.2 O’Faircheallaigh (2002:2, 3) raises the question of whose goals are being evaluated? Are these formally stated policy goals, deemed appropriate at a political level, the goals as viewed by the program administrators, or the goals of other stakeholders? O’Faircheallaigh presents an interesting discussion about the starting points of evaluation and the construction of policies and their objectives by various stakeholders in case studies concerning mining and indigenous communities.

Generally evaluations involve a mix of qualitative and quantitative research methods, although in social research the main focus may be on qualitative information as the evaluators seek to understand what has happened through the implementation. Program statistics and financial information play a part in the evaluation of program performance, but broader questions about the program’s outcomes will require qualitative information sourced from well-designed surveys and other qualitative research methods.

Frechtling&Sharp(1997)providea‘handbook’orelectronicguideintwentychapters, which discusses and walks-through evaluation and evaluative techniques. Of particular value is their understanding that an evaluation which includes both qualitative and quantitative methods (that is, a multi-method approach to evaluation) can increase both the validity and reliability of data. See the Other Learning Resources link for this topic on the PSM Program National website.

The ANAO periodically evaluates other agencies’ evaluations to establish their appropriateness. The most recent such report, the Program Evaluation in the Australian Public Service,AuditReportNo.3,PerformanceAuditAcrossAgency(1997),identifies many shortcoming in reports and identifies practices that might enhance future evaluations. Most relevant from a managing out perspective was the following recommendation about how agencies should publicly communicate evaluation findings and recommendations:

1. Provide evaluation reports to relevant stakeholders, including the provision of information on significant evaluations to the responsible minister.

2. Give summary reports for sensitive evaluations, where appropriate.

3. Include adequate information in annual reports and Portfolio Budget Statements in accordance with the relevant guidelines.


9.5.4 Evaluation and Learning

Because of the perceived risk of revealing policy failure or that a program cannot ‘prove’ itself, evaluation activities can be viewed with some fear and trepidation by those who may be involved in a program, such as program managers or organisations funded under a program. This negative view of evaluation is unfortunate, as the processes involved with evaluation should encourage the honest questioning of program intent and what it is actually achieving. As O’Faircheallaigh (2002:ix) observes, ‘a failure to systematically evaluate public policies can be very costly. It can result in the maintenance of initiatives which are absorbing public resources while failing to deal with the problems or issues they are meant to address’.

Much valuable information is generated from every evaluation, even if an evaluation is unable to resolve all the key questions it sets out to answer. It is important to embrace evaluation processes in the policy and program cycle as valuable and important opportunities to learn from experience and actively apply the learnings to improving current programs and designing better programs in the future. How can learning be promoted through evaluation? Here are some suggestions:

• Takinganinclusiveandparticipatoryapproachtoevaluationattheoutset,involving a (manageable) range of internal and external stakeholders. This way, the learning begins at the scoping and planning stage of evaluation.

• Aimingtokeepthescopeofevaluationbroadifpossible,enablingawiderrange of key questions to be pursued. A broad scope will generate more useful insights into the policy, particularly about unintended side effects.

• Allowingforstakeholderstodiscusstheresultsoftheinformationgatheringbefore drawing conclusions. Many values are at work in processing information and it is helpful to hear alternate views about what the information means. It is a key learning point in an evaluation project.

• Ensuringthatthefinalresultsoftheevaluation,thereport,isabletobewidely disseminated to those who have an interest in the policy or program.

• Conductinganactiveprocessofidentifyingthekeylearningsfromtheevaluation and planning for the application of learnings to the policy or future policies.

Note, The Magenta Book from the UK provides a helpful overview of evaluation. This can be accessed at: http://www.nationalschool.gov.uk/policyhub/downloads/Chapter_1.pdf viewed on 18 September 2008

Required Reading 9.2o’Faircheallaigh, c 2002, ‘program and policy evaluation: fundamental problems and issues’, in

A New Approach to Policy Evaluation: mining and indigenous people, ashgate, london.


Review and conclusionin this topic, we have progressed towards a more concrete understanding of the role and responsibilities of managers within this new environment, in the context of new imperatives towards managing out, across the public sector in australia. Drawing on a range of literature that examines the characteristics of well-managed organisations and that provides advice and checklists that can help you plan improved practice, we have turned our attention to practical resources. by this stage you should be able to:

1. Summarise your understanding of the challenges involved in managing out.

2. review the skills, knowledge and capacities needed to manage out effectively.

3. understand different options for obtaining information on public sector performance and outcomes, including the strengths and limits of different models for identifying and applying performance indicators.

4. understand different models of evaluation and their application in the public sector context.

Required ReadingReading 9.1 Albury, D 2005, ‘Fostering innovation in public services’, Public Money

and Management, January, pp. 51–56.

Reading 9.2 O’Faircheallaigh, C 2002, ‘Program and policy evaluation: fundamental problems and issues’, in A New Approach to Policy Evaluation: mining and indigenous people, Ashgate, London.

Further ReadingAlthausC,BridgmanP&DavisG2007,The Australian Policy Handbook, 4th ed,

Allen & Unwin, Ch 11 ‘Evaluation’.

Borins S 2000, ‘Loose cannons and rule breakers, or enterprising leaders? Some evidence about innovative public managers’, Public Administration Review, vol. 60, no. 6, pp. 498–516.

This article explores the debate concerning public sector entrepreneurship and helps elucidate the pros and cons associated with pursuit of this model.

LontiZ&GregoryR2007,‘AccountabilityorCountability?PerformanceMeasurement in the New Zealand Public Service 1992–2002’ Australian Journal of Public Administration, Vol. 66 No. 4 pp. 468–484.

Tilbery C 2006, ‘Accountability via Performance Measurement: The Case of Child Protection Services’ Australian Journal of Public Administration, Vol. 65 (3) pp. 48–61.


topic 9: Required Reading

Albury, D 2005, ‘Fostering innovation in public services’, Public Money and Management, January, pp. 51–56.


topi

c 9:

Req

uire

d R

eadi

ng

O’F

airc

heal

laig

h, C

200

2, ‘P

rogr

am a

nd

polic

y ev

alua

tion:

fund

amen

tal p

robl

ems

and

issue

s’, in

A n

ew a

ppro

ach

to p

olicy

eval

uatio

n: m

inin

g an

d in

dige

nous

peo

ple,

A

shga

te, L

ondo

n.

topic NiNE: Managing out in practice

Documents

Transcript of topic NiNE: Managing out in practice