Amazon Web Servicesdatacom.co.nz/Datacom/media/Digital-Assets/Documents/...Amazon Web Services The...

#8

DATACOM INTELLIGENCE

WHITE PAPERIssued: October 20th 2013

Datacom Group Limited | All content © Datacom 2016 | Available for release on request

#3W H I T E PA P E R

DATACO M I N T E L L I G E N C E

ISSUED April 2016

Amazon Web Services:How to make the most of AWS

2 www.datacom.co.nz | www.datacom.com.au

Datacom White Paper | M A K I N G T H E M O S T O F A W S

How to make the most of Amazon Web Services

The potential speed, agility, scalability and cost benefits of public cloud, available through its automation, hyper-scale computing, utility model and self-management of everything-as-a-code, make it compelling to many organisations, including Datacom. Gartner analysts forecast the worldwide public cloud services market to continue growing this year, by 16.5%, reaching US$204 billion by the end of 2016.

Amazon Web Services (AWS) has the most mature and arguably the most comprehensive offering of all global public cloud providers, along with the most compute capacity. In contrast to many of its competitors, AWS is well-suited to a wide variety of technologies, including Microsoft, Java and open-source applications, and Linux- and Windows-based operating systems – even providing completely serverless options. According to recent data from Synergy Research Group (Feb 2016), AWS also has the largest share of the worldwide cloud infrastructure services market, at over 31% in 2015.

Datacom is an AWS Managed Service Provider Partner across Australia, New Zealand, Malaysia and the Philippines. We have worked in partnership with AWS for years on projects of all sizes for a wide range of organisations. For example, Datacom recently took the ageing, inflexible and cumbersome infrastructure of a global energy drink manufacturer and eliminated these problems with a full data centre migration to AWS that included all hosted core and edge services.

Over the years, Datacom has encountered organisations using AWS but not getting hoped-for results. Usually, this is because they have pursued its benefits by engaging with the environment in traditional ways, suited to the on-premise era.

“Adopting public cloud can give organisations a more flexible, agile and functional IT platform that puts them in a better l ong-term position and paves the way for digital transformation.

But to fully harness its power, organisations may need to make fundamental changes to their approach to IT.”- SAYS TIM HARTILL, ASSOCIATE DIRECTOR, INTEGRATED SOLUTIONS GROUP AT DATACOM.

This paper aims to help current and potential AWS customers alike to update their approach and get the most out of the platform. It contains 10 sections that:

• raise important issues that organisations need to consider,

• outline problems they need to overcome, and

• specify changes they need to make.



Making the most of any public cloud requires preparation. Ad hoc adoption and lack of strategic planning can lead to, among other consequences, stalled projects, costly changes in direction or solutions that don’t meet business needs. So Datacom urges organisations to build at least some kind of plan for the adoption and management of AWS.

Ideally, you’ll create a coherent way to bring different services, workloads, architectures and processes together, according to a central strategy that aligns with and supports organisational goals and the overall vision. For Datacom advice, and a framework, on how to build such a strategy and plan refer to our paper, Defining The Right Cloud Strategy And Plan For Your Organisation (Feb 2016).

1. Plan first



According to Datacom customer research outlined in Before You Go Public, Read This (Oct 2015), few organisations are currently planning to go ‘all in’ to public cloud, and will, therefore, retain some of their workloads on-premise or in private cloud. Many services are, and will be for the foreseeable future, delivered via applications or workloads with a hybrid set up.

“Organisations may be keen to take advantage of AWS for front-end applications that require elasticity while keeping their databases in a private cloud or their own servers.” - SAYS KAYE HARDING, BUSINESS UNIT MANAGER (AWS AND DEVOPS) AT DATACOM.

Trans-Tasman media company, Fairfax, for instance, wanted a unified web portal that would allow customers to self-manage subscriptions and deliveries. The nature of the membership portal, and the inconsistent and spikey demand for it from users, meant that it was ideally suited to a public cloud environment. So Datacom designed the portal to take advantage of the elasticity inherent in public cloud and built it to run in AWS. The front-end was integrated with the customer database and legacy system in Fairfax’s data centres in Australia and New Zealand – all via Datacom’s dedicated hybrid cloud link. Customers now have a seamless, high-quality experience regardless of the demands placed on the system.

Hybrid cloud architecture, however, requires careful management and planning to account for key factors such as latency, security and compliance, as well as potential added complexity and transition costs. This requires a focus on both integration and interconnectedness – critical factors that are sometimes overlooked when adopting public cloud. For example, if an organisation stretches certain components by running them in AWS, what is the impact on other, reliant components?

On top of this, with greater adoption of public cloud (and proper planning), usually comes a corresponding service orientation and increasing focus on business services and related outcomes enabled by cloud. These outcomes may include risk or reputation management, reducing cost and making key services available when needed and at a suitable quality. They are often delivered through the aggregation of multiple providers, services and solutions – and various SLAs.

Making these services and outcomes happen and managing the many moving parts involved is clearly an important function and a complex task, which organisations can take on themselves or outsource, at least in part, to a qualified partner like Datacom. Whatever the approach taken, there is a growing need for those in the organisation involved in AWS to understand the way different clouds and related services interact and how they can be integrated – with each other and with other environments and types of IT.

2. Face complexity



3. Embrace conformity

“AWS provides global state-of-the-art infrastructure and services, so organisations should conform to how they do it to get the best performance and value,” - SAYS GAURAV ARORA, NATIONAL PRACTICE LEAD – AWS AT DATACOM.

For example, people often treat AWS as an extension of their infrastructure, but if they just focus on the compute and storage layer (i.e. virtual servers) then they may miss out on the automation and higher value services.”It’s contradictory, but part of the solution to the problem of added complexity described above is one of its key contributing factors: public cloud. That is, optimising AWS and reaping the potential flexibility, scalability and agility rewards on offer requires organisations to, among other things, automate and standardise many IT processes. In doing so – by conforming to the demands of the platform – organisations also put themselves in a better position to manage increasing complexity, at speed.

This can make for a more reactive relationship than many organisations are used to, but this is a minor drawback compared to the benefits available. What this conformity means in practice across operations and development is outlined below.



4. Adopt a new mentality

In cloud operations, a new method of engagement is necessary to match the tectonic shift in focus from hardware to software that public cloud engenders. Engineers no longer have direct access to infrastructure so they view servers through a portal and use software to control things. This means some people may need to adopt a new mentality and update their skills substantially. They need to move away from traditional, manual, GUI-based methods of monitoring and control to using scripts and coding to enable process automation and managing by exception.

For example, where server outages in on-premise or even highly virtualised environments are seen as a problem to be investigated or rectified, in AWS machines may be switched off at any time when they are not needed. Moreover, even if a workload does cause an issue, it can be readily destroyed and a new one redeployed in its place in a matter of seconds.

As automation progresses, engineers steadily remove manual intervention and human error. In doing so, they can free themselves up to have control over more of their organisation’s estate – and better handle increasing complexity. Datacom has seen FTE ratios improve by up to three times. They can also be more proactive and strategic day to day, moving from

run-and-operate thinking towards an innovation and value-add mentality. That is, they can get more involved in higher-value activities, such as capacity planning and service management and delivery.

Of course, this need to change may be reduced if much of the management is outsourced to a third party, but there will always be a learning curve of some kind required to ensure the business can make the most of its partner’s or partners’ services and support. And note that there is often understandable resistance to these changes. Therefore, change management is a crucial aspect of any transition to AWS and a key consideration to build into cloud strategy and planning.



5. Speed up

A common way organisations drive ever more value out of AWS is adding speed and agility to application development and management. For example, a business may want to improve time-to-market by using AWS to speed up previously month-long release cycles into weekly cycles.

However, as per Datacom’s recent paper, From the Cloud Face: Ten Lessons on Adoption and Management (March 2016), this needs to happen without losing requisite standards, discipline, compliance and life cycle and financial management. In a nutshell, this means speeding up without breaking the business – finding a balance.

To make this happen, among other key changes, infrastructure people need to pare back normal ITIL-based processes and related standards somewhat to allow faster activity without losing too much risk mitigation and rigour. In addition, developers need to work faster than ever, but wrapped in enough protection and control to avoid disaster.

A primary challenge is that there is an historic divide between operations and development people and processes. But to get the most out of cloud, it’s important to integrate these areas as much as possible. That’s why Datacom usually recommends that organisations adopt a DevOps approach, at least to a degree.

Looking at development in particular, AWS makes tools available to support DevOps, agile and lean approaches, which can help speed up application release cycles and make development in an organisation a leaner, more agile and more iterative process – compared with the traditional ‘waterfall’ project-based method.

AWS has a multitude of tools to support continuous integration and continuous delivery, including CodeDeploy, CodePipeline and CodeCommit, and supports an array of coding

languages via APIs. Using the platform and its native tools in combination with a DevOps approach to developing cloud-ready applications for the platform can result in faster, cheaper and more efficient development processes compared with developing on-premise.

“For development, conforming to AWS means building applications that are cloud-ready, in general, and tailored for the unique features of the platform, specifically. In doing so, you are constructing applications that are ready for possible global roll-out and which take advantage of the many benefits of AWS,” - SAYS KAYE HARDING.

For the Fairfax membership portal, for instance, Datacom took a lean development approach to build a minimum viable product (MVP), make it available as early as possible, and iterate from that over time. The release cycle was accelerated without any negative impact on end users or the business.

As well building new applications, organisations can of course also rebuild applications to run on AWS, as Datacom has done for a wide range of business over many years. For example, we helped a global English learning organisation based in Australia, which had found that its new business-critical application lacked scalability. We reviewed the whole application stack ecosystem and refactored it to run on AWS and take advantage of its scalability.



Although many enterprises will need to review and update their traditional, often ITIL-based, processes to maximise the value derived from AWS, ITIL is global best practice for a reason. Many of its principles and practices are invaluable to the ideal operation of AWS within an organisation.

“Organisations can bring process maturity, control and governance best practice from the legacy world to bear on AWS, at the same time as they use the environment to innovate and accelerate.” - SAYS GAURAV ARORA.

ITIL tools such as Problem Management and Configuration Management Databases (CMDBs) are useful, as are basic processes such as Change Management, Service Request and Incident Management. Working within an ITIL structure and having a correspondingly high level of process maturity helps an organisation align with AWS processes and improve outcomes.

For example, Datacom recently helped a major Australian betting organisation, which had a large AWS resource sprawl, to develop organisation-wide standards and controls within AWS, enforced using AWS Config Rules. This enabled the organisation to ensure compliance without constraining the agility and flexibility required by their digital teams and development partners.

Even simple steps such as logging incidents, their timing, what they affected and so on enable operations people to engage more effectively with AWS support teams if required to manage a technical issue.

Incidentally, amid all the coding and machine automation inherent to public cloud, Datacom has found first-hand that, with potentially mission-critical business services running in the environment, having a productive working relationship with service provider support teams – as well as having the mature, robust processes described above in place – makes a major difference if an issue arises.

6. Maintain maturity, controland governance



Business services, even something as simple as email, are built from and rely on a number of components, including applications, firewalls, switches, servers and storage. Mapping these services, the applications and infrastructure that enable them, and the interconnections and dependencies of the various components, are an integral part of planning for AWS adoption and optimisation and managing a hybrid cloud environment.

This is why Datacom recommends monitoring at an availability-of-business-service level. This means having dependency-based monitoring from the business service level down through applications and infrastructure, including AWS. We also recommend automated root-cause analysis to provide information and evidence for problem management processes and liaising with AWS support teams, if required. For this, organisations need to implement robust analysis and troubleshooting tools, as outlined below.

“In simple terms, if your AWS servers or services go down you need to know what will be affected, for how long, and what impact that will have on your organisation in order to determine and take necessary steps.” - SAYS TIM HARTILL

This means knowing what it takes to keep a high availability (HA) application operational if X or Y shuts down. And things do go down from time to time. For example, AWS servers may go offline for maintenance purposes, as part of a planned, routine outage. In this context, how many more stand-in machines are required for each service compared to traditional, on-premise IT? The approach we recommend above will improve your organisation’s ability to manage these events and ensure availability of business services.

Broadly speaking, the skills required for such mapping, monitoring and management include knowing how applications work, how infrastructure works and how they work together. Operations engineers need to co-operate with developers and application specialists to ensure applications run smoothly in AWS through the correct use of resiliency and performance techniques, and by testing and monitoring correctly. What is supported in AWS and what isn’t need to be understood – especially in a hybrid environment. As reliance on AWS increases, along with complexity, automating parts of processes as much as possible using scripts becomes increasingly important.

7. Monitor availabilityof business services



8. Combine capacityand commercials

With AWS, changing capacity has instantaneous cost implications. Capacity planning amalgamates with financial planning, in a sense. Business processes and procurement models need to be reviewed and potentially altered to account for this and ensure your organisation is benefitting from the inherent scalability, flexibility and utility nature of the platform.

An increase in capacity needs to trigger clear workflows aimed at getting commercial approvals for the greater spend – a challenge for organisations with complicated procurement processes and financial delegation models.

Reducing capacity can, of course, result in infrastructure cost savings – indeed, one of the most sought-after benefits of using AWS – but Datacom has found that some organisations are not well prepared for managing the process of turning off servers that are not in use – especially at the speed of public cloud. It may be that the new mentality of ‘server off doesn’t mean disaster’ has not permeated the operations team. Or it could mean that the focus is more on standing things up in AWS rather than planning properly for switching things off. The latter needs careful consideration to maximise cost efficiencies and account for the host of dependencies that can build up on top of the machines that have been turned on.

Key to planning for and managing this is building and implementing provisioning processes. In the same way as described in the section above, these need to account for the business services enabled by IT and the applications, infrastructure and other components that deliver it. In other words, you must know how you’re using AWS in your organisation, and how this might change over time. For example, an online retailer would prepare for capacity changes during and after peak shopping periods. It also needs to answer questions such as when can you turn things on or off and what happens if you do. Access to AWS and all the services that rely on it needs to be controlled and logged. Without all this, a server turning off may indeed mean disaster.



Adopting and making the most of AWS will almost certainly require investment in new tools. In general, Datacom recommends that organisations have a tooling strategy focusing on tools with API-based integration capabilities.

This avoids the lock-in that some proprietary tools cause, which constrains customisation, adds complexity and hampers agility. For optimal outcomes in a hybrid cloud environment too, it is better to use API-based tools – large or small – that enable cross-cloud platform integration alongside native AWS tools.

9. Review your tooling strategy

As mentioned above, in the highly automated world of AWS, using start-up and shutdown scripts should be a goal for operations teams. Alongside this, server health checks are required to ensure performance. AWS provides native tools to help with such tasks. For instance, AWS Lambda enables task scheduling that can be utilised in conjunction with scripts to wake up servers, get them to perform a job, and then shut them down – all automatically.

Other native tools worth noting include:

• AWS Service Catalog – allows organisations to centrally build and manage commonly-used and compliant catalogues of IT services – comprised of a range of components, from virtual machines and databases to complex application architectures. Once built, these IT services can be deployed automatically and repeatedly, in one click, saving time on development and management, and helping to avoid sprawl

• AWS Trusted Advisor – another useful tool for making the most of AWS, it reports on cost optimisation, performance and compliance issues, and recommends ways to improve these things

• AWS Inspector – provides an automated security assessment and rule-based compliance service at the application level

Monitoring tools have new challenges with public cloud: not all were built for this environment. For example, more machines are usually required in public cloud compared with on-premise (to account for machines switching off from time to time) to provide the same service. This means that, if monitoring agents are placed on all AWS machines, they may produce too many alerts to handle. And monitoring costs may go up. So monitoring in AWS needs a new approach, and to be tested and fine-tuned over time.

To enable availability-of-business-services monitoring, many Datacom customers currently use a mix of native AWS tools alongside others. This is practical as long as a monitoring policy is in place to make clear what the various tools cover.

Organisations should also assess their approaches to data backup as they adopt AWS. In a hybrid cloud situation, this isn’t a simple task. For backup, as with monitoring, a mixture of traditional and native AWS tools may be the best option – at least in the short term. Although backing up cloud-ready applications may be relatively straightforward in AWS, replicating traditional enterprise backup methodologies in this environment without a dramatic increase in cost is challenging.



An increasingly important skill required to make the most of AWS is commercial management. This is needed when considering the terms and conditions of the environment; licencing issues regarding applications you may want to run in AWS; and how to return services, applications and data from AWS back to the business, if required.

On licencing, it’s important to note that there may be restrictions for applications moving to AWS, and end-user organisations, not AWS, are responsible for complying with them. So the issue needs careful attention to avoid potential penalties from vendors. This is, of course, a chance to review your organisation’s licencing position and potentially reduce attendant costs.

For the Australian betting organisation mentioned above, Datacom provided consulting services around licence optimisation in AWS. The recommendations supplied will help this customer with a best practice approach to licencing compliance and cost optimisation through re-harvesting of un-used software assets. We also centralised deployment of Identity and Access Management across multiple AWS accounts using automated scripts and scheduled the AWS environment using Lambda. This allowed for a centralised and standardised approach to user account creation and grant of access across multiple AWS accounts and VPCs.

For disaster recovery in AWS, Datacom recommends testing to ensure that it works – especially before go-live of mission-critical systems. For example, we might move a customer’s business service or services from the AWS production environment to a secondary site and back again to ensure everything continues to work. This has the added benefit of revealing in advance any issues with the services and related systems, and with the disaster recovery architecture and processes. To further reduce risk, organisations should also plan for how to recover services back to the business if things don’t work out with AWS.

10. Consider licensingand disaster recovery



Reap the rewards

Organisations are of course always in a state of transition with their IT – refreshing kit, updating software, rolling out new solutions. But, as shown above, successfully adopting AWS may require profound changes to various parts of an organisation to be made. Nowadays, organisations must also accept and prepare for accelerating change as public cloud services and other technologies evolve. All this is on top of managing a sometimes sprawling and complex hybrid cloud environment.

Datacom can help. Our software and integration teams have extensive experience in application development, refresh and modification for any type of IT environment – public cloud or otherwise – and delivering the project and change management required. Our professional services, IT management and operations teams are adept at helping customers to optimise cloud adoption and ongoing management – whether they outsource the latter to us or keep it in-house. We also help them take advantage of the scale and economies of SaaS and PaaS, alongside IaaS.

Importantly, we take a technology-independent approach to solving our customers’ business problems. Our aim is to find the best solution for them, whatever that may be. So when we assess public cloud providers as part of a potential solution, our approach is both consultative and consistent.

That said, as an AWS Managed Service Provider Partner, Datacom has a close relationship with AWS, often working in tandem on projects and pushing the boundaries of what’s possible with the environment. We have AWS operations specialists, with blended software and infrastructure skills, who manage, for customers, applications that we have architected to take advantage of the unique features of AWS. We are also at the front line of new innovations in AWS and evolving best practice, as well as changes to pricing, SLAs and other aspects of the platform.

As this paper has shown, we are therefore able to advise customers and help them from an informed, experienced position, and across a wide range of areas related to AWS, including development and operations, designing and building cloud architecture, and integrating and managing complex hybrid cloud and/or multi-cloud environments.

For more information and advice on how your organisation can take full advantage of AWS, please contact us on [email protected].

Amazon Web Servicesdatacom.co.nz/Datacom/media/Digital-Assets/Documents/...Amazon Web Services The...

Documents

Transcript of Amazon Web Servicesdatacom.co.nz/Datacom/media/Digital-Assets/Documents/...Amazon Web Services The...