Monitoring and Logging on AWS - AWS Immersion Days€¦ · Events–Delivers a near real-time...

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Ashley Miller, Sr. Solutions ArchitectTampa, Florida

Date

Monitoring and Logging on AWS


Module Overview

Overview of the AWS Monitoring and Logging options across Performance, Availability, Security and Cost areas.

Common questions we will cover in this module:

• How do I capture, view and act on resource availability and state changes?• How do I track the key performance indicators across AWS resources?• How do I track API calls within my AWS accounts?• How do I track cost within my AWS accounts?• How do I know which capability to use for my use case?• How do the monitoring and logging capabilities integrate with other AWS

services?


My Application

Why monitor?


Why monitor? (continued)

Customer Experience- Are my customers getting a good experience?Performance & Cost- How are my changes impacting overall performance?Trends- Do I need to scale?Troubleshooting & Remediation- Where did the problem occur?Learning & Improvement- Can I detect or prevent this problem in the future?


Low to moderate visibilityLow degree of automation

High visibilityHigh degree of automation

Traditional Approach vs.


1

Performance & Availability Monitoring and Logging


Amazon CloudWatch


AWS CloudWatchMonitoring services for AWS Resources and AWS-based Applications.

Monitor and Store Logs

Set Alarms (react to changes)

View Graphs and Statistics

Collect and Track Metrics

What does it do?

How can you use it?

React to application log events and availability

Automatically scale EC2 instance fleet

View Operational Status and Identify Issues

Monitor CPU, Memory, Disk I/O, Network, etc.

CloudWatch Logs / CloudWatch Events

CloudWatch Alarms

CloudWatch Dashboards

CloudWatch Metrics


- Metrics- Namespaces- Dimensions- Time Stamps- Units

Amazon CloudWatch Concepts

- Statistics- Periods- Aggregation- Alarms- Regions


Metrics – Data about the performance of your systems.

Examples:

EC2 – CPUUtilization, Network In/OutEBS - DiskRead/Write & Ops/BytesRDS – BufferCacheHitRatio, CommitLatencyS3 - NumberOfObjectsCustom - put-metric-data API Call

Default Interval – 5 MinutesEnhanced Interval – 1 MinuteRetention – 2 Weeks

Amazon CloudWatch Metrics Examples


Key CloudWatch Capabilities

- Monitors your AWS resources in real time.

- CloudWatch to track built in and custom metrics.

- CloudWatch Alarms for desired conditions

- CloudWatch Events to automate responses

- CloudWatch Logs for log collection, aggregation and monitoring

- Dashboards for customized views across resources


CloudWatch Metrics & Alarms

AWSResource

YourCustom

Data

Metric Alarm Action

CloudWatch


Alarms – Send notifications based on state of monitored resources.

- Watches metric over a period of time and sends messages when a defined metric threshold is reached.

- Utilizes with Amazon Simple Notification Service (SNS) to send messages.

- Alarm state must be maintained for specified number of periods.

- Integrates with Auto Scaling

Amazon CloudWatch Alarms


Amazon CloudWatch Alarm Example

An alarm has three possible states:

OK—The metric is within the defined threshold

ALARM—The metric is outside of the defined threshold

INSUFFICIENT_DATA—The alarm has just started, the metric is not available, or not enough data is available for the metric to determine the alarm state


Alarm Actions

Action

Notification (SNS)

Auto Scaling Action

EC2 Action

Recover

Stop

Terminate

Amazon EC2Auto Recovery

Use this actiontogether withStatus Checksto automate

instance recovery


Events – Delivers a near real-time stream of system events that describe changes in Amazon Web Services (AWS) resources to targets such as Lambda, SNS & SQS.

- Events are delivered through resource state changes, CloudTrail API calls, custom publications or scheduled.

- Rules match incoming events and route them to one or more targets for processing.

- Targets are specified in rules and receive matching events.

Amazon CloudWatch Events


- Scheduled EBS Snapshots

Amazon CloudWatch Events Examples

- Automatically Tag Resources


Amazon CloudWatch LogsAmazon CloudWatch Logs lets you grab everything and monitor activity.

§ Managed service to collect and keep your logs§ Aggregate and centralize logs across multiple sources§ CloudWatch Logs Agent for Linux and Windows instances§ Integration with Metrics and Alarms§ Export data to S3 for analytics and/or archival§ Stream to Amazon ElasticSearch Service or AWS Lambda


CloudWatch Logs + Filter

AWSResource

YourCustom

Data

Metric Alarm Action

CloudWatch

FilterLogs


Amazon CloudWatch DashboardsAmazon CloudWatch Dashboards creates a single consolidated view across your resources customized by you.

- A single view for selected metrics to help you assess the health of your resources and applications across one or more regions.

- An operational playbook that provides guidance for team members during operational events about how to respond to specific incidents.

- A common view of critical resource and application measurements that can be shared by team members for faster communication flow during operational events.


CloudWatch Dashboard Example


2Security Monitoring and

Logging


Different log categories

AWS Infrastructure logs

§ AWS CloudTrail§ Amazon VPC Flow

Logs

AWS service logs

§ Amazon S3§ AWS Elastic Load

Balancing§ Amazon CloudFront§ AWS Lambda§ AWS Elastic Beanstalk§ …

Host based logs

§ Messages§ Security§ NGINX/Apache/IIS§ Windows Event Logs§ Windows

Performance Counters

§ …Security related events


AWS CloudTrailRecords AWS API calls for your account


What can you answer using a CloudTrail event?

§ Who made the API call?

§ When was the API call made?

§ What was the API call?

§ Which resources were acted up on in the API call?

§ Where was the API call made from and made to?

Supported services:http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-services.html


What does an event look like?{

"eventVersion": "1.01",

"userIdentity": {

"type": "IAMUser", // Who?

"principalId": "AIDAJDPLRKLG7UEXAMPLE",

"arn": "arn:aws:iam::123456789012:user/Alice", //Who?

"accountId": "123456789012",

"accessKeyId": "AKIAIOSFODNN7EXAMPLE",

"userName": "Alice",

"sessionContext": {

"attributes": {

"mfaAuthenticated": "false",

"creationDate": "2014-03-18T14:29:23Z"

}

}

},

"eventTime": "2014-03-18T14:30:07Z", //When?

"eventSource": "cloudtrail.amazonaws.com",

"eventName": “StartInstances", //What?

"awsRegion": "us-west-2",//Where to?

"sourceIPAddress": "72.21.198.64", // Where from?

"userAgent": "ec2-api-tools 1.6.12.2",

"requestParameters": {

"instancesSet": {

"items": [{

"instanceId": "i-ebeaf9e2“// Which resource?

}]

}

},// more event details

}


AWS CloudTrail Best Practices

1. Enable in all regions2. Enable log file validation3. Encrypted logs4. Integrate with Amazon

CloudWatch Logs5. Centralize logs from all

accounts

Benefits§ Configure all accounts to send

logs to a central security account

§ Reduce risk for log tampering§ Can be combined with Amazon

S3 CRR


AWS Config

• Get inventory of AWS resources

• Discover new and deleted resources

• Record configuration changes continuously

• Get notified when configurations change


Config Rules

• Set up rules to check configuration changes recorded• Use pre-built rules provided by AWS• Author custom rules using AWS Lambda • Invoked automatically for continuous assessment • Use dashboard for visualizing compliance and identifying

offending changes

• Community-contributed custom rules at https://github.com/awslabs/aws-config-rules

https://github.com/awslabs/aws-config-rules


Amazon VPC Flow LogsLog network traffic for Amazon VPC, subnet or single interfaces


Amazon VPC Flow Logs§ Stores logs in AWS CloudWatch Logs§ Can be enabled on

• Amazon VPC, a subnet, or a network interface• Amazon VPC & Subnet enables logging for all interfaces in the VPC/subnet• Each network interface has a unique log stream

§ Flow logs do not capture real-time log streams for your network interfaces

§ Filter desired result based on need• All, Reject, Accept• Troubleshooting or security related with alerting needs?• Think before enabling All on VPC, will you use it?


VPC Flow Logs

• Agentless• Enable per ENI, per subnet, or per VPC• Logged to AWS CloudWatch Logs• Create CloudWatch metrics from log data• Alarm on those metrics

AWSaccount

Source IP

Destination IP

Source port

Destination port

Interface Protocol Packets

Bytes Start/end time

Accept or reject


VPC Flow Logs

• Amazon ElasticsearchService

• Amazon CloudWatch Logs subscriptions


Managing, Monitoring & Processing Logs

CloudWatch Logs - Near real-time, aggregate, monitor, store, and search

Amazon Elasticsearch Service Integration (or ELK stack)- Analytics and Kibana interface

AWS Lambda & Amazon Kinesis Integration- Custom processing with your code

Export to S3- SDK & CLI batch export of logs for analytics

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.Arrow direction indicates general direction of data flow

EC2 instances

Logstashcluster on EC2

DynamoDBTables

RDS Databases(via JDBC)

SQSQueues

KinesisStreams

AmazonElasticsearchdomain

CWL Logs agent

VPCFlow Logs

CloudTrailAudit Logs

S3AccessLogs

ELBAccessLogs

CloudFrontAccessLogs

SNSNotifications

DynamoDBStreams

SESInbound

Email

CognitoEvents

KinesisStreams

CloudWatchEvents &Alarms

ConfigRules

Lambda

Filebeat agent

REST API client

S3

CloudWatchLogs


AWS Technology Partner solutions integrated with CloudTrail


3Cost Monitoring


We’ll Talk Through Some…

Frameworks Tools Best Practices


1. Understand What is Deployed and What it Costs…


By Employing Tags…

Key (Attribute): 127 Unicode characters

Value (Detail/Description): 255 Unicode characters

Tags per resource: 50 tags

Jane_Doe


…And Using The Different Types of Tags Appropriately

Resource Tags• Provide the ability to organize and search within and across resources• Filterable and Searchable• Do not appear in Detail Billing Report

Cost Allocation Tags• Ability to map AWS charges to organizational attributes for accounting purposes• Information presented in Detailed Billing Report and Cost Explorer• Available on certain services or limited to components within a service (e.g. S3 bucket but

not objects)


Tag Key Examples

Cost Center

Business Unit

Environ. Tier

Owner

Dept./ Group

Product / Application

Shutdown Time

Support ContactEndpoint

Backup


2. How does AWS create my bill?

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 45

AWS Bill Development Process (Simplified)

Collect Consumption

Data

Create the Billing Report

Entries

Apply Discounts (RI,

Spot, EDP)

Add Unused RI Charges

Make Billing Report

available to customer

1. Consumption Data is aggregated across all linked accounts, based on CloudWatch entries

2. RI discounts, Spot Discounts, EDP Discounts, and non-use charges are applied based on the aggregated set of purchases across the linked accounts


3. Proactively Monitor Your Account Billing Usage…


By Using Detailed Billing Reports andEnabling Billing Alerts…


By Using Budgets, Forecasts, and Alarms in the Cost Explorer


Billing Alarms


Respond to billing alerts in CloudWatch.

When an alarm is triggered:1. Email the project team, and the budget approver (AWS console)2. Open a Service Management Ticket in your ITSM system3. Open a Ticket in your Finance System4. Resize the project’s IT resources5. Cull/Reduce the project’s IT resources6. Shut down the project’s IT resources

Billing Alert Response


…Tools to Manage Billing Data…


4. Identify Idle Resources and Turn Off Unused Instances…

http://docs.aws.amazon.com/solutions/latest/ec2-scheduler/welcome.htmlEC2 Scheduler Solution

http://docs.aws.amazon.com/solutions/latest/ec2-scheduler/welcome.html


…Using Trusted Advisor…

Trusted Advisor


…And Amazon CloudWatch to Monitor,Collect and Track Metrics…

Amazon CloudWatch


Summary

üSetup Sensible Billing Alarms for your OrganizationüProactively Monitor Your Account Billing UsageüLeverage AWS Partner toolsüLeverage Trusted Advisor reports to:

ü Identify Idle Resources and Turn Off Unused Instancesü Identify Under-utilized Resources and Resize themü Identify Baseline Consumption needs to support RI

commitmentsüReview new Discount and Technology Options on a Monthly basis


?Questions

Monitoring and Logging on AWS - AWS Immersion Days€¦ · Events–Delivers a near real-time...

Documents

Transcript of Monitoring and Logging on AWS - AWS Immersion Days€¦ · Events–Delivers a near real-time...