Copy VPC Flow Logs to S3 Bucket CloudFormation Templates - User Guide

CloudFormation Templates - User Guide

Solution BackgroundBased on AWS Security and Audit Account best practice, we need to store security logs in a centralized bucket. For example, we have a production account, a development & test account and an audit account. We need to store CloudTrial logs, AWS Config logs and VPC flows logs to a central S3 bucket which belongs to audit account. The auditor (a user in audit account) can have read-only access right to visit the central S3 bucket.

As we know ， VPC flow logs are saved in CloudWatch Logs, we need to copy all flow logs to S3 bucket, then these log files can be replicated to central S3 bucket for archiving purpose or for further analysis.

The open source CloudFormation template from AlterLogic (https://github.com/alertlogic/cloudwatch-logs-s3-export#setup) is a good choice, however, it does not support China Regions (BJS and ZHY).

I made some modification to localize AlterLogic’s solution, because China region use different ARN name and AWS endpoint name.

Benefit This solution can work with another solution (cross account S3 bucket replication https://collaborate-corp.amazon.com/nuxeo/nxpath/default/default-domain/workspaces/ProServe%20Goldemine%20Workspace/S3%20Bucket%20Cross%20Account@view_documents?tabIds=%3AFileTab ) to replicate all VPC flow logs to the central S3 bucket in BJS or ZHY region.

How to Deploy the TemplatesThe solution contains the following files:

cwl-s3-export-new.templateThis is the major CloudFormation template

create-cwl-s3-export-new.templateThis is the nested CloudFormation template which is called by cwl-s3-export-new.template

cloudwatch-logs-s3-export.zip

2

CloudFormation Templates - User Guide

This is the lambda function code (NodeJS) which is used to extract the log data from CloudWatch Logs and save them into destination S3 target via AWS kinesis service.

Step-1: Upload template file and NodeJS package to S31. Login and switch to S32. Select a bucket which will be used as a CloudFormation template repository 3. Click on the bucket name and select Create folder

4. Input “template” and click save5. Click on bucket name – template6. Click Upload

7. Select file named cloudwatch-logs-s3-export.zip and create-cwl-s3-export-new.template and click upload

Step-2: Execute CloudFormation template1. Login and switch to CloudFormation service and click create new stack button

2. Upload the template (cwl-s3-export-new.template) to S3

3. Click Next and enter stack name and other parameters as below:Parameters Comments

Stack name The name of this stackCloudWatch log group name The name of the CloudWatch Logs Group to export to

S3 bucket.Log Format Choose the format that best describes the type of logs

in the selected log group. For VPC flow log, select AWS VPC Flow Logs

S3 bucket name: S3 bucket to archive CloudWatch logs into.

3

CloudFormation Templates - User Guide

Logs folder name Add a name of the folder to place logs into. If you leave this parameter empty, the name of the stack will be used.

CloudWatch subscription filter Specify the CloudWatch subscription filter to be used to filter data sent to the S3.

Lambda package S3 bucket prefix The prefix of the S3 bucket which contains Lambda package to be used for archiving CloudWath Logs to S3. Note that the name of the bucket must contain a region name suffix. 

Lambda package name Object name containing Lambda package to be used for archiving CloudWath Logs to S3.This should be: cloudwatch-logs-s3-export.zip

Kinesis shards count Number of shards to create for the Kinesis subscription stream.

Create Template URL Prefix Prefix of the Bucket URL that contains '/templates/create-cwl-s3-export-new.template' CloudFormation template, which sets up a reliable export of the CloudWatch Logs to S3 bucket

4. Click Next twice, select the following:

5. Click Create8. Wait for the template to finish, you will see the following screen if AWS deploy

this template successfully.

9. If you check the destination S3 bucket, you can find the replicated VPC flow logs like this:

4

CloudFormation Templates - User Guide

10. For each log file, the content looks like this

LimitationsThese CloudFormation templates can only be used in China Regions (BJS and ZHY). Neither BJS region nor ZHY region has Kinesis Firehose service now, so Kinesis stream is the only choice.

5