How to Provide Restrected Access to Swift … · Servers can only access Swift Amazon Web Services...

Copyright © NTT Communications Corporation. All rights reserved.

Private Swift EndpointHow to Provide Restrected Access to Swift

Yuki NishiwakiMasaki MatsuhitaMahito Ogura

NTT CommunicationsTechnology Development

1


About_us.ini[DEFAULT]Company = NTT CommunicationsDepartment = Cloud Unit, Technology Development

[mmasaki]Name = Masaki MatsushitaContribution= Trove and Oslo

[ukinau]Name = Yuki Nishiwaki Contribution= Rally and Devstack

2


Agenda: Private Swift Endpoint

1. Background2. Similar Solution in AWS: VPC Endpoint 3. Our Implementation in OpenStack4. Operational Improvements5. A Review and Future Tasks

3


Background: A Web Application

Swift

LoadBalancerfor API

the Internet

OpenStack Internal N

W

OpenStack Interval NW

External NW

User Tenant

DMZ

Internal

No Internet Access

4


Swift

the Internet


W

Background: Save log files from Application Servers

We want to save log files from app servers

log fileslog files

log fileslog files

log fileslog files

LoadBalancer

5


LoadBalancer

Background: Accessing Swift through the InternetUsers

Swift

the Internet


W

6


Background: How to Reach Swift from Closed NW?

Swift

LoadBalancer

the Internet

Can’t Access Swift


W

7


No Routers


Background: How to Reach Swift from Closed NW?

Swift

LoadBalancer

the Internet



W

Provide Internet Access only to Use Swift?8



W

LoadBalancer

DNSCache

C.C.C.C

Z.Z.Z.Z

Our Interest

How to provide access to Swift for closed privatenetwork while preventing internet access?

the Internet

Swift

9


Private Swift Endpoint

● Provide Swift endpoint for closed network○ Servers can only access Swift

● Amazon Web Services (AWS) provides “VPC Endpoint”○ Private Endpoint for VPC (Virtual Private Cloud)○ Endpoint Policy: Restrict Access to S3

■ Only specified buckets (containers)

10


VPC in Amazon Web Services

VPC1 (10.0.0.0/16) VPC2 Public Subnet (10.0.0.0/24)

RouterVM VM

Private Subnet (10.0.1.0/24)Internet

Gateway

the Internet

VM VM No Routes

11




RouterVM VM

Private Subnet (10.0.1.0/24)Internet

Gateway

VM VM

S3

12




RouterVM VM

Private Subnet (10.0.1.0/24)

InternetGateway

the Internet

VM VM

VPCEndpoint

S3

13




RouterVM VM

Private Subnet (10.0.1.0/24)

InternetGateway

the Internet

VM VM

VPCEndpoint

S3

Endpoint PolicyAllow “arn:aws:s3:::bucket_name”

14


Implemetation of Private Endpoint

15



W

LoadBalancer

DNSCache

C.C.C.C

Z.Z.Z.Z

Our Interest

How to provide access to Swift for closed privatenetwork while preventing internet access?

the Internet

Swift

16


Basic Idea: Add “API NW” as External Network

Swift

Internet


W

DNSfor Public

C.C.C.C

LoadBalancer

Z.Z.Z.Z

DNSfor Private

API N

W

10.255.0.7

10.255.0.2

API NW is NOT connected to the Internet

17



W

DNS Server for Private Network

Swift

Internet

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z

API only N

W

10.255.0.2

DNSfor Private

10.255.0.7

otb.nttcloud.net IN A 10.255.0.2

You should register the following A record

IP of Private Endpoint

If you have the following endpoints

Service Service Endpoint

Object Store http://otb.nttcloud.net:8080/v1

Identity http://otb.nttcloud.net:5000/v2.0

18


Use DNS Server for Private Network

Swift

Internet


W

DNSfor Public

C.C.C.C

LoadBalancer

Z.Z.Z.Z

DNSfor Private

API N

W

10.255.0.7

10.255.0.2

The user set “10.255.0.7” as DNS server distributed to client

19


Swift

Internet


W

DNSfor Public

C.C.C.C

LoadBalancer

Z.Z.Z.Z

DNSfor Private Query: otb.nttcloud.net

Response: otb.nttcloud.net IN A 10.255.0.2

API flow from private network(1)

API N

W

10.255.0.7

10.255.0.2

20


Swift

Internet

DNSfor Public

C.C.C.C

LoadBalancer

Z.Z.Z.Z

DNSfor Private

API N

W

10.255.0.2

API flow from private network(2)

Access to 10.255.0.2 (otb.nttcloud.net)


W

21


Pros & Cons for OpenStack Administrators

Pros:● Using only normal features of Neutron● Simple● Easy installation into existing environments

Cons:● Need to maintain DNS records for private endpoints

22


Pros & Cons for End Users

Pros:● Can introduce API NW by their own

Cons:● Need to change DNS server ● Private IP addresses for API NW are reserved by

providers

23


Challenges

24


Challenges to improve usability and operationability

Usability ☐ The users don't need to be conscious of the dns ☐ Access the dns without configuring each tenant router

Operatinablitiy ☐ Eliminiate the extra dns server

25



Usability ☐ The users don't need to be conscious of the dns ☐ Access the dns without configuring each tenant router


26


Swift

Internet

API N

W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z

API only N

W

10.255.0.2

Goal of the first challnge

Use something techniquesso that the users don’t need to be conscious of the dns, when use API NW

DNSfor API

10.255.0.7

27


Use dnsmasq_dns_servers option in neutron(1)

Internet

API N

W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z

API only N

W

API only N

W

10.255.0.2

dnsmasq

neutrondhcp-agent

dnsmasqneutron

dhcp-agent

Only resolve the name of the instance in same network.“<instance name>.<dns_domain>”

Query: ntt.comRes: NXDOMAIN

Query: lb.openstackRes: X.X.X.3 28


Use dnsmasq_dns_servers option in neutron(2)

Internet

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z

DNSfor API

API only N

W

API only N

W

10.255.0.7

10.255.0.2

dnsmasq

neutrondhcp-agent

dnsmasqneutron

dhcp-agent

Administrator can set forwarder of dnsmasq managed by neutron.

The query for unkown name is forwarded to the sepecified dns.

Query: ntt.comRes: 210.226.39.112

forwader: C.C.C.C

29


The dnsmasq use same network as instance

Internet

API N

W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z

DNSfor API

API only N

W

10.255.0.7

10.255.0.2

dnsmasq

neutrondhcp-agent

dnsmasqneutron

dhcp-agent

Caution: The forwarding of dnsmasq is sent from the same network as other instance.

If it refers to outside dns, we can’t forward query from private network.

Not connection to C.C.C.C from private

Query: Forward to C.C.C.C

30


What setting should we change

AppServer

AppServer

AppServer

Private

DM

Z

RT

X.X.X.3

DNS CacheServer

C.C.C.C

LB

[DEFAULT]# Comma-separated list of DNS servers which will be used by dnsmasq# as forwarders.dnsmasq_dns_servers = <your dns server address>

We can’t set forwarder per tenant

dhcp_agent.ini

31


LoadBalancer

Swift

Internet


W Z.Z.Z.Z

API N

W API only N

W

10.255.0.2

Change the forwarder in each situation

Use the DNS for API NW as forwarder.

DNSfor API

10.255.0.7

DNSCache

C.C.C.C

dnsmasqneutron

dhcp-agentdnsmasq

neutrondhcp-agent

32


Swift

Internet

API N

W LoadBalancer

Z.Z.Z.Z

API only N

W

10.255.0.2

How to change the forwarder by which external network connected

Prepare the dummy IP that the user access to.which interface isn’t concerned.

DNSfor API

10.255.0.7

C.C.C.CLO-IF

DNSCache

C.C.C.C

dnsmasqneutron

dhcp-agentdnsmasq

neutrondhcp-agent

Add static-route via APIC.C.C.C via 10.255.0.7

Fowarder:C.C.C.C

Fowarder:C.C.C.C

33



Usability ☑ The users don't need to be conscious of the dns ☐ Access the dns without configuring each tenant router


34


Goal of second challenge

ProblemIt’s bothersome to inject the static-route to the router wherever call openstack api via api network

Some solutions● Automate to inject static route● Use the router outside OpenStack ← we adopted

35


Swift

Internet


W

DNSCache

C.C.C.C

The router outside OpenStack mediate

Routing TableC.C.C.C via 10.255.0.7

third party

default gateway is C.C.C.1DNSfor API

C.C.C.CLO-IF

LoadBalancer

Z.Z.Z.Z

API N

W

10.255.0.2

10.255.0.7

RTC.C.C.1

36


Reslove the bothersome to care about static-route

Swift

Internet


W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z10.255.0.2

RT

DNSfor API

10.255.0.7

C.C.C.CLO-IF

Routing TableC.C.C.C via 10.255.0.7

API N

W

37



Usability ☑ The users don't need to be conscious of the dns ☑ Access the dns without configuring each tenant router


38


Goal of third challenge

Problem In order to enable the user to call api via API NW, we need to prepare the dns dedicated to API NW.

Goal Don’t need to maitain extra dns server and extra A records

39


Internet

Swift


W

How to achieve it

Routing Table(static route)C.C.C.C via 10.255.0.7Z.Z.Z.Z via 10.255.0.2 => Routing to LB in the same way as dns

LoadBalancer

Z.Z.Z.Z

API N

W

DNSCache

C.C.C.C10.255.0.7

10.255.0.2

RT

DNSfor API

We don’t need the dns for API.Because the user can access the loadbalancer with Z.Z.Z.Z

40



Usability ☑ The users don't need to be conscious of the dns ☑ Access the dns without configuring each tenant router

Operatinablitiy ☑ Eliminiate the extra dns server

41


Conclusive answer

Swift

Internet


W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z10.255.0.2

RT

10.255.0.7

dnsmasqneutron

dhcp-agentdnsmasq

neutrondhcp-agent

API N

W

42


Resolving name flow from network having internet connectibity

Swift

Internet


W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z10.255.0.2

RT

10.255.0.7

dnsmasqneutron

dhcp-agentdnsmasq

neutrondhcp-agent

API N

W

43


API call flow from network having internet connectibity

Swift

Internet


W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z

API N

W

10.255.0.2

RT

10.255.0.7

dnsmasqneutron

dhcp-agentdnsmasq

neutrondhcp-agent

44


Resolving name flow from private network

Swift


W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z10.255.0.2

RT

10.255.0.7

dnsmasqneutron

dhcp-agentdnsmasq

neutrondhcp-agent

Routing Table(static route)0.0.0.0 via C.C.C.1

forwader: C.C.C.C

API N

W

Routing Table(static route)C.C.C.C via 10.255.0.7

Internet

Query: otb.ft.nttcloud.net

Response:Z.Z.Z.Z

3

1

2

4

5

45


API call flow from private network

Swift

Internet


W

DNSCache

C.C.C.C

LoadBalancer

Z.Z.Z.Z10.255.0.2

RT

10.255.0.7

dnsmasqneutron

dhcp-agentdnsmasq

neutrondhcp-agent

Routing Table(static route)0.0.0.0 via C.C.C.1

Routing Table(static route)Z.Z.Z.Z via 10.255.0.2

API N

W

Access to:Z.Z.Z.Z

1

2

3

46


Demo

47


5. Summary

48


Summary

Achievement● The users can call API without special work from ClosedNW

○ All the users have to do is to plug “API NW”○ Don’t need to change the endpoint url or the ip

How● Add “API NW” as extenal network● Prepare extra router outside OpenStack● Inject static route to extra router

49


Next Challenge: Endpoint Policy

How1. Use Nginx(mod_ruby) + redis

○ Temporal workaround (currently implemented as PoC)

2. Use proactive enforcement in Congress + wsgi filter○ Permanent solution

What

● Restrict the resource the user can access via endpoint○ Container ( Bucket in AWS )

50


Thanks for your attention!

51

How to Provide Restrected Access to Swift … · Servers can only access Swift Amazon Web Services...

Documents

Transcript of How to Provide Restrected Access to Swift … · Servers can only access Swift Amazon Web Services...