...
| Status | ||
|---|---|---|
|
...
...
Objective
Provide a central management API and administrative UI that automates the lifecycle of cloud infrastructure elements.
Success metrics
...
Assumptions
Milestones
...
Glossary
Cloud Foundation
A set of global services that manage the OpenMethods' cloud infrastructure. These services include:
- Cloud Manager
- Docker Registry
- AWS Cloudwatch
- Redundant OAuth Gateways
Shard
An isolated and independent copy of the OpenMethods cloud infrastructure. A Shard contains three types of infrastructure elements:
- Core Components - Application Manager, Redundant Cloud Gateways
- Shared Versioned Components - Web MediaBar, Config Server, etc
- Dedicated Versioned Components - HIS, QueueAdapter, etc
Deployment Ring
A logical grouping of Shared and Dedicated components related to a single major.minor version of a product. A Deployment Ring may contain components located around the world or within a single AWS region depending on the scope and nature of its parent Shard.
Cloud Management VPC
An AWS network structure that is intended to contain all services of the Cloud Foundation. There is a single Cloud Management VPC located in US_EAST_1 (Northern Virginia). All Shard VPCs will have connections and routes to this central Cloud Management VPC.
Shard Management VPC
An AWS network structure that is intended to contain all Core and Shared Versioned Components for a Shard. Each Shard has a single Shard Management VPC located in the home region for the Shard. For instance, the GDPR Shard's Management VPC is located in EU_WEST_3 (Paris).
Shard Service VPC
An AWS network structure that is intended to contain a set of Dedicated Versioned Components for a Shard. A Shard will have any number of Service VPCs that are possibly located around the world. The General Population Shard is meant to service customers around the world and as such, it will have many Service VPCs. A Service VPC will also provide secure access to customer resources through an IPSEC infrastructure provided by Aviatrix.
Requirements
...
| Status | ||||
|---|---|---|---|---|
|
...
Architecture
Cloud Foundation
Shard Management Infrastructure
Deployment Ring Shared Infrastructure
Putting it all Together
AWS Network Design
There are several constraints that must be kept in mind when designing a world wide AWS cloud platform:
...
| Page Properties | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Objective
Provide a central management API and administrative UI that automates the lifecycle of cloud infrastructure elements.
Success metrics
| Goal | Metric |
|---|---|
Assumptions
Milestones
| Roadmap Planner | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Glossary
Cloud Foundation
A set of global services that manage the OpenMethods' cloud infrastructure. These services include:
- Cloud Manager
- Docker Registry
- AWS Cloudwatch
- Redundant OAuth Gateways
Shard
An isolated and independent copy of the OpenMethods cloud infrastructure. A Shard contains three types of infrastructure elements:
- Shared Versioned Components - Web MediaBar, Config Server, etc
- Dedicated Versioned Components - HIS, QueueAdapter, etc
Deployment Ring
A logical grouping of Shared and Dedicated components related to a single major.minor version of a product. A Deployment Ring may contain components located around the world or within a single AWS region depending on the scope and nature of its parent Shard.
Cloud Management VPC
An AWS network structure that is intended to contain all services of the Cloud Foundation. There is a single Cloud Management VPC located in US_EAST_1 (Northern Virginia). All Shard VPCs will have connections and routes to this central Cloud Management VPC.
Shard Management VPC
An AWS network structure that is intended to contain all Core and Shared Versioned Components for a Shard. Each Shard has a single Shard Management VPC located in the home region for the Shard. For instance, the GDPR Shard's Management VPC is located in EU_WEST_3 (Paris).
Shard Service VPC
An AWS network structure that is intended to contain a set of Dedicated Versioned Components for a Shard. A Shard will have any number of Service VPCs that are possibly located around the world. The General Population Shard is meant to service customers around the world and as such, it will have many Service VPCs. A Service VPC will also provide secure access to customer resources through an IPSEC infrastructure provided by Aviatrix.
Requirements
| # | Requirement | User Story | Importance | Jira Issue | Notes | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 |
| ||||||||||
| 2 |
Architecture
Cloud Foundation
Deployment Ring Shared Infrastructure
Putting it all Together
AWS Network Design
There are several constraints that must be kept in mind when designing a world wide AWS cloud platform:
- Limited number of AWS resources (some can be increased, some can’t)
- Avoid networking conflicts between AWS regions as well as customer networks
- Consistent numbering for our various cloud shards (general population, GDPR, GovCloud, etc)
- Ease automated assignment of networks
- Support for two availability zones per region
There are three types of VPCs we will be utilizing: Cloud Management VPC, Shard Management VPCs and Shard Service VPCs. A single Cloud Management VPC holds the centralized cloud management services for the OpenMethods Cloud solution. The Cloud Management VPC will occupy the 10.250.0.0/16 address space. Each cloud shard will have a single Shard Management VPC that hosts the common, centralized components that are shared between customers that reside in the shard. An initial Class B network is dedicated to each shard. The Shard Management VPC occupies the lower 8 Class C networks, which are split between two availability zones for redundancy. The remaining address space is used for Shard Service VPCs to contain instances. Additional Class B networks can be assigned to a Shard if more address space is needed.
| Shard Networking | ||||
|---|---|---|---|---|
Shard Id | Base CIDR | Shard Management VPC CIDR | Availability Zone A Subnet | Availability Zone B Subnet |
1 – Development | 10.249.0.0/16 | 10.249.0.0/21 | 10.249.0.0/22 (1019 hosts) | 10.249.4.0/22 |
2 – Gen Pop | 10.248.0.0/16 | 10.248.0.0/21 | 10.248.0.0/22 | 10.248.4.0/22 |
3 – GDPR | 10.247.0.0/16 | 10.247.0.0/21 | 10.247.0.0/22 | 10.247.4.0/22 |
4 - Canada? | 10.246.0.0/16 | 10.246.0.0/21 | 10.246.0.0/22 | 10.246.4.0/22 |
| 5 - GovCloud | 10.245.0.0/16 | 10.245.0.0/21 | 10.245.0.0/22 | 10.245.4.0/22 |
00000000 00000000 00001000 00000000
The benefit of this approach allows you to tell that the server is in a Shard Management VPC and which shard it’s a member of just based on the server’s IP address. An address that falls within 10.249.0.0-10.249.7.255 will always be a Shard Management VPC and the range of the third number will tell you which Shard Service VPC it belongs to, i.e 10.249.8.0 belongs to Shard 1 North Virginia.
The second type of VPC we will use is a Shard Service VPC. Shard Service VPCs will host the services related to specific customers and the IPSec connectivity needed to communicate with any on-premise CTI or data endpoints. Each shard will have any number of service VPCs distributed around the globe. As with Shard Management VPCs, Shard Service VPCs typically belong to a single cloud shard and are not shared. However, in the case of internal development, QA, and training environments the underlying VPCs and services are shared between these logical Shards. The full service VPC address space is laid out below with grey rows not being implemented today but may be in the future:
| Geography | AWS Region | Service VPC Id | Service VPC CIDR | Availability Zone A Subnet | Availability Zone B Subnet |
|---|---|---|---|---|---|
| 1 – Dev/QA/Train | N. Virginia (us-east-1) | 1 | 10.249.8.0/21 | 10.249.8.0/22 (1019 hosts) | 10.249.12.0/22 |
Mumbai (ap-south-1) | 2 | 10.249.16.0/21 | 10.249.16.0/22 | 10.249.20.0/22 | |
| 2 - General Population | N. Virginia (us-east-1) | 1 | 10.248.8.0/21 | 10.248.8.0/22 (1019 hosts) | 10.248.12.0/22 |
| Singapore (ap-southeast-1) | 2 | 10.248.16.0/21 | 10.248.16.0/22 | 10.248.20.0/22 | |
| N California (us-west-1) | 3 | 10.248.24.0/21 | 10.248.24.0/22 | 10.248.28.0/22 | |
| Ohio (us-east-2) | 4 | 10.248.32.0/21 | 10.248.32.0/22 | 10.248.36.0/22 | |
| 3 - GDPR | Paris (eu-west-3) | 1 | 10.247.8.0/21 | 10.247.8.0/22 (1019 hosts) | 10.247.12.0/22 |
| 4 - Canada | N. Virginia (us-east-1) | 1 | 10.246.8.0/21 | 10.246.8.0/22 (1019 hosts) | 10.246.12.0/22 |
| 5 - XYZ |
User interaction and design
Create Shard
- An adminstrator logs in
Open Questions
| Question | Answer | Date Answered |
|---|---|---|
Out of Scope
Deprecated
| Info |
|---|
The cloud networking has undergone a significant rework. The following is the original direction that has been abandoned |
There are three types of VPCs we will be utilizing: Cloud Management VPC, Shard Management VPCs and Shard Service VPCs. A single Cloud Management VPC holds the centralized cloud management services for the OpenMethods Cloud solution. Each cloud shard will have a single Shard Management VPC that hosts the common, centralized components that are shared between customers that reside in the shard. These Shard Management VPCs will share the 10.255.0.0/16 address space. The first (left to right) three bits of the third byte will be used to indicate which of 8 possible shards the Shard Management VPC belongs to. The forth bit is used to subnet between the two availability zones used for redundancy. The remaining address space is used for instances within the VPC and availability zone subnets. Here’s the table of shard prefixes in binary:
Shard Id | Bit Prefix | Core VPC CIDR | Availability Zone A Subnet | Availability Zone B Subnet |
|---|---|---|---|---|
1 – Development | 000 | 10.255.0.0/19 | 10.255.0.0/20 (4091 hosts) | 10.255.16.0/20 |
2 – Gen Pop | 001 | 10.255.32.0/19 | 10.255.32.0/20 | 10.255.48.0/20 |
3 – GDPR | 010 | 10.255.64.0/19 | 10.255.64.0/20 | 10.255.80.0/20 |
4 - GovCloud? | 011 | 10.255.96.0/19 | 10.255.96.0/20 | 10.255.112.0/20 |
5 | 100 | 10.255.128.0/19 | 10.255.128.0/20 | 10.255.144.0/20 |
6 | 101 | 10.255.160.0/19 | 10.255.160.0/20 | 10.255.176.0/20 |
7 | 110 | 10.255.192.0/19 | 10.255.192.0/20 | 10.255.208.0/20 |
8 | 111 | 10.255.224.0/19 | 10.255.224.0/20 | 10.255.240.0/20 |
The benefit of this approach allows you to tell that the server is in a Shard Management VPC and which shard it’s a member of just based on the server’s IP address. An address that starts with 10.255.x.x will always be a Shard Management VPC and the range of the third number will tell you which shard the Shard Management VPC belongs to, i.e 10.255.0.0 – 10.255.31.255 belongs to shard 1.
...
For example, we consider Canada, US, and Mexico to be a single area called North America, while AWS has multiple regions within this area. To support the multi-dimensional deployments of Service VPCs, we will use a slightly different network addressing scheme. Each geographical region will have one (or more as we expand) /16 address space used across the service vpcs in that area:
Geographical Area | Base CIDR | Expansion CIDR |
|---|---|---|
North America | 10.254.0.0/16 | 10.249.0.0/16 |
EMEA | 10.253.0.0/16 | 10.248.0.0/16 |
APAC | 10.252.0.0/16 | 10.247.0.0/16 |
Oceana | 10.251.0.0/16 | 10.246.0.0/16 |
South America | 10.250.0.0/16 | 10.245.0.0/16 |
As with Shard Management VPCs, Shard Service VPCs belong to a single cloud shard and are not shared. The ownership of the Service VPC is designated by the first (left to right) three bits of the third byte. Unlike Core VPCs, the address space needs to be divided up further to allow multiple Service VPCs to be created and to limit the size of the VPC subnets to eliminate customer overlap where possible. This is handled by using the 4th bit of the third octet to identify the Service VPC specifically. The fifth bit is used to segment off the availability zone subnet space. The full service VPC address space is laid out below with grey rows not being implemented today but may be in the future:
| Geography | Shard Id | Shard Prefix | Service VPC Id | Service VPC Prefix | Service VPC CIDR | Availability Zone A Subnet | Availability Zone B Subnet |
|---|---|---|---|---|---|---|---|
| North America | 1 – Development | 000 | 1 | 0 | 10.254.0.0/20 | 10.254.0.0/21 (2043 hosts) | 10.254.8.0/21 |
2 | 1 | 10.254.16.0/20 | 10.254.16.0/21 | 10.254.24.0/21 | |||
2 - General Population | 001 | 1 | 0 | 10.254.32.0/20 | 10.254.32.0/21 | 10.254.40.0/21 | |
2 | 1 | 10.254.48.0/20 | 10.254.48.0/21 | 10.254.56.0/21 | |||
3 - GDPR | 010 | 1 | 0 | 10.254.64.0/20 | 10.254.64.0/21 | 10.254.72.0/21 | |
2 | 1 | 10.254.80.0/20 | 10.254.80.0/21 | 10.254.88.0/21 | |||
4 - Gov Cloud | 011 | 1 | 0 | 10.254.96.0/20 | 10.254.96.0/21 | 10.254.104.0/21 | |
2 | 1 | 10.254.112.0/20 | 10.254.112.0/21 | 10.254.120.0/21 | |||
| EMEA | 1 – Development | 000 | 1 | 0 | 10.253.0.0/20 | 10.253.0.0/21 (2043 hosts) | 10.253.8.0/21 |
| 2 | 1 | 10.253.16.0/20 | 10.253.16.0/21 | 10.253.24.0/21 | |||
| 2 - General Population | 001 | 1 | 0 | 10.253.32.0/20 | 10.253.32.0/21 | 10.253.40.0/21 | |
| 2 | 1 | 10.253.48.0/20 | 10.253.48.0/21 | 10.253.56.0/21 | |||
| 3 - GDPR | 010 | 1 | 0 | 10.253.64.0/20 | 10.253.64.0/21 | 10.253.72.0/21 | |
| 2 | 1 | 10.253.80.0/20 | 10.253.80.0/21 | 10.253.88.0/21 | |||
| 4 - Gov Cloud | 011 | 1 | 0 | 10.253.96.0/20 | 10.253.96.0/21 | 10.253.104.0/21 | |
| 2 | 1 | 10.253.112.0/20 | 10.253.112.0/21 | 10.253.120.0/21 | |||
| APAC | 1 – Development | 000 | 1 | 0 | 10.252.0.0/20 | 10.252.0.0/21 (2043 hosts) | 10.252.8.0/21 |
| 2 | 1 | 10.252.16.0/20 | 10.252.16.0/21 | 10.252.24.0/21 | |||
| 2 - General Population | 001 | 1 | 0 | 10.252.32.0/20 | 10.252.32.0/21 | 10.252.40.0/21 | |
| 2 | 1 | 10.252.48.0/20 | 10.252.48.0/21 | 10.252.56.0/21 | |||
| 3 - GDPR | 010 | 1 | 0 | 10.252.64.0/20 | 10.252.64.0/21 | 10.252.72.0/21 | |
| 2 | 1 | 10.252.80.0/20 | 10.252.80.0/21 | 10.252.88.0/21 | |||
| 4 - Gov Cloud | 011 | 1 | 0 | 10.252.96.0/20 | 10.252.96.0/21 | 10.252.104.0/21 | |
| 2 | 1 | 10.252.112.0/20 | 10.252.112.0/21 | 10.252.120.0/21 | |||
| Oceana | 1 – Development | 000 | 1 | 0 | |||
| 2 | 1 | ||||||
| 2 - General Population | 001 | 1 | 0 | ||||
| 2 | 1 | ||||||
| 3 - GDPR | 010 | 1 | 0 | ||||
| 2 | 1 | ||||||
| 4 - Gov Cloud | 011 | 1 | 0 | ||||
| 2 | 1 | ||||||
| South America | 1 – Development | 000 | 1 | 0 | |||
| 2 | 1 | ||||||
| 2 - General Population | 001 | 1 | 0 | ||||
| 2 | 1 | ||||||
| 3 - GDPR | 010 | 1 | 0 | ||||
| 2 | 1 | ||||||
| 4 - Gov Cloud | 011 | 1 | 0 | ||||
| 2 | 1 |
The vast majority of the time there will only be 1 Service VPC in a given geographical area associated with a shard. As with Core VPCs, you’ll be able to tell which geographical region based on the second octet. The associated shard can be determined by looking at the range of the third octet, 10.254.0.0 – 10.254.31.255 indicates shard 1. The Service VPC is determined by the sub range, 10.254.0.0 – 10.254.15.255 for Service VPC 1. The availability zone can be determined by high or low values of the third octet within the VPC range.
User interaction and design
Create Shard
- An adminstrator logs in
Open Questions
...