[Documentation] Node Metadata Manager design #209

MAC Manager Design Specification

Overview

MAC manager is an Alcor microservice which manages the lifecycle of all MAC addresses. Its responsibilities include but not limited to, allocating a unique mac address for a given port, releasing the mac address when not needed, and onboarding a region-specific mac ranges. It interacts with Subnet Manager and Port Manager to allocate mac addresses for gateway ports and customer ports.

Service Requirements

Allocate virtual MAC address for all the tenants/vpcs, e.g., (VPC 1, Port 1) → MAC1
Release (port, MAC) association when a port is released
Guarantee uniqueness of MAC address in region scope. Each region has seperate Alcor controller.
Algorithm with random, but less conflict possibility
Reserved MAC address by IANA should be avoided [1]
Fall back mechanism if there is a conflict among MAC addresses
Allow onboarding a list of MAC ranges for a region
Delete a list of MAC ranges for a region.
APIs to activate and deactivate a MAC address. MAC address has 3 states such as "allocated & activated", "allocated & deactivated", and "free (= not allocated yet)"
Working together with other services including subnet manager and port manager.
Concurrency control mechanism should be provided to process multiple concurrent MAC allocation requests.

REST APIs

API Snapshot

API Name Method Request Response

Verify MAC State

GET

/macs/{mac}, /v4/macs/{mac}

MAC state [sample]

Create MAC State

POST

/macs, /v4/macs

MAC state [sample]

Activate MAC Address

PUT

/macs/{mac}, /v4/macs/{mac}

MAC state [sample]

Deactivate MAC Address

PUT

/macs/{mac}, /v4/macs/{mac}

MAC state [sample]

Release MAC State

DELETE

/macs/{mac}, /v4/macs/{mac}

MAC state [sample]

Get MAC Range

GET

/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}

MAC range state [sample]

List MAC Ranges

GET

/macs/ranges, /v4/macs/ranges

MAC range states [sample]

Create MAC Range

POST

/macs/ranges, /v4/macs/ranges

MAC range state [sample]

Update MAC Range

PUT

/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}

MAC range state [sample]

Delete MAC Range

DELETE

/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}

deleted MAC range id [sample]

API Name	Method	Request	Response
Verify/Get MAC State	GET	/macs/{mac}, /v4/macs/{mac}	MAC state [sample]
Create MAC State	POST	/macs, /v4/macs	MAC state [sample]
Create MAC State From Range	POST	/macs/range/{rangeid}, /v4/macs/range/{rangeid}	MAC state [sample]
Activate MAC Address	PUT	/macs/{mac}, /v4/macs/{mac}	MAC state [MacState_Put1]
Deactivate MAC Address	PUT	/macs/{mac}, /v4/macs/{mac}	MAC state [sample]
Release MAC State	DELETE	/macs/{mac}, /v4/macs/{mac}	MAC state [sample]
Get MAC Range	GET	/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}	MAC range state [sample]
List MAC Ranges	GET	/macs/ranges, /v4/macs/ranges	MAC range states [sample]
Create MAC Range	POST	/macs/ranges, /v4/macs/ranges	MAC range state [sample]
Update MAC Range	PUT	/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}	MAC range state [sample]
Delete MAC Range	DELETE	/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}	deleted MAC range id [sample]

API Specification

(1) Get/Verify MAC state by MAC address

Method: GET
Request: /macs/{mac}, /v4/macs/{mac}
Request Parameter: @PathVariable String mac
Response: mac state
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/macs/00-14-2A-3B-47-37

Response:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": "f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "mac": "00-14-2A-3B-47-37",
    “state” “Active”
  }
}

~~(2) Allocate MAC Address for Port~~(2) Allocate MAC Address

Method: POST
Request: "/macs", "/v4/macs"
Request Parameter:
Response: mac state
Normal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example

Request:
http://127.0.0.1:8080/macs

Body:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df"
    "mac": “null”,
    “state”: “null”
  }
}

Response:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "mac": "00-14-2A-3B-47-37",
    “state” “Active”
  }
}

(3) Allocate MAC Address From MAC Range

Method: POST
Request: "/macs/range/{rangeid}", "/v4/macs/{rangeid}"
Request Parameter: @PathVariable String rangeid
Response: mac state
Normal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example

Request:
http://127.0.0.1:8080/macs

Body:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df"
    "mac": “null”,
    “state”: “null”
  }
}

Response:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "mac": "00-14-2A-3B-47-37",
    “state” “Active”
  }
}

~~(3) Activate MAC Address~~(4) Activate MAC Address

Method: PUT
Request: /macs/{mac}", "/v4/macs/{mac}
Request Parameter: @PathVariable String mac
Response: mac state
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/macs/00-14-2A-3B-47-37

Body:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df"
    "mac": “00-14-2A-3B-47-37”,
    “state”: “Active”
  }
}

Response:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": "f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "mac": "00-14-2A-3B-47-37",
    “state”: “Active”
  }
}

~~(4) Deactivate MAC Address~~(5) Deactivate MAC Address

Method: PUT
Request: /macs/{mac}", "/v4/macs/{mac}
Request Parameter: @PathVariable String mac
Response: mac state
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/macs/00-14-2A-3B-47-37

Body:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df"
    "mac": “00-14-2A-3B-47-37”,
    “state”: “Inactive”
  }
}

Response:
{
  "mac_state": {
    "project_id": "3dda2801-d675-4688-a63f-dcda8d327f50",
    "vpc_id": "9192a4d4-ffff-4ece-b3f0-8d36e3d88038",
    "port_id": "f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "mac": "00-14-2A-3B-47-37",
    “state”: “Inactive”
  }
}

~~(5) Delete/Release MAC State By MAC Address~~(6) Delete/Release MAC State By MAC Address

Method: DELETE
Request: /macs/{mac}", "/v4/macs/{mac}
Request Parameter: @PathVariable String mac
Response: deleted MAC address
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/mac/{00-14-2A-3B-47-37}

Response:
{
  "mac": "00-14-2A-3B-47-37"
}

~~(6) Get MAC Range by MAC Range Id~~(7) Get MAC Range by MAC Range Id

Method: GET
Request: /macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}
Request Parameter: @PathVariable String rangeid
Response: mac range state
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/macs/ranges/range0

Response:
{
  "mac_range": {
    “range_id": "range0”,
    "from": “00-14-2A-00-00-00”,
    "to": “00-14-2A-FF-FF-FF”,
    "state": "Active"
   }
}

~~(7) List All MAC Ranges~~(8) List All MAC Ranges

Method: GET
Request: /macs/ranges, /v4/macs/ranges
Request Parameter:
Response: mac range state list
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/macs/ranges

Response:
{"mac_ranges":
[
    {"range_id":"range0",
    "from":"00-AA-BB-AA-AA-00",
    "to":"00-AA-BB-BB-BB-00",
    "state":"Inactive"},
    {"range_id":"range1",
    "from":"00-AA-BB-00-00-00",
    "to":"00-AA-BB-FF-FF-FF",
    "state":"Active"}
]}

~~(8) Onboard MAC Range~~(9) Onboard MAC Range

Method: POST
Request: /macs/ranges
Request Parameter:
Response: mac range state
Normal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example

Request:
http://127.0.0.1:8080/macs/ranges

Body:
{
  "mac_range": {
    “range_id": "range1”,
    "from": “00-14-2A-3B-47-00”,
    "to": “00-14-2A-3B-47-FF”,
    "state": "Active"
   }
}

Response:
{
  “mac_range”: {
    “id": "range1”,
    "from": “00-14-2A-3B-47-00”,
    "to": “00-14-2A-3B-47-FF”,
    "state": "Active"
   }
}

~~(9) Update MAC Range~~(10) Update MAC Range

Method: PUT
Request: /macs/ranges/rangeid
Request Parameter:
Response: mac range state
Normal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example

Request:
http://127.0.0.1:8080/macs/ranges/range1

Body:
{
    "mac_range": {
    “range_id": "range1”,
    "from": “00-14-2A-3B-47-11”,
    "to": “00-14-2A-3B-47-FF”,
    "state": "Inactive"
    }
}

Response:
{
    “mac_range”: {
    “range_id": "range1”,
    "from": “00-14-2A-3B-47-11”,
    "to": “00-14-2A-3B-47-FF”,
    "state": "Inactive"
    }
}

~~(10) Remove MAC Range by Range Id~~(11) Remove MAC Range by Range Id

Method: DELETE
Request: /macs/ranges/range1
Request Parameter:
Response: mac range id
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/macs/ranges/range1

Response:
{
  "range_id": "range1"
}

Database Data Schema

MAC Address State

One MAC address falls into one of three states:

State	Details
Active	MAC address is allocated to a port AND it is in use. This is default.
Inactive	MAC address is allocated to a port AND it is NOT in use.
Free	MAC address is NOT allocated to a port yet.

MAC Range

MAC State

Figure 1. MAC data schema and management

Concurrency Handling

To handle concurrent MAC allocation requests, the following three things should be considered:

(1) Performance

MAC allocation time per request should be reasonable even if there are multiple simultaneous requests. For this purpose, this system performs MAC creation and MAC allocation processes in parallel using MAC pool as shown in [Figure 1]. In other words, it creates MAC addresses and stores them in MAC pool in advance. When there are MAC allocation requests, MAC pool provides pre-generated MAC addresses. During request processing, MAC creation keeps generating new MAC addresses.

(2) Uniqueness

Allocated MAC addresses should be unique in spite of concurrent MAC allocation requests. MAC manager achieves this goal by checking in two ways. First, it checks MAC address uniqueness when it creates a new MAC address. Second, it confirms the uniqueness when it allocates a MAC to a port. Additionally, MAC allocation is stored in a hash table whose key is MAC address. Therefore, the system can detect duplicated MAC addresses.

(3) Fallback

In case the uniqueness check for MAC address fails, it allocates new MAC addresses to deactivated ports first. If ports are active, it reallocates new MAC addresses to them until the conflict is resolved.

Comparison with OpenStack Neutron

Neutron creates a random MAC address when it creates port. There are 3 options to create a MAC address.

(1) Non DVR

When network is configured as non-distributed virtual router by default, Neutron uses 6 octets of base MAC address for VIFs. The first 3 octets of the base MAC address will remain unchanged. If the 4th octet is not 00, it will also be used. The other octets will be randomly generated [2].

enable_distributed_routing=False, base_mac = fa:16:3e:00:00:00

(2) Distributed virtual router (enable_distributed_routing=True)

Every unique DVR those are instantiated by Neutron uses unique base MAC address, which is called ‘dvr_base_mac’. Similar to non DVR, the first 3 octets are unchanged. If the 4th octet is not 00, it is unchanged. The other octets are randomly generated. The ‘dvr_base_mac’ must be different from ‘base_mac’ to avoid mixing them up with MACs allocated for tenant ports [2].

dvr_base_mac = fa:16:3f:00:00:00

(3) Static MAC address assigned by admin

When it creates new port, admin can assign MAC address along with other port information using the following option. Only admin can use this option because updating MAC can cause IP address loss [3].

openstack port create --mac-address <mac-address> MAC address of this port (admin only)

References

Node Manager - Meta Data Management

Table of Contents

Overview

The node manager is a control plane component responsible of managing and configurating physical servers/nodes/hosts inside a data center. It maintains the detailed configuration for each node and collect its health state continuously in a configurable interval. The node health states could be used collectively in many user scenarios, for example, determine if a deployment of network resource (for example, port) is successful, or if a load balancing backend is reachable.

User Cases

Port manager talks to node manager to pull node details from a node id (UUID).

Service Requirements

Basic Requirements

Manage mapping from node id to node info (including IP, mac, veth)
Populating the node mapping in the controller launch time (through a configuration file) during new region or data center buildout
Expose a CURD API to register, update, or delete a node.
Define a health model, data schema and APIs for node health
Working together with other services including port manager.
Require HA and high scalability as 100,000+ nodes could upload health data continuously.

Advanced Requirements

Node info extensible to more node resources including FPGA etc.
Fall back mechanism when a node state is stall (e.g. proactively pull instead of waiting for agent to push)

REST APIs

API Snapshot

API Name	Method	Request	Response
Get Node Info.	GET	/nodes/{nodeid}, /v4/nodes/{nodeid}	Node Information [sample]
List All Nodes	GET	/nodes, /v4//nodes	All Nodes [sample]
Create Node	POST	/nodes, /v4/nodes	Node Information [sample]
Create Nodes Bulk	POST	/nodes/upload, /v4/nodes/upload	Tottal Number of Created Nodes [sample]
Update Node	PUT	/nodes/{nodeid}, /v4/nodes/{nodeid}	Node Information [sample]
Delete Node	DELETE	/nodes/{nodeid}, /v4/nodes/{nodeid}	Node Id [NodeInfo_Delete]

API Specification

(1) Get Node Info By Node Id

Method: GET
Request: /nodes/{nodeid}, /v4/nodes/{nodeid}
Request Parameter: @PathVariable String nodeid
Response: node information
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/nodes/f37810eb-7f83-45fa-a4d4-1b31e75399df

Response:
{
  "node": {
    "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "name": "node1",
    "ip": "10.0.0.2",
    "mac": "00-14-2A-3B-47-37",
    "veth": "veth0",
    "GRPC": "9090",
    "status": "running”
  }
}

(2) Get All Nodes

Method: GET
Request: /nodes/", "/v4/nodes/
Request Parameter:
Response: list of all nodes
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/nodes/f37810eb-7f83-45fa-a4d4-1b31e75399df

Response:
{[{
"node": {
"node_id": "f37810eb-7f83-45fa-a4d4-1b31e75399df",
"name": "node1",
"ip": "10.0.0.3",
"mac": "00-14-2A-3B-47-37",
"veth": "veth1",
"GRPC": "9090",
"status": “stopped”
}
},
{[{
"node": {
"node_id": " c57510eb-bf73-45fa-b2d6-3ccce75defee",
"name": "node2",
"ip": "10.0.0.5",
"mac": "00-14-2A-3B-AC-AC",
"veth": "veth5",
"GRPC": "9090",
"status": “stopped”
}
}]
}

(3) Create Node

Method: POST
Request: "/nodes", "/v4/nodes"
Request Parameter:
Response: node information
Normal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example

Request:
http://127.0.0.1:8080/macs

Body:
{
  "node": {
    "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "name": "node1",
    "ip": "10.0.0.2",
    "mac": "00-14-2A-3B-47-37",
    "veth": "veth0",
    "GRPC": "9090",
    "status": “”
  }
}

Response:
{
  "node": {
    "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "name": "node1",
    "ip": "10.0.0.2",
    "mac": "00-14-2A-3B-47-37",
    "veth": "veth0",
    "GRPC": "9090",
    "status": “stopped”
  }
}

(4) Create Nodes From File Upload

Method: POST
Request: "/nodes/upload", "/v4/nodes/upload"
Request Parameter: @RequestParam MultipartFile file
Response: total number of created nodes
Normal response codes: 201
Error response codes: 400, 409, 412, 500, 503
Example

Request:
http://127.0.0.1:8080/nodes/upload

RequestParameters:
{
[file="@/mnt/alcor/config/machine.json"]
}

Response:
{
  "total nodes": 100
}

(5) Update Node By Node Id

Method: PUT
Request: /nodes/{nodeid}", "/v4/nodes/{nodeid}
Request Parameter: @PathVariable String nodeid
Response: updated node information
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/macs/00-14-2A-3B-47-37

Body:
http://127.0.0.1:8080/nodes/f37810eb-7f83-45fa-a4d4-1b31e75399df
Body:
{
  "node": {
    "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "name": "node1",
    "ip": "10.0.0.3",
    "mac": "00-14-2A-3B-47-37",
    "veth": "veth1",
    "GRPC": "9090",
    "status": “stopped”
  }
}

Response:
{
  "node": {
    "node_id": " f37810eb-7f83-45fa-a4d4-1b31e75399df",
    "name": "node1",
    "ip": "10.0.0.3",
    "mac": "00-14-2A-3B-47-37",
    "veth": "veth1",
    "GRPC": "9090",
    "status": “stopped”
  }
}

(6) Delete Node

Method: DELETE
Request: /nodes/{nodeid}", "/v4/nodes/{nodeid}
Request Parameter: @PathVariable String nodeid
Response: deleted node id
Normal response codes: 200
Error response codes: 400, 412, 500
Example

Request:
http://127.0.0.1:8080/nodes/f37810eb-7f83-45fa-a4d4-1b31e75399df

Response:
{
  "node": " f37810eb-7f83-45fa-a4d4-1b31e75399df"
}

Database Data Schema

Node Info

Node info. contains meta data information of nodes.

node_id	name	ip	mac	veth	gRPC_server_port
f37810eb	node1	10.0.0.2	AA-BB-CC-11-22-33	veth0	9090

System Structure

Figure 1. System structure of node meta data manager

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Documentation] Node Metadata Manager design #209

Uh oh!

Diff view

Diff view

There are no files selected for viewing

MAC Manager Design Specification

Overview

Service Requirements

REST APIs

API Snapshot

API Specification

Database Data Schema

MAC Address State

MAC Range

MAC State

Concurrency Handling

(1) Performance

(2) Uniqueness

(3) Fallback

Comparison with OpenStack Neutron

(1) Non DVR

(2) Distributed virtual router (enable_distributed_routing=True)

(3) Static MAC address assigned by admin

References

Node Manager - Meta Data Management

Overview

User Cases

Service Requirements

Basic Requirements

Advanced Requirements

REST APIs

API Snapshot

API Specification

Database Data Schema

Node Info

System Structure

References

API Name	Method	Request	Response
Verify MAC State	GET	/macs/{mac}, /v4/macs/{mac}	MAC state [sample]
Create MAC State	POST	/macs, /v4/macs	MAC state [sample]
Activate MAC Address	PUT	/macs/{mac}, /v4/macs/{mac}	MAC state [sample]
Deactivate MAC Address	PUT	/macs/{mac}, /v4/macs/{mac}	MAC state [sample]
Release MAC State	DELETE	/macs/{mac}, /v4/macs/{mac}	MAC state [sample]
Get MAC Range	GET	/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}	MAC range state [sample]
List MAC Ranges	GET	/macs/ranges, /v4/macs/ranges	MAC range states [sample]
Create MAC Range	POST	/macs/ranges, /v4/macs/ranges	MAC range state [sample]
Update MAC Range	PUT	/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}	MAC range state [sample]
Delete MAC Range	DELETE	/macs/ranges/{rangeid}, /v4/macs/ranges/{rangeid}	deleted MAC range id [sample]

[Documentation] Node Metadata Manager design #209

Are you sure you want to change the base?

Uh oh!

[Documentation] Node Metadata Manager design #209

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

MAC Manager Design Specification

Overview

Service Requirements

REST APIs

API Snapshot

API Specification

Database Data Schema

MAC Address State

MAC Range

MAC State

Concurrency Handling

(1) Performance

(2) Uniqueness

(3) Fallback

Comparison with OpenStack Neutron

(1) Non DVR

(2) Distributed virtual router (enable_distributed_routing=True)

(3) Static MAC address assigned by admin

References

Node Manager - Meta Data Management

Overview

User Cases

Service Requirements

Basic Requirements

Advanced Requirements

REST APIs

API Snapshot

API Specification

Database Data Schema

Node Info

System Structure

References