-
Notifications
You must be signed in to change notification settings - Fork 220
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixes #34205 - External IPAM Integration #810
Open
grizzthedj
wants to merge
1
commit into
theforeman:develop
Choose a base branch
from
grizzthedj:external-ipam
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,6 +10,7 @@ Ashley Penney <[email protected]> | |
Baptiste Agasse <[email protected]> | ||
Brandon Weeks <[email protected]> | ||
Christian Arnold <[email protected]> | ||
Christopher Smith <[email protected]> | ||
Corey Osman <[email protected]> | ||
Daniel Baeurer <[email protected]> | ||
Daniel Helgenberger <[email protected]> | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
--- | ||
:enabled: false | ||
|
||
# Built-in providers: | ||
# 1. phpIPAM: externalipam_phpipam | ||
# 2. Netbox: externalipam_netbox | ||
|
||
:use_provider: externalipam_netbox | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
# url is the hostname and path of the Netbox instance | ||
:url: 'https://netbox.example.com' | ||
|
||
# token is the Netbox API token | ||
:token: 'netbox_token' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
--- | ||
# url is the hostname and path of the phpIPAM instance. | ||
:url: 'https://phpipam.example.com' | ||
|
||
# The phpIPAM user name for authentication. Please note that an API Key also needs to be | ||
# setup with the exact same name as the user name configured here. When setting up the API | ||
# Key in phpIPAM, "User token" must be used for the "App Security" setting. | ||
:user: 'ipam_user' | ||
|
||
# The password for above user account. Note that this is the password of the user, and not | ||
# the API Key itself. | ||
:password: 'ipam_password' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
require 'yaml' | ||
require 'json' | ||
require 'net/http' | ||
require 'uri' | ||
require 'externalipam/ipam_helper' | ||
|
||
module Proxy::Ipam | ||
# Class to handle authentication and HTTP transactions with External IPAM providers | ||
class ApiResource | ||
grizzthedj marked this conversation as resolved.
Show resolved
Hide resolved
|
||
include ::Proxy::Log | ||
include Proxy::Ipam::IpamHelper | ||
|
||
def initialize(params = {}) | ||
@api_base = params[:api_base] | ||
@token = params[:token] | ||
@auth_header = params[:auth_header] || 'Authorization' | ||
end | ||
|
||
def get(path, params = nil) | ||
url = @api_base + path | ||
url += "?#{URI.encode_www_form(params)}" if params | ||
uri = URI(url) | ||
request = Net::HTTP::Get.new(uri) | ||
request[@auth_header] = @token | ||
request['Accept'] = 'application/json' | ||
request(request, uri) | ||
end | ||
|
||
def delete(path) | ||
uri = URI(@api_base + path) | ||
request = Net::HTTP::Delete.new(uri) | ||
request[@auth_header] = @token | ||
request['Accept'] = 'application/json' | ||
request(request, uri) | ||
end | ||
|
||
def post(path, body = nil) | ||
uri = URI(@api_base + path) | ||
request = Net::HTTP::Post.new(uri) | ||
request.body = body | ||
grizzthedj marked this conversation as resolved.
Show resolved
Hide resolved
|
||
request[@auth_header] = @token | ||
request['Accept'] = 'application/json' | ||
request['Content-Type'] = 'application/json' | ||
request(request, uri) | ||
end | ||
|
||
private | ||
|
||
def request(request, uri) | ||
Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == 'https') do |http| | ||
http.request(request) | ||
end | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
module ::Proxy::Ipam | ||
class ConfigurationLoader | ||
def load_classes | ||
require 'externalipam/dependency_injection' | ||
require 'externalipam/ipam_api' | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
module Proxy::Ipam | ||
module DependencyInjection | ||
include Proxy::DependencyInjection::Accessors | ||
def container_instance | ||
@container_instance ||= ::Proxy::Plugins.instance.find { |p| p[:name] == :externalipam }[:di_container] | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
require 'externalipam/externalipam_plugin' | ||
require 'externalipam/configuration_loader' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
require 'externalipam/phpipam/phpipam_plugin' | ||
require 'externalipam/netbox/netbox_plugin' | ||
|
||
module Proxy::Ipam | ||
class Plugin < ::Proxy::Plugin | ||
plugin :externalipam, ::Proxy::VERSION | ||
uses_provider | ||
default_settings use_provider: nil | ||
rackup_path File.expand_path('http_config.ru', __dir__) | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
require 'externalipam/ipam_api' | ||
|
||
map '/ipam' do | ||
run Proxy::Ipam::Api | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
require 'yaml' | ||
require 'json' | ||
require 'monitor' | ||
require 'concurrent' | ||
require 'time' | ||
require 'externalipam/ipam_helper' | ||
require 'singleton' | ||
|
||
module Proxy::Ipam | ||
# Class for managing temp in-memory cache to prevent same IP's being suggested in race conditions | ||
class IpCache | ||
include Singleton | ||
include Proxy::Log | ||
include Proxy::Ipam::IpamHelper | ||
|
||
DEFAULT_CLEANUP_INTERVAL = 180 | ||
|
||
def initialize | ||
@m = Monitor.new | ||
@ip_cache = {'': {}} | ||
start_cleanup_task | ||
end | ||
|
||
def provider_name(provider) | ||
@provider = provider | ||
end | ||
|
||
def get_cidr(group_name, cidr) | ||
@ip_cache.dig(group_name, cidr) | ||
end | ||
|
||
def get_ip(group_name, cidr, mac) | ||
@ip_cache.dig(group_name, cidr, mac, :ip) | ||
end | ||
|
||
def ip_exists?(group_name, cidr, ip) | ||
subnet_hash = get_cidr(group_name, cidr) | ||
return false if subnet_hash.nil? | ||
subnet_hash&.any? { |mac, cached_ip| cached_ip[:ip] == ip } | ||
end | ||
|
||
def ip_expired?(group_name, cidr, ip) | ||
return true unless ip_exists?(group_name, cidr, ip) | ||
subnet_hash = get_cidr(group_name, cidr) | ||
subnet_hash&.any? { |mac, cached_ip| cached_ip[:ip] == ip && expired(cached_ip[:timestamp]) } | ||
end | ||
|
||
def cleanup_interval | ||
DEFAULT_CLEANUP_INTERVAL | ||
end | ||
|
||
def add(group_name, cidr, ip, mac = nil) | ||
logger.debug("Adding IP '#{ip}' to cache for subnet '#{cidr}' in group '#{group_name}' for IPAM provider #{@provider}") | ||
@m.synchronize do | ||
mac_addr = mac.nil? || mac.empty? ? SecureRandom.uuid : mac | ||
grizzthedj marked this conversation as resolved.
Show resolved
Hide resolved
|
||
group_hash = @ip_cache[group_name] | ||
|
||
if group_hash&.key?(cidr) | ||
@ip_cache[group_name][cidr][mac_addr] = { ip: ip.to_s, timestamp: Time.now } | ||
else | ||
@ip_cache[group_name] = { cidr => { mac_addr => { ip: ip.to_s, timestamp: Time.now }}} | ||
end | ||
end | ||
end | ||
|
||
private | ||
|
||
def expired(ip_expiration) | ||
Time.now - ip_expiration > DEFAULT_CLEANUP_INTERVAL | ||
end | ||
|
||
def start_cleanup_task | ||
logger.info("Starting ip cache maintenance for External IPAM provider, used by /next_ip.") | ||
@timer_task = Concurrent::TimerTask.new(execution_interval: DEFAULT_CLEANUP_INTERVAL) { clean_cache } | ||
@timer_task.execute | ||
end | ||
|
||
# @ip_cache structure | ||
# | ||
# Groups of subnets are cached under the External IPAM Group name. For example, | ||
# "IPAM Group Name" would be the section name in phpIPAM. All IP's cached for subnets | ||
# that do not have an External IPAM group specified, they are cached under the "" key. IP's | ||
# are cached using one of two possible keys: | ||
# 1). Mac Address | ||
# 2). UUID (Used when Mac Address not specified) | ||
# | ||
# { | ||
# "": { | ||
# "192.0.2.0/24":{ | ||
# "00:0a:95:9d:68:10": {"ip": "192.0.2.1", "timestamp": "2019-09-17 12:03:43 -D400"}, | ||
# "906d8bdc-dcc0-4b59-92cb-665935e21662": {"ip": "192.0.2.2", "timestamp": "2019-09-17 11:43:22 -D400"} | ||
# }, | ||
# }, | ||
# "IPAM Group Name": { | ||
# "198.51.100.0/24":{ | ||
# "00:0a:95:9d:68:33": {"ip": "198.51.100.1", "timestamp": "2019-09-17 12:04:43 -0400"}, | ||
# "00:0a:95:9d:68:34": {"ip": "198.51.100.2", "timestamp": "2019-09-17 12:05:48 -0400"}, | ||
# "00:0a:95:9d:68:35": {"ip": "198.51.100.3", "timestamp:: "2019-09-17 12:06:50 -0400"} | ||
# } | ||
# }, | ||
# "Another IPAM Group": { | ||
# "203.0.113.0/24":{ | ||
# "00:0a:95:9d:68:55": {"ip": "203.0.113.1", "timestamp": "2019-09-17 12:04:43 -0400"}, | ||
# "00:0a:95:9d:68:56": {"ip": "203.0.113.2", "timestamp": "2019-09-17 12:05:48 -0400"} | ||
# } | ||
# } | ||
# } | ||
def clean_cache | ||
@m.synchronize do | ||
entries_deleted = 0 | ||
total_entries = 0 | ||
|
||
@ip_cache.each do |group, subnets| | ||
subnets.each do |cidr, macs| | ||
macs.each do |mac, ip| | ||
if expired(ip[:timestamp]) | ||
@ip_cache[group][cidr].delete(mac) | ||
entries_deleted += 1 | ||
end | ||
total_entries += 1 | ||
end | ||
@ip_cache[group].delete(cidr) if @ip_cache[group][cidr].nil? || @ip_cache[group][cidr].empty? | ||
@ip_cache.delete(group) if @ip_cache[group].nil? || @ip_cache[group].empty? | ||
end | ||
end | ||
|
||
cache_count = total_entries - entries_deleted | ||
logger.debug("Removing #{entries_deleted} entries from IP cache for IPAM provider #{@provider}") if entries_deleted > 0 | ||
logger.debug("Current count of IP cache entries for IPAM provider #{@provider}: #{cache_count}") if entries_deleted > 0 | ||
end | ||
end | ||
end | ||
end |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note for myself / @ekohl - I am fine with bringing this feature into core, after all External IPAM is already in Foreman core, however, I would suggest to ship providers as RPM/DEB subpackages so products owners can decide what to ship and what not. Opinion?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does not introduce additional dependencies and the Smart Proxy is already quite smart in not loading the module if it's inactive so I'd just ship it in the same RPM/deb. Subpackages only make sense to reduce additional dependencies or reducing runtime overhead at the cost of more complex installation. In this case I think it's not worth it but I appreciate the consideration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay let me rephrase it this way - I do not want to support this in Red Hat Satellite at this moment. Perhaps we can disable it via the installer scenario?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently we don't have good facilities for that. It would need some thought.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok this is an internal discussion, I do not want block this effort.