Skip to content

Commit

Permalink
(maint) uncouple from the puppet_litmus gem
Browse files Browse the repository at this point in the history
vagrant is still todo
  • Loading branch information
h0tw1r3 committed Apr 30, 2024
1 parent a134735 commit db14259
Show file tree
Hide file tree
Showing 12 changed files with 182 additions and 127 deletions.
22 changes: 11 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ $ cat ~/.fog
##### Setting up a new machine

```ruby
$ bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::abs --targets localhost action=provision platform=ubuntu-1604-x86_64 inventory=/Users/tp/workspace/git/provision/inventory.yaml
$ bundle exec bolt task run provision::abs --targets localhost action=provision platform=ubuntu-1604-x86_64

Started on localhost...
Finished on localhost:
Expand All @@ -118,7 +118,7 @@ Ran on 1 node in 1.44 seconds
##### Tearing down a finished machine

```ruby
$ bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::abs --targets localhost action=tear_down inventory=/Users/tp/workspace/git/provision/inventory.yaml node_name=yh6f4djvz7o3te6.delivery.puppetlabs.net
$ bundle exec bolt task run provision::abs --targets localhost action=tear_down node_name=yh6f4djvz7o3te6.delivery.puppetlabs.net

Started on localhost...
Finished on localhost:
Expand All @@ -139,7 +139,7 @@ Containers by default will be managed in the current [docker context](https://do
#### Provision

```ruby
$ bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::docker --targets localhost action=provision platform=ubuntu:14.04 inventory=/Users/tp/workspace/git/provision/inventory.yaml
$ bundle exec bolt task run provision::docker --targets localhost action=provision platform=ubuntu:14.04

Started on localhost...
Finished on localhost:
Expand All @@ -166,13 +166,13 @@ These defaults can be overriden by passing the flags with different values i.e.
```

```ruby
bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::docker --targets localhost action=provision platform=ubuntu:14.04 inventory=/Users/tp/workspace/git/provision/inventory.yaml vars='{ "docker_run_opts": ["-p 8086:8086", "-p 3000:3000"]}'
bundle exec bolt task run provision::docker --targets localhost action=provision platform=ubuntu:14.04 vars='{ "docker_run_opts": ["-p 8086:8086", "-p 3000:3000"]}'
```

#### Tear down

```ruby
$ bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::docker --targets localhost action=tear_down inventory=/Users/tp/workspace/git/provision/inventory.yaml node_name=localhost:2222
$ bundle exec bolt task run provision::docker --targets localhost action=tear_down node_name=localhost:2222

Started on localhost...
Finished on localhost:
Expand All @@ -197,7 +197,7 @@ Tested with vagrant images:
provision

```ruby
$ bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::vagrant --targets localhost action=provision platform=ubuntu/xenial64 inventory=/Users/tp/workspace/git/provision/inventory.yaml
$ bundle exec bolt task run provision::vagrant --targets localhost action=provision platform=ubuntu/xenial64

Started on localhost...
Finished on localhost:
Expand All @@ -217,7 +217,7 @@ This leads to errors when anything tries to execute `puppet` commands on the tes
To add the Puppet agent binary path to the *secure_path* please run the `provision::fix_secure_path` Bolt task:

```ruby
$ bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::fix_secure_path path=/opt/puppetlabs/bin -i inventory.yaml -t ssh_nodes
$ bundle exec bolt task run provision::fix_secure_path path=/opt/puppetlabs/bin -i inventory.yaml -t ssh_nodes

Started on 127.0.0.1:2222...
Finished on 127.0.0.1:2222:
Expand All @@ -229,7 +229,7 @@ Ran on 1 target in 0.84 sec
tear_down

```ruby
$ bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::vagrant --targets localhost action=tear_down inventory=/Users/tp/workspace/git/provision/inventory.yaml node_name=127.0.0.1:2222
$ bundle exec bolt task run provision::vagrant --targets localhost action=tear_down node_name=127.0.0.1:2222

Started on localhost...
Finished on localhost:
Expand Down Expand Up @@ -284,7 +284,7 @@ In the provision step you can invoke bundle exec rake 'litmus:provision_list[tes
Manual invocation of the provision service task from a workflow can be done using:

```ruby
bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::provision_service --targets localhost action=provision platform=centos-7-v20200813 inventory=/Users/tp/workspace/git/provision/inventory.yaml
bolt task run provision::provision_service --targets localhost action=provision platform=centos-7-v20200813
```

Or using Litmus:
Expand Down Expand Up @@ -320,7 +320,7 @@ provision

```powershell
PS> $env:LITMUS_HYPERV_VSWITCH = 'internal_nat'
PS> bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::vagrant --targets localhost action=provision platform=centos/7 inventory=/Users/tp/workspace/git/provision/inventory.yaml hyperv_smb_username=tp hyperv_smb_password=notMyrealPassword
PS> bundle exec bolt task run provision::vagrant --targets localhost action=provision platform=centos/7 hyperv_smb_username=tp hyperv_smb_password=notMyrealPassword
Started on localhost...
Finished on localhost:
Expand Down Expand Up @@ -353,7 +353,7 @@ Testing/development/debugging it is better to use ruby directly, you will need t
Testing using bolt, the second step

```ruby
bundle exec bolt --modulepath /Users/tp/workspace/git/ task run provision::docker --targets localhost action=provision platform=ubuntu:14.04 inventory=/Users/tp/workspace/git/provision/inventory.yaml
bundle exec bolt task run provision::docker --targets localhost action=provision platform=ubuntu:14.04
```

## License
Expand Down
15 changes: 2 additions & 13 deletions lib/docker_helper.rb
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# frozen_string_literal: true

require 'json'
require 'puppet_litmus'

def docker_exec(container, command)
run_local_command("docker exec #{container} #{command}")
Expand Down Expand Up @@ -34,18 +33,8 @@ def docker_image_os_release_facts(image)
os_release_facts
end

def docker_tear_down(node_name, inventory_location)
extend PuppetLitmus::InventoryManipulation
inventory_full_path = File.join(inventory_location, '/spec/fixtures/litmus_inventory.yaml')
raise "Unable to find '#{inventory_full_path}'" unless File.file?(inventory_full_path)

inventory_hash = inventory_hash_from_inventory_file(inventory_full_path)
node_facts = facts_from_node(inventory_hash, node_name)
remove_docker = "docker rm -f #{node_facts['container_id']}"
run_local_command(remove_docker)
remove_node(inventory_hash, node_name)
puts "Removed #{node_name}"
File.open(inventory_full_path, 'w') { |f| f.write inventory_hash.to_yaml }
def docker_tear_down(node)
run_local_command("docker rm -f #{node['facts']['container_id']}")
{ status: 'ok' }
end

Expand Down
105 changes: 105 additions & 0 deletions lib/inventory_helper.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# frozen_string_literal: true

require 'yaml'
require 'delegate'

class InventoryHelper < SimpleDelegator

Check failure on line 6 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Style/Documentation: Missing top-level documentation comment for `class InventoryHelper`.

Check failure on line 6 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Style/Documentation: Missing top-level documentation comment for `class InventoryHelper`.
def initialize(location)
@location = location
super(refresh)
end

# Load inventory from location in YAML format
# or generate a default structure
#
# @return [Hash]
def refresh
x = YAML.load_file(@location) if File.file?(@location)
{ 'version' => 2, 'groups' => [] }.merge(x || {})
end

# Save inventory to location in yaml format
def save
File.open(@location, 'wb+') { |f| f.write(self.to_yaml) }

Check failure on line 23 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Style/RedundantSelf: Redundant `self` detected. (https://rubystyle.guide#no-self-unless-required)

Check failure on line 23 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Style/RedundantSelf: Redundant `self` detected. (https://rubystyle.guide#no-self-unless-required)
end

# Adds a node to a group specified, if group_name exists in inventory hash.
#
# @param node [Hash] node to add to the group
# @param group [String] group of nodes to limit the search for the node_name in
# @return [Hash] inventory_hash with node added to group if group_name exists in inventory hash.
def add(node, group)
# check if group exists
if self['groups'].any? { |g| g['name'] == group }
self['groups'].each do |g|
g['targets'].push(node) if g['name'] == group
end
else
# add new group
self['groups'].push({ 'name' => group, 'targets' => [node] })
end

self
end

# Lookup a node
#
# @param uri [String] uri of node to find
# @param name [String] name of node to find
# @param group [String] limit search to group
# @return [Hash] inventory target
def lookup(uri=nil, name: nil, group: nil)

Check failure on line 51 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Layout/SpaceAroundEqualsInParameterDefault: Surrounding space missing in default value assignment. (https://rubystyle.guide#spaces-around-equals)

Check failure on line 51 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Layout/SpaceAroundEqualsInParameterDefault: Surrounding space missing in default value assignment. (https://rubystyle.guide#spaces-around-equals)
value = uri || name
key = uri.nil? ? 'name' : 'uri'

self['groups'].each do |g|
if (group && group == g['name']) || group.nil?

Check failure on line 56 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Style/Next: Use `next` to skip iteration. (https://rubystyle.guide#no-nested-conditionals)

Check failure on line 56 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Style/Next: Use `next` to skip iteration. (https://rubystyle.guide#no-nested-conditionals)
g['targets'].each do |t|
return t if t[key] == value
end
end
end

raise "Failed to lookup target for #{key} #{value} in inventory"
end

# Remove node
#
# @param node [Hash]
# @return [Hash] inventory_hash with node of node_name removed.
def remove(node)
self['groups'].map! do |g|
g['targets'].select! { |target| target != node }

Check failure on line 72 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Style/InverseMethods: Use `reject!` instead of inverting `select!`.

Check failure on line 72 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Style/InverseMethods: Use `reject!` instead of inverting `select!`.
g
end

self
end

class << self
attr_accessor :instances

def open(location = nil)
# Inventory location is an optional task parameter.
location = location.nil? ? Dir.pwd : location
location = if File.directory?(location)
# DEPRECATED: puppet_litmus <= 1.3.0 support
if Gem.loaded_specs['puppet_litmus'] && Gem.loaded_specs['puppet_litmus'].version <= Gem::Version.new('1.3.0')
File.join(location, 'spec', 'fixtures', 'litmus_inventory.yaml')
else
File.join(location, 'inventory.yaml')
end
else
location
end

@instances ||= Hash.new

Check failure on line 96 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Style/EmptyLiteral: Use hash literal `{}` instead of `Hash.new`. (https://rubystyle.guide#literal-array-hash)

Check failure on line 96 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Style/EmptyLiteral: Use hash literal `{}` instead of `Hash.new`. (https://rubystyle.guide#literal-array-hash)
@instances[location] = self.new(location) unless @instances.key? location

Check failure on line 97 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Style/RedundantSelf: Redundant `self` detected. (https://rubystyle.guide#no-self-unless-required)

Check failure on line 97 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Style/RedundantSelf: Redundant `self` detected. (https://rubystyle.guide#no-self-unless-required)
return @instances[location]

Check failure on line 98 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Style/RedundantReturn: Redundant `return` detected. (https://rubystyle.guide#no-explicit-return)

Check failure on line 98 in lib/inventory_helper.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Style/RedundantReturn: Redundant `return` detected. (https://rubystyle.guide#no-explicit-return)
end
end

protected

attr_accessor :location
end
24 changes: 0 additions & 24 deletions lib/task_helper.rb
Original file line number Diff line number Diff line change
@@ -1,29 +1,5 @@
# frozen_string_literal: true

def sanitise_inventory_location(location)
# Inventory location is an optional task parameter.
location = location.nil? ? Dir.pwd : location
# If not specified use the current directory + inventory.yaml
if File.exist?(location) && File.directory?(location)
# DEPRECATED: puppet_litmus <= 1.3.0 support
if Gem.loaded_specs['puppet_litmus'].version <= Gem::Version.new('1.3.0')
File.join(location, 'spec', 'fixtures', 'litmus_inventory.yaml')
else
File.join(location, 'inventory.yaml')
end
else
location
end
end

def get_inventory_hash(inventory_full_path)
if File.file?(inventory_full_path)
inventory_hash_from_inventory_file(inventory_full_path)
else
{ 'version' => 2, 'groups' => [{ 'name' => 'docker_nodes', 'targets' => [] }, { 'name' => 'ssh_nodes', 'targets' => [] }, { 'name' => 'winrm_nodes', 'targets' => [] }] }
end
end

def run_local_command(command, dir = Dir.pwd)
require 'open3'
stdout, stderr, status = Open3.capture3(command, chdir: dir)
Expand Down
3 changes: 2 additions & 1 deletion tasks/abs.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
}
},
"files": [
"provision/lib/task_helper.rb"
"provision/lib/task_helper.rb",
"provision/lib/inventory_helper.rb"
]
}
41 changes: 19 additions & 22 deletions tasks/abs.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,13 @@
require 'json'
require 'net/http'
require 'yaml'
require 'puppet_litmus'
require 'etc'
require 'date'
require_relative '../lib/task_helper'
require_relative '../lib/inventory_helper'

# Provision and teardown vms through ABS.
class ABSProvision
include PuppetLitmus::InventoryManipulation

# Enforces a k8s.infracore.puppet.net domain, but allows selection of prod,
# stage, etc hostname from the environment variable +ABS_SUBDOMAIN+ so that
# CI can test vms from staging.
Expand All @@ -23,7 +21,7 @@ def abs_host
"#{subdomain}.k8s.infracore.puppet.net"
end

def provision(platform, inventory_location, vars)
def provision(platform, inventory, vars)
uri = URI.parse("https://#{abs_host}/api/v2/request")
jenkins_build_url = if ENV['CI'] == 'true' && ENV['TRAVIS'] == 'true'
ENV.fetch('TRAVIS_JOB_WEB_URL', nil)
Expand Down Expand Up @@ -85,7 +83,6 @@ def provision(platform, inventory_location, vars)

raise "Timeout: unable to get a 200 response in #{poll_duration} seconds" if reply.code != '200'

inventory_hash = get_inventory_hash(inventory_location)
data = JSON.parse(reply.body)
data.each do |host|
if platform_uses_ssh(host['type'])
Expand All @@ -109,27 +106,26 @@ def provision(platform, inventory_location, vars)
var_hash = YAML.safe_load(vars)
node['vars'] = var_hash
end
add_node_to_group(inventory_hash, node, group_name)
inventory.add(node, group_name)
end

File.open(inventory_location, 'w') { |f| f.write inventory_hash.to_yaml }
inventory.save
{ status: 'ok', nodes: data.length }
end

def tear_down(node_name, inventory_location)
if File.file?(inventory_location)
inventory_hash = inventory_hash_from_inventory_file(inventory_location)
facts = facts_from_node(inventory_hash, node_name)
platform = facts['platform']
job_id = facts['job_id']
end
def tear_down(node_name, inventory)
node = inventory.lookup(name: node_name)
facts = node['facts']
platform = facts['platform']
job_id = facts['job_id']

targets_to_remove = []
inventory_hash['groups'].each do |group|
inventory['groups'].each do |group|
group['targets'].each do |node|

Check failure on line 124 in tasks/abs.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 7.24, Ruby Ver: 2.7)

Lint/ShadowingOuterLocalVariable: Shadowing outer local variable - `node`.

Check failure on line 124 in tasks/abs.rb

View workflow job for this annotation

GitHub Actions / Spec / Spec tests (Puppet: ~> 8.0, Ruby Ver: 3.2)

Lint/ShadowingOuterLocalVariable: Shadowing outer local variable - `node`.
targets_to_remove.push(node['uri']) if node['facts']['job_id'] == job_id
targets_to_remove.push(node) if node['facts']['job_id'] == job_id
end
end

uri = URI.parse("https://#{abs_host}/api/v2/return")
headers = { 'X-AUTH-TOKEN' => token_from_fogfile('abs'), 'Content-Type' => 'application/json' }
payload = { 'job_id' => job_id,
Expand All @@ -143,16 +139,17 @@ def tear_down(node_name, inventory_location)
raise "Error: #{reply}: #{reply.message}" unless reply.code == '200'

targets_to_remove.each do |target|
remove_node(inventory_hash, target)
inventory.remove(target)
end
File.open(inventory_location, 'w') { |f| f.write inventory_hash.to_yaml }
{ status: 'ok', removed: targets_to_remove }
inventory.save

{ status: 'ok', removed: targets_to_remove.map { |t| t['name'] } }
end

def task(action:, platform: nil, node_name: nil, inventory: nil, vars: nil, **_kwargs)
inventory_location = sanitise_inventory_location(inventory)
result = provision(platform, inventory_location, vars) if action == 'provision'
result = tear_down(node_name, inventory_location) if action == 'tear_down'
inventory = InventoryHelper.open(inventory)
result = provision(platform, inventory, vars) if action == 'provision'
result = tear_down(node_name, inventory) if action == 'tear_down'
result
end

Expand Down
3 changes: 2 additions & 1 deletion tasks/docker.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
},
"files": [
"provision/lib/task_helper.rb",
"provision/lib/docker_helper.rb"
"provision/lib/docker_helper.rb",
"provision/lib/inventory_helper.rb"
]
}
Loading

0 comments on commit db14259

Please sign in to comment.