-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement foreman test environment #1047
Open
bastelfreak
wants to merge
3
commits into
lsst-it:production
Choose a base branch
from
bastelfreak:bastelfreak
base: production
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,241 @@ | ||
| # Testing foreman | ||
|
|
||
| This is a rough guide for applying the code base on an existing Foreman 3.8/3.9 | ||
| installation on AlmaLinux 8. | ||
|
|
||
| This works by introducing a nother site, called `test`. The idea is to have | ||
| test systems that are isolated from the rest of the LSST infrastructure. For | ||
| example IPA isn't managed and no real route53 keys are provided. | ||
|
|
||
| ## Setup the VM | ||
|
|
||
| We use Hetzner for cloud instances to test setups: | ||
|
|
||
| ``` | ||
| hcloud server create --image=alma-8 --name=$host --type=cpx41 --ssh-key='bastelfreak' | ||
| hcloud server set-rdns $host --ip=95.217.179.41 --hostname=$host | ||
| hcloud server set-rdns $host --ip=2a01:4f9:c012:acee::1 --hostname=$host | ||
| ``` | ||
|
|
||
| (Now also add matching A/AAAA records to make this easier) | ||
|
|
||
| ``` | ||
| ssh-keygen -f ~/.ssh/known_hosts -R $host | ||
| ssh-keyscan $host >> ~/.ssh/known_hosts | ||
| ``` | ||
|
|
||
| ## Patching | ||
|
|
||
| ``` | ||
| sed --in-place 's/SELINUX=permissive/SELINUX=disabled/' /etc/selinux/config | ||
| echo 'if [ $TERM == "alacritty" ]; then export TERM=xterm-256color; fi' > /etc/profile.d/terminal.sh | ||
| LC_ALL=en_US.UTF-8 dnf -y update | ||
| LC_ALL=en_US.UTF-8 dnf -y install vim glibc-all-langpacks git bash-completion epel-release | ||
| crb enable | ||
| sync | ||
| reboot | ||
| ``` | ||
|
|
||
| ### Make vim less shitty | ||
|
|
||
| also this provides a persistent undo history in case I derp in config files | ||
|
|
||
| ``` | ||
| mkdir -p ~/.vim/{backupdir,undodir} | ||
| wget https://gist.githubusercontent.com/bastelfreak/a3cfa50db2a7be92c47f246f8f22ca5c/raw/dab14889680d4a8bbcb83580185ca2e5040d5947/vla.vimrc -O ~/.vimrc | ||
| ``` | ||
|
|
||
| ### Helpful tools | ||
|
|
||
| Those are helpful during testing | ||
|
|
||
| ``` | ||
| dnf -y install htop tig jq | ||
| ``` | ||
|
|
||
| ## install Puppet + Foreman | ||
|
|
||
| ``` | ||
| dnf -y install https://yum.puppet.com/puppet7-release-el-8.noarch.rpm | ||
| dnf -y install https://yum.theforeman.org/releases/3.8/el8/x86_64/foreman-release.rpm | ||
| dnf -y module enable foreman:el8 | ||
| dnf -y install foreman-installer | ||
| foreman-installer --enable-foreman-plugin-puppetdb | ||
| dnf -y install puppetdb puppetdb-termini postgresql-contrib | ||
| ``` | ||
|
|
||
| Output from the installer should be like this: | ||
|
|
||
| ``` | ||
| [root@lsst ~]# foreman-installer --enable-foreman-plugin-puppetdb | ||
| 2024-02-11 18:57:31 [NOTICE] [root] Loading installer configuration. This will take some time. | ||
| 2024-02-11 18:57:33 [NOTICE] [root] Running installer with log based terminal output at level NOTICE. | ||
| 2024-02-11 18:57:33 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions. | ||
| 2024-02-11 18:57:35 [NOTICE] [configure] Starting system configuration. | ||
| 2024-02-11 18:58:30 [NOTICE] [configure] 250 configuration steps out of 1244 steps complete. | ||
| 2024-02-11 18:58:39 [NOTICE] [configure] 500 configuration steps out of 1247 steps complete. | ||
| 2024-02-11 18:58:45 [NOTICE] [configure] 750 configuration steps out of 1272 steps complete. | ||
| 2024-02-11 18:58:56 [NOTICE] [configure] 1000 configuration steps out of 1272 steps complete. | ||
| 2024-02-11 19:00:12 [NOTICE] [configure] 1250 configuration steps out of 1272 steps complete. | ||
| 2024-02-11 19:00:15 [NOTICE] [configure] System configuration has finished. | ||
| Executing: foreman-rake upgrade:run | ||
| Success! | ||
| * Foreman is running at https://foreman | ||
| Initial credentials are admin / s2hYUi7oEksKxaNM | ||
| * Foreman Proxy is running at https://foreman | ||
|
|
||
| The full log is at /var/log/foreman-installer/foreman.log | ||
| [root@lsst ~]# | ||
| ``` | ||
|
|
||
| ### Configure r10k | ||
|
|
||
| # Install r10k + control-repo | ||
|
|
||
| First we want to stop puppet so it doesn't make unexpected changes in the | ||
| background after code got deployed. | ||
|
|
||
| ``` | ||
| systemctl disable --now puppet | ||
| ``` | ||
|
|
||
| Now install r10k | ||
|
|
||
| ``` | ||
| source /etc/profile.d/puppet-agent.sh | ||
| # required if we're on Puppet 7, which contains Ruby 2.7. newer faraday wants ruby 3 | ||
| puppet resource package faraday ensure=2.8.1 provider=puppet_gem | ||
| puppet resource package r10k ensure=installed provider=puppet_gem | ||
| ln -s /opt/puppetlabs/puppet/bin/r10k /usr/local/bin/ | ||
| ``` | ||
|
|
||
| configure r10k | ||
|
|
||
| ``` | ||
| mkdir -p /etc/puppetlabs/r10k | ||
| cat > /etc/puppetlabs/r10k/r10k.yaml << EOF | ||
| --- | ||
| pool_size: 8 | ||
| deploy: | ||
| generate_types: true | ||
| purge_levels: | ||
| - deployment | ||
| exclude_spec: true | ||
| incremental: true | ||
| :postrun: [] | ||
| :cachedir: /opt/puppetlabs/puppet/cache/r10k | ||
| :sources: | ||
| puppet: | ||
| basedir: /etc/puppetlabs/code/environments | ||
| remote: https://github.com/bastelfreak/lsst-control | ||
| EOF | ||
| ``` | ||
|
|
||
| deploy the code | ||
|
|
||
| ``` | ||
| r10k deploy environment production bastelfreak --modules --verbose --color | ||
| ``` | ||
|
|
||
| ## Configure PuppetDB | ||
|
|
||
| Setup the database and user | ||
|
|
||
| ``` | ||
| su --login postgres --command 'createuser --no-createdb --no-createrole --no-superuser puppetdb' | ||
| su --login postgres --command 'createuser --no-createdb --no-createrole --no-superuser puppetdb_read' | ||
| su --login postgres --command 'createdb --encoding UTF8 --owner postgres puppetdb' | ||
| su --login postgres --command "psql puppetdb --command 'revoke create on schema public from public'" | ||
| su --login postgres --command "psql puppetdb --command 'grant create on schema public to puppetdb'" | ||
| su --login postgres --command "psql puppetdb --command 'alter default privileges for user puppetdb in schema public grant select on tables to puppetdb_read'" | ||
| su --login postgres --command "psql puppetdb --command 'alter default privileges for user puppetdb in schema public grant usage on sequences to puppetdb_read'" | ||
| su --login postgres --command "psql puppetdb --command 'alter default privileges for user puppetdb in schema public grant execute on functions to puppetdb_read'" | ||
| su --login postgres --command "psql puppetdb --command 'create extension pg_trgm'" | ||
| su --login postgres --command "psql puppetdb --command \"ALTER USER puppetdb WITH PASSWORD 'PASSWORD'\"" | ||
| su --login postgres --command "psql puppetdb --command \"ALTER USER puppetdb_read WITH PASSWORD 'PASSWORD'\"" | ||
| ``` | ||
|
|
||
| Tell PuppetDB to use the database | ||
|
|
||
| ``` | ||
| echo '[database]' > /etc/puppetlabs/puppetdb/conf.d/database.ini | ||
| echo 'subname = //127.0.0.1:5432/puppetdb' >> /etc/puppetlabs/puppetdb/conf.d/database.ini | ||
| echo 'username = puppetdb' >> /etc/puppetlabs/puppetdb/conf.d/database.ini | ||
| echo 'password = PASSWORD' >> /etc/puppetlabs/puppetdb/conf.d/database.ini | ||
| echo '[read-database]' >> /etc/puppetlabs/puppetdb/conf.d/database.ini | ||
| echo 'subname = //127.0.0.1:5432/puppetdb' >> /etc/puppetlabs/puppetdb/conf.d/database.ini | ||
| echo 'username = puppetdb_read' >> /etc/puppetlabs/puppetdb/conf.d/database.ini | ||
| echo 'password = PASSWORD' >> /etc/puppetlabs/puppetdb/conf.d/database.ini | ||
| ``` | ||
|
|
||
| Start PuppetDB | ||
|
|
||
| ``` | ||
| systemctl enable --now puppetdb | ||
| ``` | ||
|
|
||
| Update Puppetserver to talk to PuppetDB | ||
| ``` | ||
| puppet config set --section server storeconfigs true | ||
| puppet config set --section main reports foreman,puppetdb | ||
| echo -e "[main]\nserver_urls = https://$(hostname -f):8081/\nsoft_write_failure = true" > /etc/puppetlabs/puppet/puppetdb.conf | ||
| systemctl restart puppetserver | ||
| ``` | ||
|
|
||
| ## configure node in foreman | ||
|
|
||
| We need to ensure foreman knows the environment `bastelfreak` before we can | ||
| assign it | ||
|
|
||
| * login at https://foreman/ | ||
| * got to https://foreman/foreman_puppet/environments, import new environments | ||
|
|
||
| We need to set the environment in foreman | ||
|
|
||
| * login at https://foreman/ | ||
| * select the node, click edit | ||
| * should bring you to https://foreman/hosts/foreman/edit | ||
| * At environment, select `bastelfreak` | ||
| * save | ||
|
|
||
| We need to set the role and site | ||
|
|
||
| * login at https://foreman/ | ||
| * At https://foreman/hosts/foreman/edit, go to `Parameters` | ||
| * Select `Add Parameter` | ||
| * Name=site, Value=test; save | ||
| * Repeat: Name=role, Value=foreman; save | ||
|
|
||
|
|
||
| At the moment a full puppet run doesn't succeed, but we can apply the following tags: | ||
|
|
||
| ``` | ||
| puppet agent -t --tags accounts,prometheus,chrony,yumrepo,auditd,tftp,convenience,debugutils,rsyslog,discovery,puppetserver,host,irqbalance,ssh,lldpd,sysstat,r10k,webhook,timezone,selinux,yum,docker,firewall,foreman_envsync,resolv_conf,sudo,postgresql_conf,udevd,reboot.target | ||
| ``` | ||
|
|
||
| Due to this we miss some migrations: | ||
|
|
||
| ``` | ||
| systemctl restart foreman | ||
| foreman-rake db:migrate | ||
| ``` | ||
|
|
||
| Then we can reboot: | ||
|
|
||
| ``` | ||
| sync; reboot | ||
| ``` | ||
|
|
||
| ## Rebuilding the instance | ||
|
|
||
| ``` | ||
| hcloud server rebuild $host --image=alma-8 | ||
| ssh-keygen -f ~/.ssh/known_hosts -R $host | ||
| ssh-keyscan $host >> ~/.ssh/known_hosts | ||
| ``` | ||
|
|
||
| ## Final updates | ||
|
|
||
| **update**: After a bit of playing with Hiera, Puppet now succeeds within two | ||
| runs. The `foreman_config_entry` resources only work on the second run, maybe | ||
| because they have a missing dependency to one of the foreman packages. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| --- | ||
| resolv_conf::nameservers: | ||
| - "185.12.64.2" | ||
| - "185.12.64.1" | ||
| - "2a01:4ff:ff00::add:2" | ||
| - "2a01:4ff:ff00::add:1" | ||
| profile::core::foreman::manage_smee: false | ||
| foreman_proxy::plugin::dns::route53::aws_access_key: "foo" | ||
| foreman_proxy::plugin::dns::route53::aws_secret_key: "foo" | ||
| puppet::server::puppetdb::server: "%{trusted.certname}" | ||
| r10k::sources: | ||
| control: | ||
| remote: "https://github.com/bastelfreak/lsst-control" | ||
| basedir: "/etc/puppetlabs/code/environments" | ||
| invalid_branches: "correct" | ||
| lookup_options: | ||
| r10k::sources: | ||
| merge: | ||
| strategy: "first" | ||
|
|
||
| puppet::server_puppetserver_version: &server_version '7.15.0' | ||
| puppet::server_version: '7.15.0' | ||
| puppet_agent::package_version: '7.28.0' | ||
| profile::core::yum::versionlock: | ||
| puppetdb-termini: | ||
| ensure: "present" | ||
| version: "7.16.0" | ||
| release: "1.el8" | ||
| before: "Package[puppetdb-termini]" | ||
|
|
||
| foreman::repo::repo: "3.8" | ||
| foreman::version: "3.8.0" | ||
| puppetdb::globals::version: '7.16.0' | ||
|
|
||
| profile::core::common::manage_sssd: false | ||
| profile::core::common::manage_network_manager: false | ||
| profile::core::common::manage_krb5: false | ||
| profile::core::common::manage_ipa: false |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the plugin doesn't exist anymore because the code moved into foreman core. The Hiera data is currently unused. I didn't investigate yet how to configure the columns on modern foreman.