Merge pull request #769 from shaderecker/ansible

Ansible updates

README.md

@@ -290,9 +290,9 @@ If you would like to contribute, you can add other cloud deployments like Chef o
 You can find an [Ansible](https://www.ansible.com/) based T-Pot deployment in the [`cloud/ansible`](cloud/ansible) folder.  
 The Playbook in the [`cloud/ansible/openstack`](cloud/ansible/openstack) folder is reusable for all **OpenStack** clouds out of the box.
 
-It first creates all resources (security group, network, subnet, router), deploys a new server and then installs and configures T-Pot.
+It first creates all resources (security group, network, subnet, router), deploys one (or more) new servers and then installs and configures T-Pot on them.
 
-You can have a look at the Playbook and easily adapt the deploy role for other [cloud providers](https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html).
+You can have a look at the Playbook and easily adapt the deploy role for other [cloud providers](https://docs.ansible.com/ansible/latest/scenario_guides/cloud_guides.html). Check out [Ansible Galaxy](https://galaxy.ansible.com/search?keywords=&order_by=-relevance&page=1&deprecated=false&type=collection&tags=cloud) for more cloud collections.
 
 *Please note*: Cloud providers usually offer adjusted Debian OS images, which might not be compatible with T-Pot. There is no cloud provider support provided of any kind.
 
@@ -304,7 +304,7 @@ You can find [Terraform](https://www.terraform.io/) configuration in the [`cloud
 This can be used to launch a virtual machine, bootstrap any dependencies and install T-Pot in a single step.
 
 Configuration for **Amazon Web Services** (AWS) and **Open Telekom Cloud** (OTC) is currently included.  
-This can easily be extended to support other [Terraform providers](https://www.terraform.io/docs/providers/index.html).
+This can easily be extended to support other [Terraform providers](https://registry.terraform.io/browse/providers?category=public-cloud%2Ccloud-automation%2Cinfrastructure).
 
 *Please note*: Cloud providers usually offer adjusted Debian OS images, which might not be compatible with T-Pot. There is no cloud provider support provided of any kind.
 

cloud/.gitignore

@@ -0,0 +1,10 @@
+# Ansible
+*.retry
+
+# Terraform
+**/.terraform
+**/terraform.*
+
+# OpenStack clouds
+clouds.yaml
+secure.yaml

cloud/ansible/.gitignore

@@ -1,2 +0,0 @@
-# Ansible
-*.retry

cloud/ansible/README.md

@@ -2,15 +2,16 @@
 
 Here you can find a ready-to-use solution for your automated T-Pot deployment using [Ansible](https://www.ansible.com/).  
 It consists of an Ansible Playbook with multiple roles, which is reusable for all [OpenStack](https://www.openstack.org/) based clouds (e.g. Open Telekom Cloud, Orange Cloud, Telefonica Open Cloud, OVH) out of the box.  
-Apart from that you can easily adapt the deploy role to use other [cloud providers](https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html) (e.g. AWS, Azure, Digital Ocean, Google).
+Apart from that you can easily adapt the deploy role to use other [cloud providers](https://docs.ansible.com/ansible/latest/scenario_guides/cloud_guides.html). Check out [Ansible Galaxy](https://galaxy.ansible.com/search?keywords=&order_by=-relevance&page=1&deprecated=false&type=collection&tags=cloud) for more cloud collections.
 
-The Playbook first creates all resources (security group, network, subnet, router), deploys a new server and then installs and configures T-Pot.
+The Playbook first creates all resources (security group, network, subnet, router), deploys one (or more) new servers and then installs and configures T-Pot on them.
 
 This example showcases the deployment on our own OpenStack based Public Cloud Offering [Open Telekom Cloud](https://open-telekom-cloud.com/en).
 
 # Table of contents
 - [Preparation of Ansible Master](#ansible-master)
   - [Ansible Installation](#ansible)
+  - [OpenStack Collection Installation](#collection)
   - [Agent Forwarding](#agent-forwarding)
 - [Preparations in Open Telekom Cloud Console](#preparation)
   - [Create new project](#project)
@@ -18,8 +19,9 @@ This example showcases the deployment on our own OpenStack based Public Cloud Of
   - [Import Key Pair](#key-pair)
 - [Clone Git Repository](#clone-git)
 - [Settings and recommended values](#settings)
-  - [Clouds.yaml](#clouds-yaml)
+  - [clouds.yaml](#clouds-yaml)
   - [Ansible remote user](#remote-user)
+  - [Number of instances to deploy](#number)
   - [Instance settings](#instance-settings)
   - [User password](#user-password)
   - [Configure `tpot.conf.dist`](#tpot-conf)
@@ -56,6 +58,11 @@ In short (if you already have Python3/pip3 installed):
 pip3 install ansible
 ```
 
+<a name="collection"></a>
+## OpenStack Collection Installation
+For interacting with OpenStack resources in Ansible, you need to install the collection from Ansible Galaxy:  
+`ansible-galaxy collection install openstack.cloud`
+
 <a name="agent-forwarding"></a>
 ## Agent Forwarding
 If you run the Ansible Playbook remotely on your Ansible Master Server, Agent Forwarding must be enabled in order to let Ansible connect to newly created machines.  
@@ -112,7 +119,7 @@ All Ansible related files are located in the [`cloud/ansible/openstack`](opensta
 You can configure all aspects of your Elastic Cloud Server and T-Pot before using the Playbook:
 
 <a name="clouds-yaml"></a>
-## Clouds.yaml
+## clouds.yaml
 Located at [`openstack/clouds.yaml`](openstack/clouds.yaml).  
 Enter your Open Telekom Cloud API user credentials here (username, password, project name, user domain name):  
 ```
@@ -126,22 +133,36 @@ clouds:
       user_domain_name: OTC-EU-DE-000000000010000XXXXX
 ```
 You can also perform different authentication methods like sourcing OpenStack OS_* environment variables or providing an inline dictionary.  
-For more information have a look in the [os_server](https://docs.ansible.com/ansible/latest/modules/os_server_module.html) Ansible module documentation.
+For more information have a look in the [openstack.cloud.server](https://docs.ansible.com/ansible/latest/collections/openstack/cloud/server_module.html) Ansible module documentation.
+
+If you already have your own `clouds.yaml` file or have multiple clouds in there, you can specify which one to use in the `openstack/my_os_cloud.yaml` file:
+```
+# Enter the name of your cloud to use from clouds.yaml
+cloud: open-telekom-cloud
+```
 
 <a name="remote-user"></a>
 ## Ansible remote user
 You may have to adjust the `remote_user` in the Ansible Playbook under [`openstack/deploy_tpot.yaml`](openstack/deploy_tpot.yaml) depending on your Debian base image (e.g. on Open Telekom Cloud the default Debian user is `linux`).
 
+<a name="number"></a>
+## Number of instances to deploy
+You can adjust the number of VMs/T-Pots that you want to create in [`openstack/deploy_tpot.yaml`](openstack/deploy_tpot.yaml):
+```
+loop: "{{ range(0, 1) }}"
+```
+One instance is set as the default, increase to your liking.
+
 <a name="instance-settings"></a>
 ## Instance settings
-Located at [`openstack/roles/deploy/vars/main.yaml`](openstack/roles/deploy/vars/main.yaml).  
+Located at [`openstack/roles/create_vm/vars/main.yaml`](openstack/roles/create_vm/vars/main.yaml).  
 Here you can customize your virtual machine specifications:
   - Choose an availability zone. For Open Telekom Cloud reference see [here](https://docs.otc.t-systems.com/en-us/endpoint/index.html).
   - Change the OS image (For T-Pot we need Debian)
   - (Optional) Change the volume size
   - Specify your key pair (:warning: Mandatory)
   - (Optional) Change the instance type (flavor)  
-    `s2.medium.8` corresponds to 1 vCPU and 8GB of RAM and is the minimum required flavor.  
+    `s3.medium.8` corresponds to 1 vCPU and 8GB of RAM and is the minimum required flavor.  
     A full list of Open Telekom Cloud flavors can be found [here](https://docs.otc.t-systems.com/en-us/usermanual/ecs/en-us_topic_0177512565.html).
 
 ```
@@ -149,7 +170,7 @@ availability_zone: eu-de-03
 image: Standard_Debian_10_latest
 volume_size: 128
 key_name: your-KeyPair
-flavor: s2.medium.8
+flavor: s3.medium.8
 ```
 
 <a name="user-password"></a>
@@ -200,7 +221,7 @@ Enable this by uncommenting the role in the [deploy_tpot.yaml](openstack/deploy_
 #    - custom_hpfeeds
 ```
 
-You can specify custom HPFEEDS in [`openstack/roles/custom_hpfeeds/templates/hpfeeds.cfg`](openstack/roles/custom_hpfeeds/templates/hpfeeds.cfg).  
+You can specify custom HPFEEDS in [`openstack/roles/custom_hpfeeds/files/hpfeeds.cfg`](openstack/roles/custom_hpfeeds/files/hpfeeds.cfg).  
 That file contains the defaults (turned off) and you can adapt it for your needs, e.g. for SISSDEN:
 ```
 myENABLE=true
@@ -216,6 +237,7 @@ myFORMAT=json
 <a name="deploy"></a>
 # Deploying a T-Pot :honey_pot::honeybee:
 Now, after configuring everything, we can finally start deploying T-Pots!  
+
 Go to the [`openstack`](openstack) folder and run the Ansible Playbook with:  
 `ansible-playbook deploy_tpot.yaml`  
 (Yes, it is as easy as that :smile:)
@@ -223,15 +245,13 @@ Go to the [`openstack`](openstack) folder and run the Ansible Playbook with:
 If you are running on a machine which asks for a sudo password, you can use:  
 `ansible-playbook --ask-become-pass deploy_tpot.yaml`
 
-The Playbook will first install required packages on the Ansible Master and then deploy a new server instance.  
+The Playbook will first install required packages on the Ansible Master and then deploy one (or more) new server instances.  
-After that, T-Pot gets installed and configured on the newly created host, optionally custom configs are applied and finally it reboots.
+After that, T-Pot gets installed and configured on them, optionally custom configs are applied and finally it reboots.
 
 Once this is done, you can proceed with connecting/logging in to the T-Pot according to the [documentation](https://github.com/telekom-security/tpotce#ssh-and-web-access).
 
 <a name="documentation"></a>
 # Further documentation
 - [Ansible Documentation](https://docs.ansible.com/ansible/latest/)
-- [Cloud modules — Ansible Documentation](https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html)
+- [openstack.cloud.server – Create/Delete Compute Instances from OpenStack](https://docs.ansible.com/ansible/latest/collections/openstack/cloud/server_module.html)
-- [os_server – Create/Delete Compute Instances from OpenStack — Ansible Documentation](https://docs.ansible.com/ansible/latest/modules/os_server_module.html)
 - [Open Telekom Cloud Help Center](https://docs.otc.t-systems.com/)
-- [Open Telekom Cloud API Overview](https://docs.otc.t-systems.com/en-us/api/wp/en-us_topic_0052070394.html)

cloud/ansible/openstack/deploy_tpot.yaml

@@ -4,13 +4,22 @@
   roles:
     - check
 
-- name: Deploy instance
+- name: Deploy instances
   hosts: localhost
-  roles:
+  vars_files: my_os_cloud.yaml
-    - deploy
+  tasks:
+    - name: Create security group and network
+      ansible.builtin.include_role:
+        name: create_net
+    - name: Create one or more instances
+      ansible.builtin.include_role:
+        name: create_vm
+      loop: "{{ range(0, 1) }}"
+      loop_control:
+        extended: yes
 
-- name: Install T-Pot on new instance
+- name: Install T-Pot
-  hosts: TPOT
+  hosts: tpot
   remote_user: linux
   become: yes
   gather_facts: no

cloud/ansible/openstack/my_os_cloud.yaml

@@ -0,0 +1,2 @@
+# Enter the name of your cloud to use from clouds.yaml
+cloud: open-telekom-cloud

cloud/ansible/openstack/requirements.yaml

@@ -0,0 +1,2 @@
+collections:
+- name: openstack.cloud

cloud/ansible/openstack/roles/check/tasks/main.yaml

@@ -1,20 +1,19 @@
 - name: Install dependencies
-  package:
+  ansible.builtin.package:
     name:
       - gcc
-      - pwgen
       - python3-dev
       - python3-setuptools
       - python3-pip
     state: present
 
 - name: Install openstacksdk
-  pip:
+  ansible.builtin.pip:
     name: openstacksdk
     executable: pip3
 
 - name: Check if agent forwarding is enabled
-  fail:
+  ansible.builtin.fail:
     msg: Please enable agent forwarding to allow Ansible to connect to the remote host!
   ignore_errors: yes
   when: lookup('env','SSH_AUTH_SOCK') == ""

cloud/ansible/openstack/roles/create_net/tasks/main.yaml

@@ -0,0 +1,33 @@
+- name: Create security group
+  openstack.cloud.security_group:
+    cloud: "{{ cloud }}"
+    name: sg-tpot-any
+    description: tpot any-any
+
+- name: Add rules to security group
+  openstack.cloud.security_group_rule:
+    cloud: "{{ cloud }}"
+    security_group: sg-tpot-any
+    remote_ip_prefix: 0.0.0.0/0
+
+- name: Create network
+  openstack.cloud.network:
+    cloud: "{{ cloud }}"
+    name: network-tpot
+
+- name: Create subnet
+  openstack.cloud.subnet:
+    cloud: "{{ cloud }}"
+    network_name: network-tpot
+    name: subnet-tpot
+    cidr: 192.168.0.0/24
+    dns_nameservers:
+      - 1.1.1.1
+      - 8.8.8.8
+
+- name: Create router
+  openstack.cloud.router:
+    cloud: "{{ cloud }}"
+    name: router-tpot
+    interfaces:
+      - subnet-tpot

cloud/ansible/openstack/roles/create_vm/tasks/main.yaml

@@ -0,0 +1,24 @@
+- name: Generate T-Pot name
+  ansible.builtin.set_fact:
+    tpot_name: "t-pot-ansible-{{ lookup('password', '/dev/null chars=ascii_lowercase,digits length=6') }}"
+
+- name: Create instance {{ ansible_loop.index }} of {{ ansible_loop.length }}
+  openstack.cloud.server:
+    cloud: "{{ cloud }}"
+    name: "{{ tpot_name }}"
+    availability_zone: "{{ availability_zone }}"
+    image: "{{ image }}"
+    boot_from_volume: yes
+    volume_size: "{{ volume_size }}"
+    key_name: "{{ key_name }}"
+    timeout: 200
+    flavor: "{{ flavor }}"
+    security_groups: sg-tpot-any
+    network: network-tpot
+  register: tpot
+
+- name: Add instance to inventory
+  ansible.builtin.add_host:
+    hostname: "{{ tpot_name }}"
+    ansible_host: "{{ tpot.server.public_v4 }}"
+    groups: tpot

cloud/ansible/openstack/roles/create_vm/vars/main.yaml

@@ -2,4 +2,4 @@ availability_zone: eu-de-03
 image: Standard_Debian_10_latest
 volume_size: 128
 key_name: your-KeyPair
-flavor: s2.medium.8
+flavor: s3.medium.8

cloud/ansible/openstack/roles/custom_ews/tasks/main.yaml

@@ -1,5 +1,5 @@
 - name: Copy ews configuration file
-  template:
+  ansible.builtin.template:
     src: ews.cfg
     dest: /data/ews/conf
     owner: root
@@ -7,7 +7,7 @@
     mode: 0644
 
 - name: Patching tpot.yml with custom ews configuration file
-  lineinfile:
+  ansible.builtin.lineinfile:
     path: /opt/tpot/etc/tpot.yml
     insertafter: "/opt/ewsposter/ews.ip"
     line: "     - /data/ews/conf/ews.cfg:/opt/ewsposter/ews.cfg"

cloud/ansible/openstack/roles/custom_hpfeeds/tasks/main.yaml

@@ -1,5 +1,5 @@
 - name: Copy hpfeeds configuration file
-  copy:
+  ansible.builtin.copy:
     src: hpfeeds.cfg
     dest: /data/ews/conf
     owner: tpot
@@ -8,5 +8,5 @@
   register: config
 
 - name: Applying hpfeeds settings
-  command: /opt/tpot/bin/hpfeeds_optin.sh --conf=/data/ews/conf/hpfeeds.cfg
+  ansible.builtin.command: /opt/tpot/bin/hpfeeds_optin.sh --conf=/data/ews/conf/hpfeeds.cfg
   when: config.changed == true

cloud/ansible/openstack/roles/deploy/tasks/main.yaml

@@ -1,58 +0,0 @@
-- name: Create T-Pot name
-  shell: echo t-pot-ansible-$(pwgen -ns 6 -1)
-  register: tpot_name
-
-- name: Create security group
-  os_security_group:
-    cloud: open-telekom-cloud
-    name: sg-tpot-any
-    description: tpot any-any
-
-- name: Add rules to security group
-  os_security_group_rule:
-    cloud: open-telekom-cloud
-    security_group: sg-tpot-any
-    remote_ip_prefix: 0.0.0.0/0
-
-- name: Create network
-  os_network:
-    cloud: open-telekom-cloud
-    name: network-tpot
-
-- name: Create subnet
-  os_subnet:
-    cloud: open-telekom-cloud
-    network_name: network-tpot
-    name: subnet-tpot
-    cidr: 192.168.0.0/24
-    dns_nameservers:
-      - 1.1.1.1
-      - 8.8.8.8
-
-- name: Create router
-  os_router:
-    cloud: open-telekom-cloud
-    name: router-tpot
-    interfaces:
-      - subnet-tpot
-
-- name: Launch an instance
-  os_server:
-    cloud: open-telekom-cloud
-    name: "{{ tpot_name.stdout }}"
-    availability_zone: "{{ availability_zone }}"
-    image: "{{ image }}"
-    boot_from_volume: yes
-    volume_size: "{{ volume_size }}"
-    key_name: "{{ key_name }}"
-    timeout: 200
-    flavor: "{{ flavor }}"
-    security_groups: sg-tpot-any
-    network: network-tpot
-  register: tpot
-
-- name: Add instance to inventory
-  add_host:
-    hostname: "{{ tpot_name.stdout }}"
-    ansible_host: "{{ tpot.server.public_v4 }}"
-    groups: TPOT

cloud/ansible/openstack/roles/install/tasks/main.yaml

@@ -1,29 +1,29 @@
 - name: Waiting for SSH connection
-  wait_for_connection:
+  ansible.builtin.wait_for_connection:
 
 - name: Gathering facts
-  setup:
+  ansible.builtin.setup:
 
 - name: Cloning T-Pot install directory
-  git:
+  ansible.builtin.git:
     repo: "https://github.com/telekom-security/tpotce.git"
     dest: /root/tpot
 
 - name: Prepare to set user password
-  set_fact:
+  ansible.builtin.set_fact:
     user_name: "{{ ansible_user }}"
     user_salt: "s0mew1ck3dTpoT"
   no_log: true
 
 - name: Changing password for user {{ user_name }}
-  user:
+  ansible.builtin.user:
    name: "{{ ansible_user }}"
    password: "{{ user_password | password_hash('sha512', user_salt) }}"
    state: present
    shell: /bin/bash
 
 - name: Copy T-Pot configuration file
-  template:
+  ansible.builtin.template:
     src: ../../../../../../iso/installer/tpot.conf.dist
     dest: /root/tpot.conf
     owner: root
@@ -31,15 +31,15 @@
     mode: 0644
 
 - name: Install T-Pot on instance -  be patient, this might take 15 to 30 minutes depending on the connection speed.
-  command: /root/tpot/iso/installer/install.sh --type=auto --conf=/root/tpot.conf
+  ansible.builtin.command: /root/tpot/iso/installer/install.sh --type=auto --conf=/root/tpot.conf
 
 - name: Delete T-Pot configuration file
-  file:
+  ansible.builtin.file:
     path: /root/tpot.conf
     state: absent
 
 - name: Change unattended-upgrades to take default action
-  blockinfile:
+  ansible.builtin.blockinfile:
     dest: /etc/apt/apt.conf.d/50unattended-upgrades
     block: |
       Dpkg::Options {

cloud/ansible/openstack/roles/reboot/tasks/main.yaml

@@ -1,10 +1,10 @@
 - name: Finally rebooting T-Pot
-  command: shutdown -r now
+  ansible.builtin.command: shutdown -r now
   async: 1
   poll: 0
 
 - name: Next login options
-  debug:
+  ansible.builtin.debug:
     msg:
     - "*****    SSH Access:"
     - "*****    ssh {{ ansible_user }}@{{ ansible_host }} -p 64295"

cloud/terraform/.gitignore

@@ -1,2 +0,0 @@
-**/.terraform
-**/terraform.*

cloud/terraform/README.md

@@ -1,7 +1,7 @@
 # T-Pot Terraform
 This [Terraform](https://www.terraform.io/) configuration can be used to launch a virtual machine, bootstrap any dependencies and install T-Pot in a single step.  
 Configuration for Amazon Web Services (AWS) and Open Telekom Cloud (OTC) is currently included.
-This can easily be extended to support other [Terraform providers](https://www.terraform.io/docs/providers/index.html).
+This can easily be extended to support other [Terraform providers](https://registry.terraform.io/browse/providers?category=public-cloud%2Ccloud-automation%2Cinfrastructure).
 
 [Cloud-init](https://cloudinit.readthedocs.io/en/latest/) is used to bootstrap the instance and install T-Pot on startup.