diff --git a/CHANGELOG.md b/CHANGELOG.md index 0bfabca1..32a21a92 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +## 20190701 +- **Reworked Ansible T-Pot Deployment** + - Transitioned from bash script to all Ansible + - Reusable Ansible Playbook for OpenStack clouds + - Example Showcase with our Open Telekom Cloud + - Adaptable for other cloud providers + ## 20190626 - **HPFEEDS Opt-In commandline option** - Pass a hpfeeds config file as a commandline argument diff --git a/README.md b/README.md index 77b8958b..1f68c4bd 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ Furthermore we use the following tools - [Post Install User](#postinstall) - [Post Install Auto](#postinstallauto) - [Cloud Deployments](#cloud) - - [Ansible Deployment on Open Telekom Cloud](#ansible-otc) + - [Ansible](#ansible) - [Terraform](#terraform) - [First Run](#firstrun) - [System Placement](#placement) @@ -118,15 +118,16 @@ Furthermore we use the following tools - **Fix #332** - If T-Pot, opposed to the requirements, does not have full internet access netselect-apt fails to determine the fastest mirror as it needs ICMP and UDP outgoing. Should netselect-apt fail the default mirrors will be used. - **Improve install speed with apt-fast** - - Migrating from a stable base install to Debian (Sid) requires downloading lots of packages. Depending on your geo location the download speed was already improved by introducing netselect-apt to determine the fastest mirror. Wit -h apt-fast the downloads will be even faster by downloading packages not only in parallel but also with multiple connections per package. -- **Added Ansible T-Pot Deployment on Open Telekom Cloud** - - Reusable Ansible Playbooks for all cloud providers - - Example Showcase with our Open Telekom Cloud + - Migrating from a stable base install to Debian (Sid) requires downloading lots of packages. Depending on your geo location the download speed was already improved by introducing netselect-apt to determine the fastest mirror. With apt-fast the downloads will be even faster by downloading packages not only in parallel but also with multiple connections per package. - **HPFEEDS Opt-In commandline option** - Pass a hpfeeds config file as a commandline argument - hpfeeds config is saved in `/data/ews/conf/hpfeeds.cfg` - Update script restores hpfeeds config +- **Ansible T-Pot Deployment** + - Transitioned from bash script to all Ansible + - Reusable Ansible Playbook for OpenStack clouds + - Example Showcase with our Open Telekom Cloud + - Adaptable for other cloud providers # Technical Concept @@ -329,16 +330,17 @@ The installer will start automatically and guide you through the install process ## Cloud Deployments Located in the [`cloud`](cloud) folder. -Currently there is an example with Ansible. -If you would like to contribute you can add other cloud deployments like Terraform, Chef or Puppet. +Currently there are examples with Ansible & Terraform. +If you would like to contribute, you can add other cloud deployments like Chef or Puppet or extend current methods with other cloud providers. - -### Ansible Deployment on Open Telekom Cloud -You can find an Ansible Playbook based automated T-Pot Deployment in the [`cloud/open-telekom-cloud`](cloud/open-telekom-cloud) folder. -The Playbooks in the [`cloud/open-telekom-cloud/ansible`](cloud/open-telekom-cloud/ansible) folder are reusable across all cloud providers (like AWS, Azure, Digital Ocean). -The [`deploy_ansible_otc_t-pot.sh`](cloud/open-telekom-cloud/deploy_ansible_otc_t-pot.sh) script is an example of how it works with our own Public Cloud Offering [Open Telekom Cloud](https://open-telekom-cloud.com/en). -It first creates a new Elastic Cloud Server via the Open Telekom Cloud API and then invokes the Ansible Playbooks to install and configure T-Pot. -You can have a look at the script and easily adapt it for other cloud providers. + +### Ansible Deployment +You can find an [Ansible](https://www.ansible.com/) based T-Pot deployment in the [`cloud/ansible`](cloud/ansible) folder. +The Playbook in the [`cloud/ansible/openstack`](cloud/ansible/openstack) folder is reusable for all OpenStack clouds out of the box. + +It first creates a new server and then installs and configures T-Pot. + +You can have a look at the Playbook and easily adapt the deploy role for other [cloud providers](https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html). ### Terraform Configuration diff --git a/cloud/ansible/.gitignore b/cloud/ansible/.gitignore new file mode 100644 index 00000000..41f83492 --- /dev/null +++ b/cloud/ansible/.gitignore @@ -0,0 +1,2 @@ +# Ansible +*.retry diff --git a/cloud/ansible/README.md b/cloud/ansible/README.md new file mode 100644 index 00000000..07668865 --- /dev/null +++ b/cloud/ansible/README.md @@ -0,0 +1,253 @@ +# T-Pot Ansible + +Here you can find a ready-to-use solution for your automated T-Pot deployment using [Ansible](https://www.ansible.com/). +It consists of an Ansible Playbook with multiple roles, which is reusable for all [OpenStack](https://www.openstack.org/) based clouds (e.g. Open Telekom Cloud, Orange Cloud, Telefonica Open Cloud, OVH) out of the box. +Apart from that you can easily adapt the deploy role to use other [cloud providers](https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html) (e.g. AWS, Azure, Digital Ocean, Google). + +The Playbook first creates a new server and then installs and configures T-Pot. + +This example showcases the deployment on our own OpenStack based Public Cloud Offering [Open Telekom Cloud](https://open-telekom-cloud.com/en). + +# Table of contents +- [Preparation of Ansible Master](#ansible-master) + - [Ansible Installation](#ansible) + - [Agent Forwarding](#agent-forwarding) +- [Preparations in Open Telekom Cloud Console](#preparation) + - [Create new project](#project) + - [Create API user](#api-user) + - [Import Key Pair](#key-pair) + - [Create VPC, Subnet and Security Group](#vpc-subnet-securitygroup) +- [Clone Git Repository](#clone-git) +- [Settings and recommended values](#settings) + - [OpenStack authentication variables](#os-auth) + - [Ansible remote user](#remote-user) + - [Instance settings](#instance-settings) + - [User password](#user-password) + - [Configure `tpot.conf.dist`](#tpot-conf) + - [Optional: Custom `ews.cfg`](#ews-cfg) + - [Optional: Custom HPFEEDS](#hpfeeds) +- [Deploying a T-Pot](#deploy) +- [Further documentation](#documentation) + + +# Preparation of Ansible Master +You can either run the Ansible Playbook locally on your Linux or macOS machine or you can use an ECS (Elastic Cloud Server) on Open Telekom Cloud, which I did. +I used Ubuntu 18.04 for my Ansible Master Server, but other OSes are fine too. +Ansible works over the SSH Port, so you don't have to add any special rules to your Security Group. + + +## Ansible Installation +Example for Ubuntu 18.04: +At first we need to add the repository and install Ansible: +`sudo apt-add-repository --yes --update ppa:ansible/ansible` +`sudo apt install ansible` + +For other OSes and Distros have a look at the official [Ansible Documentation](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html). + + +## Agent Forwarding +Agent Forwarding must be enabled in order to let Ansible do its work. +- On Linux or macOS: + - Create or edit `~/.ssh/config` + - If you run the Ansible Playbook remotely on your Ansible Master Server: + ``` + Host ANSIBLE_MASTER_IP + ForwardAgent yes + ``` + - If you run the Ansible Playbook locally, enable it for all hosts, as this includes newly generated T-Pots: + ``` + Host * + ForwardAgent yes + ``` +- On Windows using Putty for connecting to your Ansible Master Server: +![Putty Agent Forwarding](doc/putty_agent_forwarding.png) + + +# Preparations in Open Telekom Cloud Console +(You can skip this if you have already set up an API account, VPC, Subnet and Security Group) +(Just make sure you know the naming for everything, as you will need it to configure the Ansible variables.) + +Before we can start deploying, we have to prepare the Open Telekom Cloud tenant. +For that, go to the [Web Console](https://auth.otc.t-systems.com/authui/login) and log in with an admin user. + + +## Create new project +I strongly advise you to create a separate project for the T-Pots in your tenant. +In my case I named it `tpot`. + +![Create new project](doc/otc_1_project.gif) + + +## Create API user +The next step is to create a new user account, which is restricted to the project. +This ensures that the API access is limited to that project. + +![Create API user](doc/otc_2_user.gif) + + +## Import Key Pair +:warning: Now log in with the newly created API user account and select your project. + +![Login as API user](doc/otc_3_login.gif) + + +Import your SSH public key. + +![Import SSH Public Key](doc/otc_4_import_key.gif) + + +## Create VPC, Subnet and Security Group +- VPC (Virtual Private Cloud) and Subnet: + +![Create VPC and Subnet](doc/otc_5_vpc_subnet.gif) + +- Security Group: +The configured Security Group should allow all incoming TCP / UDP traffic. +If you want to secure the management interfaces, you can limit the incoming "allow all" traffic to the port range of 1-64000 and allow access to ports > 64000 only from your trusted IPs. + +![Create Security Group](doc/otc_6_sec_group.gif) + + +# Clone Git Repository +Clone the `tpotce` repository to your Ansible Master: +`git clone https://github.com/dtag-dev-sec/tpotce.git` +All Ansible related files are located in the [`cloud/ansible/openstack`](../../cloud/ansible/openstack) folder. + + +# Settings and recommended values +You can configure all aspects of your Elastic Cloud Server and T-Pot before using the Playbook. +The settings are located in the following Ansible vars files: + + +## OpenStack authentication variables +Located at [`openstack/roles/deploy/vars/os_auth.yaml`](openstack/roles/deploy/vars/os_auth.yaml). +Enter your Open Telekom Cloud API user credentials here (username, password, project name, user domain name): +``` +auth_url: https://iam.eu-de.otc.t-systems.com/v3 +username: your_api_user +password: your_password +project_name: eu-de_your_project +os_user_domain_name: OTC-EU-DE-000000000010000XXXXX +``` +You can also perform different authentication methods like sourcing your `.ostackrc` file or using the OpenStack `clouds.yaml` file. +For more information have a look in the [os_server](https://docs.ansible.com/ansible/latest/modules/os_server_module.html) Ansible module documentation. + + +## Ansible remote user +You may have to adjust the `remote_user` in the Ansible Playbook under [`openstack/deploy_tpot.yaml`](openstack/deploy_tpot.yaml) depending on your Debian base image (e.g. on Open Telekom Cloud the default Debian user is `linux`). + + +## Instance settings +Located at [`openstack/roles/deploy/vars/main.yaml`](openstack/roles/deploy/vars/main.yaml). +Here you can customize your virtual machine specifications: + - Specify the region name + - Choose an availability zone. For Open Telekom Cloud reference see [here](https://docs.otc.t-systems.com/en-us/endpoint/index.html). + - Change the OS image (For T-Pot we need Debian 9) + - (Optional) Change the volume size + - Specify your key pair + - (Optional) Change the instance type (flavor) + `s2.medium.8` corresponds to 1 vCPU and 8GB of RAM and is the minimum required flavor. + A full list of Open telekom Cloud flavors can be found [here](https://docs.otc.t-systems.com/en-us/usermanual/ecs/en-us_topic_0035470096.html). + - Specify the security group + - Specify the network ID (For Open Telekom Cloud you can find the ID in the Web Console under `Virtual Private Cloud --> your-vpc --> your-subnet --> Network ID`; In general for OpenStack clouds you can use the `python-openstackclient` to retrieve information about your resources) + +``` +region_name: eu-de +availability_zone: eu-de-03 +image: Standard_Debian_9_latest +volume_size: 128 +key_name: your-KeyPair +flavor: s2.medium.8 +security_groups: your-sg +network: your-network-id +``` + + +## User password +Located at [`openstack/roles/install/vars/main.yaml`](openstack/roles/install/vars/main.yaml). +Here you can set the password for your Debian user (**you should definitely change that**). +``` +user_password: LiNuXuSeRPaSs# +``` + + +## Configure `tpot.conf.dist` +The file is located in [`iso/installer/tpot.conf.dist`](../../iso/installer/tpot.conf.dist). +Here you can choose: + - between the various T-Pot editions + - a username for the web interface + - a password for the web interface (**you should definitely change that**) + +``` +# tpot configuration file +# myCONF_TPOT_FLAVOR=[STANDARD, SENSOR, INDUSTRIAL, COLLECTOR, NEXTGEN] +myCONF_TPOT_FLAVOR='STANDARD' +myCONF_WEB_USER='webuser' +myCONF_WEB_PW='w3b$ecret' +``` + + +## Optional: Custom `ews.cfg` +Enable this by uncommenting the role in the [deploy_tpot.yaml](openstack/deploy_tpot.yaml) playbook. +``` +# - custom_ews +``` + +You can use a custom config file for `ewsposter`. +e.g. when you have your own credentials for delivering data to our [Sicherheitstacho](https://sicherheitstacho.eu/start/main). +You can find the `ews.cfg` template file here: [`openstack/roles/custom_ews/templates/ews.cfg`](openstack/roles/custom_ews/templates/ews.cfg) and adapt it for your needs. + +For setting custom credentials, these settings would be relevant for you (the rest of the file can stay as is): +``` +[MAIN] +... +contact = your_email_address +... + +[EWS] +... +username = your_username +token = your_token +... +``` + + +## Optional: Custom HPFEEDS +Enable this by uncommenting the role in the [deploy_tpot.yaml](openstack/deploy_tpot.yaml) playbook. +``` +# - custom_hpfeeds +``` + +You can specify custom HPFEEDS in [`openstack/roles/custom_hpfeeds/templates/hpfeeds.cfg`](openstack/roles/custom_hpfeeds/templates/hpfeeds.cfg). +That file contains the defaults (turned off) and you can adapt it for your needs, e.g. for SISSDEN: +``` +myENABLE=true +myHOST=hpfeeds.sissden.eu +myPORT=10000 +myCHANNEL=t-pot.events +myCERT=/opt/ewsposter/sissden.pem +myIDENT=your_user +mySECRET=your_secret +myFORMAT=json +``` + + +# Deploying a T-Pot :honey_pot::honeybee: +Now, after configuring everything, we can finally start deploying T-Pots! +Go to the [`openstack`](openstack) folder and run the Ansible Playbook with: +`ansible-playbook deploy_tpot.yaml` +(Yes, it is as easy as that :smile:) + +If you are running on a machine which asks for a sudo password, you can use: +`ansible-playbook --ask-become-pass deploy_tpot.yaml` + +The Playbook will first install required packages on the Ansible Master and then deploy a new server instance. +After that, T-Pot gets installed and configured on the newly created host, optionally custom configs are applied and finally it reboots. + + +# Further documentation +- [Ansible Documentation](https://docs.ansible.com/ansible/latest/) +- [Cloud modules — Ansible Documentation](https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html) +- [os_server – Create/Delete Compute Instances from OpenStack — Ansible Documentation](https://docs.ansible.com/ansible/latest/modules/os_server_module.html) +- [Open Telekom Cloud Help Center](https://docs.otc.t-systems.com/) +- [Open Telekom Cloud API Overview](https://docs.otc.t-systems.com/en-us/api/wp/en-us_topic_0052070394.html) diff --git a/cloud/open-telekom-cloud/doc/otc_1_project.gif b/cloud/ansible/doc/otc_1_project.gif similarity index 100% rename from cloud/open-telekom-cloud/doc/otc_1_project.gif rename to cloud/ansible/doc/otc_1_project.gif diff --git a/cloud/open-telekom-cloud/doc/otc_2_user.gif b/cloud/ansible/doc/otc_2_user.gif similarity index 100% rename from cloud/open-telekom-cloud/doc/otc_2_user.gif rename to cloud/ansible/doc/otc_2_user.gif diff --git a/cloud/open-telekom-cloud/doc/otc_3_login.gif b/cloud/ansible/doc/otc_3_login.gif similarity index 100% rename from cloud/open-telekom-cloud/doc/otc_3_login.gif rename to cloud/ansible/doc/otc_3_login.gif diff --git a/cloud/open-telekom-cloud/doc/otc_4_import_key.gif b/cloud/ansible/doc/otc_4_import_key.gif similarity index 100% rename from cloud/open-telekom-cloud/doc/otc_4_import_key.gif rename to cloud/ansible/doc/otc_4_import_key.gif diff --git a/cloud/open-telekom-cloud/doc/otc_5_vpc_subnet.gif b/cloud/ansible/doc/otc_5_vpc_subnet.gif similarity index 100% rename from cloud/open-telekom-cloud/doc/otc_5_vpc_subnet.gif rename to cloud/ansible/doc/otc_5_vpc_subnet.gif diff --git a/cloud/open-telekom-cloud/doc/otc_6_sec_group.gif b/cloud/ansible/doc/otc_6_sec_group.gif similarity index 100% rename from cloud/open-telekom-cloud/doc/otc_6_sec_group.gif rename to cloud/ansible/doc/otc_6_sec_group.gif diff --git a/cloud/open-telekom-cloud/doc/putty_agent_forwarding.png b/cloud/ansible/doc/putty_agent_forwarding.png similarity index 100% rename from cloud/open-telekom-cloud/doc/putty_agent_forwarding.png rename to cloud/ansible/doc/putty_agent_forwarding.png diff --git a/cloud/ansible/openstack/ansible.cfg b/cloud/ansible/openstack/ansible.cfg new file mode 100644 index 00000000..0e7d2cb7 --- /dev/null +++ b/cloud/ansible/openstack/ansible.cfg @@ -0,0 +1,5 @@ +[defaults] +host_key_checking = false + +[ssh_connection] +scp_if_ssh = true diff --git a/cloud/ansible/openstack/deploy_tpot.yaml b/cloud/ansible/openstack/deploy_tpot.yaml new file mode 100644 index 00000000..88909b17 --- /dev/null +++ b/cloud/ansible/openstack/deploy_tpot.yaml @@ -0,0 +1,25 @@ +- name: Check host prerequisites + hosts: localhost + become: yes + become_user: root + become_method: sudo + roles: + - check + +- name: Deploy instance + hosts: localhost + roles: + - deploy + +- name: Install T-Pot on new instance + hosts: TPOT + remote_user: linux + become: yes + become_user: root + become_method: sudo + gather_facts: no + roles: + - install +# - custom_ews +# - custom_hpfeeds + - reboot diff --git a/cloud/ansible/openstack/roles/check/tasks/main.yaml b/cloud/ansible/openstack/roles/check/tasks/main.yaml new file mode 100644 index 00000000..bc020151 --- /dev/null +++ b/cloud/ansible/openstack/roles/check/tasks/main.yaml @@ -0,0 +1,28 @@ +- name: Install pwgen + package: + name: pwgen + state: present + +- name: Install setuptools + package: + name: python-setuptools + state: present + +- name: Install pip + package: + name: python-pip + state: present + +- name: Install openstacksdk + pip: + name: openstacksdk + +- name: Set fact for agent forwarding + set_fact: + agent_forwarding: "{{ lookup('env','SSH_AUTH_SOCK') }}" + +- name: Check if agent forwarding is enabled + fail: + msg: Please enable agent forwarding to allow Ansible to connect to the remote host! + ignore_errors: yes + when: agent_forwarding == "" diff --git a/cloud/open-telekom-cloud/ansible/roles/custom_ews/tasks/main.yaml b/cloud/ansible/openstack/roles/custom_ews/tasks/main.yaml similarity index 57% rename from cloud/open-telekom-cloud/ansible/roles/custom_ews/tasks/main.yaml rename to cloud/ansible/openstack/roles/custom_ews/tasks/main.yaml index b49d4df4..197403bd 100644 --- a/cloud/open-telekom-cloud/ansible/roles/custom_ews/tasks/main.yaml +++ b/cloud/ansible/openstack/roles/custom_ews/tasks/main.yaml @@ -11,14 +11,3 @@ path: /opt/tpot/etc/tpot.yml insertafter: '/opt/ewsposter/ews.ip' line: ' - /data/ews/conf/ews.cfg:/opt/ewsposter/ews.cfg' - -- name: Copy hpfeeds configuration file - template: - src: ../templates/hpfeeds.cfg - dest: /data/ews/conf - owner: root - group: root - mode: 0644 - -- name: Applying hpfeeds settings - command: /opt/tpot/bin/hpfeeds_optin.sh --conf=/data/ews/conf/hpfeeds.cfg diff --git a/cloud/open-telekom-cloud/ansible/roles/custom_ews/templates/ews.cfg b/cloud/ansible/openstack/roles/custom_ews/templates/ews.cfg similarity index 79% rename from cloud/open-telekom-cloud/ansible/roles/custom_ews/templates/ews.cfg rename to cloud/ansible/openstack/roles/custom_ews/templates/ews.cfg index 2dfc89e6..a775d04b 100644 --- a/cloud/open-telekom-cloud/ansible/roles/custom_ews/templates/ews.cfg +++ b/cloud/ansible/openstack/roles/custom_ews/templates/ews.cfg @@ -35,7 +35,7 @@ jsondir = /data/ews/json/ [GLASTOPFV3] glastopfv3 = true -nodeid = glastopfv3-{{ HPNAME }} +nodeid = glastopfv3-{{ ansible_hostname }} sqlitedb = /data/glastopf/db/glastopf.db malwaredir = /data/glastopf/data/files/ @@ -59,18 +59,18 @@ malwaredir = [COWRIE] cowrie = true -nodeid = cowrie-{{ HPNAME }} +nodeid = cowrie-{{ ansible_hostname }} logfile = /data/cowrie/log/cowrie.json [DIONAEA] dionaea = true -nodeid = dionaea-{{ HPNAME }} +nodeid = dionaea-{{ ansible_hostname }} malwaredir = /data/dionaea/binaries/ sqlitedb = /data/dionaea/log/dionaea.sqlite [HONEYTRAP] honeytrap = true -nodeid = honeytrap-{{ HPNAME }} +nodeid = honeytrap-{{ ansible_hostname }} newversion = true payloaddir = /data/honeytrap/attacks/ attackerfile = /data/honeytrap/log/attacker.log @@ -83,55 +83,55 @@ targetip = [EMOBILITY] eMobility = false -nodeid = emobility-{{ HPNAME }} +nodeid = emobility-{{ ansible_hostname }} logfile = /data/emobility/log/centralsystemEWS.log [CONPOT] conpot = true -nodeid = conpot-{{ HPNAME }} +nodeid = conpot-{{ ansible_hostname }} logfile = /data/conpot/log/conpot*.json [ELASTICPOT] elasticpot = true -nodeid = elasticpot-{{ HPNAME }} +nodeid = elasticpot-{{ ansible_hostname }} logfile = /data/elasticpot/log/elasticpot.log [SURICATA] suricata = true -nodeid = suricata-{{ HPNAME }} +nodeid = suricata-{{ ansible_hostname }} logfile = /data/suricata/log/eve.json [MAILONEY] mailoney = true -nodeid = mailoney-{{ HPNAME }} +nodeid = mailoney-{{ ansible_hostname }} logfile = /data/mailoney/log/commands.log [RDPY] rdpy = true -nodeid = rdpy-{{ HPNAME }} +nodeid = rdpy-{{ ansible_hostname }} logfile = /data/rdpy/log/rdpy.log [VNCLOWPOT] vnclowpot = true -nodeid = vnclowpot-{{ HPNAME }} +nodeid = vnclowpot-{{ ansible_hostname }} logfile = /data/vnclowpot/log/vnclowpot.log [HERALDING] heralding = true -nodeid = heralding-{{ HPNAME }} +nodeid = heralding-{{ ansible_hostname }} logfile = /data/heralding/log/auth.csv [CISCOASA] ciscoasa = true -nodeid = ciscoasa-{{ HPNAME }} +nodeid = ciscoasa-{{ ansible_hostname }} logfile = /data/ciscoasa/log/ciscoasa.log [TANNER] tanner = true -nodeid = tanner-{{ HPNAME }} +nodeid = tanner-{{ ansible_hostname }} logfile = /data/tanner/log/tanner_report.json [GLUTTON] glutton = true -nodeid = glutton-{{ HPNAME }} +nodeid = glutton-{{ ansible_hostname }} logfile = /data/glutton/log/glutton.log diff --git a/cloud/ansible/openstack/roles/custom_hpfeeds/tasks/main.yaml b/cloud/ansible/openstack/roles/custom_hpfeeds/tasks/main.yaml new file mode 100644 index 00000000..421d1ed6 --- /dev/null +++ b/cloud/ansible/openstack/roles/custom_hpfeeds/tasks/main.yaml @@ -0,0 +1,10 @@ +- name: Copy hpfeeds configuration file + template: + src: ../templates/hpfeeds.cfg + dest: /data/ews/conf + owner: root + group: root + mode: 0644 + +- name: Applying hpfeeds settings + command: /opt/tpot/bin/hpfeeds_optin.sh --conf=/data/ews/conf/hpfeeds.cfg diff --git a/cloud/open-telekom-cloud/ansible/roles/custom_ews/templates/hpfeeds.cfg b/cloud/ansible/openstack/roles/custom_hpfeeds/templates/hpfeeds.cfg similarity index 100% rename from cloud/open-telekom-cloud/ansible/roles/custom_ews/templates/hpfeeds.cfg rename to cloud/ansible/openstack/roles/custom_hpfeeds/templates/hpfeeds.cfg diff --git a/cloud/ansible/openstack/roles/deploy/tasks/main.yaml b/cloud/ansible/openstack/roles/deploy/tasks/main.yaml new file mode 100644 index 00000000..78f26742 --- /dev/null +++ b/cloud/ansible/openstack/roles/deploy/tasks/main.yaml @@ -0,0 +1,34 @@ +- name: Create T-Pot name + shell: echo t-pot-ansible-$(pwgen -ns 6 -1) + register: tpot_name + +- name: Import OpenStack authentication variables + include_vars: + file: roles/deploy/vars/os_auth.yaml + +- name: Launch an instance + os_server: + auth: + auth_url: "{{ auth_url }}" + username: "{{ username }}" + password: "{{ password }}" + project_name: "{{ project_name }}" + os_user_domain_name: "{{ os_user_domain_name }}" + name: "{{ tpot_name.stdout }}" + region_name: "{{ region_name }}" + availability_zone: "{{ availability_zone }}" + image: "{{ image }}" + boot_from_volume: yes + volume_size: "{{ volume_size }}" + key_name: "{{ key_name }}" + timeout: 200 + flavor: "{{ flavor }}" + security_groups: "{{ security_groups }}" + network: "{{ network }}" + register: tpot + +- name: Add instance to inventory + add_host: + hostname: "{{ tpot_name.stdout }}" + ansible_host: "{{ tpot.server.public_v4 }}" + groups: TPOT diff --git a/cloud/ansible/openstack/roles/deploy/vars/main.yaml b/cloud/ansible/openstack/roles/deploy/vars/main.yaml new file mode 100644 index 00000000..c0697442 --- /dev/null +++ b/cloud/ansible/openstack/roles/deploy/vars/main.yaml @@ -0,0 +1,8 @@ +region_name: eu-de +availability_zone: eu-de-03 +image: Standard_Debian_9_latest +volume_size: 128 +key_name: your-KeyPair +flavor: s2.medium.8 +security_groups: your-sg +network: your-network-id diff --git a/cloud/ansible/openstack/roles/deploy/vars/os_auth.yaml b/cloud/ansible/openstack/roles/deploy/vars/os_auth.yaml new file mode 100644 index 00000000..fdb1a29b --- /dev/null +++ b/cloud/ansible/openstack/roles/deploy/vars/os_auth.yaml @@ -0,0 +1,5 @@ +auth_url: https://iam.eu-de.otc.t-systems.com/v3 +username: your_api_user +password: your_password +project_name: eu-de_your_project +os_user_domain_name: OTC-EU-DE-000000000010000XXXXX diff --git a/cloud/open-telekom-cloud/ansible/roles/install/tasks/main.yaml b/cloud/ansible/openstack/roles/install/tasks/main.yaml similarity index 59% rename from cloud/open-telekom-cloud/ansible/roles/install/tasks/main.yaml rename to cloud/ansible/openstack/roles/install/tasks/main.yaml index 9e4fd51a..f6e65e2f 100644 --- a/cloud/open-telekom-cloud/ansible/roles/install/tasks/main.yaml +++ b/cloud/ansible/openstack/roles/install/tasks/main.yaml @@ -3,28 +3,29 @@ delay: 30 timeout: 300 -- name: Gathering Facts +- name: Gathering facts setup: -- name: Cloning t-pot install directory +- name: Cloning T-Pot install directory git: - repo: 'https://github.com/dtag-dev-sec/tpotce.git' + repo: "https://github.com/dtag-dev-sec/tpotce.git" dest: /root/tpot - name: Prepare to set user password set_fact: - user_password: "{{ lookup('env', 'LINUX_PASS') }}" - user_salt: 's0mew1ck3dTpoT' + user_name: "{{ ansible_user }}" + user_password: "{{ user_password }}" + user_salt: "s0mew1ck3dTpoT" -- name: Changing password for user linux to {{ user_password }} +- name: Changing password for user {{ user_name }} to {{ user_password }} user: - name: "linux" + name: "{{ ansible_user }}" password: "{{ user_password | password_hash('sha512', user_salt) }}" state: present shell: /bin/bash update_password: always -- name: Copy t-pot configuration file +- name: Copy T-Pot configuration file template: src: ../../../../../../iso/installer/tpot.conf.dist dest: /root/tpot.conf @@ -32,10 +33,10 @@ group: root mode: 0644 -- name: Install t-pot on ECS - be patient, this might take 15 to 30 minutes depending on the connection speed. No further output is given. +- name: Install T-Pot on instance - be patient, this might take 15 to 30 minutes depending on the connection speed. No further output is given. command: /root/tpot/iso/installer/install.sh --type=auto --conf=/root/tpot.conf -- name: Delete t-pot configuration file +- name: Delete T-Pot configuration file file: path: /root/tpot.conf state: absent diff --git a/cloud/ansible/openstack/roles/install/vars/main.yaml b/cloud/ansible/openstack/roles/install/vars/main.yaml new file mode 100644 index 00000000..ce234a97 --- /dev/null +++ b/cloud/ansible/openstack/roles/install/vars/main.yaml @@ -0,0 +1 @@ +user_password: LiNuXuSeRPaSs# diff --git a/cloud/ansible/openstack/roles/reboot/tasks/main.yaml b/cloud/ansible/openstack/roles/reboot/tasks/main.yaml new file mode 100644 index 00000000..1490ebfb --- /dev/null +++ b/cloud/ansible/openstack/roles/reboot/tasks/main.yaml @@ -0,0 +1,15 @@ +- name: Finally rebooting T-Pot in one minute + shell: /sbin/shutdown -r -t 1 + become: true + +- name: Next login options + debug: + msg: + - "***** SSH Access:" + - "***** ssh {{ ansible_user }}@{{ ansible_host }} -p 64295" + - "" + - "***** Web UI:" + - "***** https://{{ ansible_host }}:64297" + - "" + - "***** Admin UI:" + - "***** https://{{ ansible_host }}:64294" diff --git a/cloud/open-telekom-cloud/.ecs_settings.sh b/cloud/open-telekom-cloud/.ecs_settings.sh deleted file mode 100644 index bb4a8423..00000000 --- a/cloud/open-telekom-cloud/.ecs_settings.sh +++ /dev/null @@ -1,15 +0,0 @@ -# Set password for user linux -linuxpass=LiNuXuSeRPaSs# - -# Custom EWS config -custom_ews=false - -# Set ECS related stuff -instance=s2.medium.8 -imagename=Standard_Debian_9_latest -subnet=your-subnet -vpcname=your-vpc -secgroup=your-sg -keyname=your-KeyPair -disksize=128 -az=eu-de-03 diff --git a/cloud/open-telekom-cloud/.gitignore b/cloud/open-telekom-cloud/.gitignore deleted file mode 100644 index 8d8f15eb..00000000 --- a/cloud/open-telekom-cloud/.gitignore +++ /dev/null @@ -1,11 +0,0 @@ -# Ansible -*.retry - -# Generated hosts -hosts/ - -# Cloned git repository -otc-tools/ - -# All log files -*.log diff --git a/cloud/open-telekom-cloud/.otc_env.sh b/cloud/open-telekom-cloud/.otc_env.sh deleted file mode 100644 index 337617c1..00000000 --- a/cloud/open-telekom-cloud/.otc_env.sh +++ /dev/null @@ -1,5 +0,0 @@ -export OS_USERNAME=your_api_user -export OS_PASSWORD=your_password -export OS_USER_DOMAIN_NAME=OTC-EU-DE-000000000010000XXXXX -export OS_PROJECT_NAME=eu-de_your_project -export OS_AUTH_URL=https://iam.eu-de.otc.t-systems.com/v3 diff --git a/cloud/open-telekom-cloud/README.md b/cloud/open-telekom-cloud/README.md deleted file mode 100644 index e40acdd5..00000000 --- a/cloud/open-telekom-cloud/README.md +++ /dev/null @@ -1,228 +0,0 @@ -# Ansible T-Pot Deployment on Open Telekom Cloud :cloud: - -Here you can find a ready-to-use solution for your automated T-Pot deployment using [Ansible](https://www.ansible.com/). -It consists of multiple Ansible Playbooks, which can be reused across all Cloud Providers (like AWS, Azure, Digital Ocean). -This example showcases the deployment on our own Public Cloud Offering [Open Telekom Cloud](https://open-telekom-cloud.com/en). - -# Table of contents -- [Installation of Ansible Master](#installation) - - [Packages](#packages) - - [Agent Forwarding](#agent-forwarding) -- [Preparations in Open Telekom Cloud Console](#preparation) - - [Create new project](#project) - - [Create API user](#api-user) - - [Import Key Pair](#key-pair) - - [Create VPC, Subnet and Security Group](#vpc-subnet-securitygroup) -- [Clone Git Repository](#clone-git) -- [Settings and recommended values](#settings) - - [Configure `.otc_env.sh`](#otc-env) - - [Configure `.ecs_settings.sh`](#ecs-settings) - - [Configure `tpot.conf.dist`](#tpot-conf) - - [Optional: Custom `ews.cfg` and HPFEEDS](#ews-hpfeeds) -- [Deploying a T-Pot](#deploy) -- [Further documentation](#documentation) - - -# Installation of Ansible Master -You can either run the deploy script locally on your Linux or MacOS machine or you can use an ECS (Elastic Cloud Server) on Open Telekom Cloud, which I did. -I used Ubuntu 18.04 for my Ansible Master Server, but other OSes are fine too. -Ansible works over the SSH Port, so you don't have to add any special rules to you Security Group. - - -## Packages -At first we need to add the repository and install Ansible: -`sudo apt-add-repository --yes --update ppa:ansible/ansible` -`sudo apt install ansible` - -Also we need **pwegen** (for creating T-Pot names) and **jq** (a JSON processor): -`sudo apt install pwgen jq` - - -## Agent Forwarding -Agent forwarding must be enabled in order to let Ansible do its work. -- On Linux or MacOS: - - Create or edit `~/.ssh/config` - - If you execute the script remotely on your Ansible Master Server: - ``` - Host ANSIBLE_MASTER_IP - ForwardAgent yes - ``` - - If you execute the script locally, enable it for all Hosts, as this includes newly generated T-Pots: - ``` - Host * - ForwardAgent yes - ``` -- On Windows using Putty: -![Putty Agent Forwarding](doc/putty_agent_forwarding.png) - - -# Preparations in Open Telekom Cloud Console -(You can skip this if you have already set up an API account, VPC and ...) -(Just make sure you know the naming for everything, as you will need it to configure the script.) - -Before we can start deploying, we have to prepare the Open Telekom Cloud Tennant. -For that, go to the [Web Console](https://auth.otc.t-systems.com/authui/login) and log in with an admin user. - - -## Create new project -I strongly advise you, to create a separate project for the T-Pots in your tennant. -In my case I named it `tpot`. - -![Create new project](doc/otc_1_project.gif) - - -## Create API user -The next step is to create a new user account, which is restricted to the project. -This ensures that the API access is limited to that project. - -![Create API user](doc/otc_2_user.gif) - - -## Import Key Pair -:warning: Now log in with the newly created user account and select your project. - -![Login as API user](doc/otc_3_login.gif) - - -Import your SSH public key. - -![Import SSH Public Key](doc/otc_4_import_key.gif) - - -## Create VPC, Subnet and Security Group -- VPC (Virtual Private Cloud) and Subnet: - -![Create VPC and Subnet](doc/otc_5_vpc_subnet.gif) - -- Security Group: -The configured Security Group should allow all incoming TCP / UDP traffic. -If you want to secure the management interfaces, you can limit the incoming "allow all" traffic to the port range of 1-64000 and allow access to ports > 64000 only from your trusted IPs. - -![Create Security Group](doc/otc_6_sec_group.gif) - - -# Clone Git Repository -Clone the `tpotce` repository to your Ansible Master: -`git clone https://github.com/dtag-dev-sec/tpotce.git` -All Ansible and automatic deployment related files are located in the [`cloud/open-telekom-cloud`](../../cloud/open-telekom-cloud) folder. - - -# Settings and recommended values -You can configure all aspects of your ECS and T-Pot before using the script. -The settings are located in the following files: - - -## Configure `.otc_env.sh` -Enter your Open Telekom Cloud API user credentials here (username, password, tennant-ID, project name): -``` -export OS_USERNAME=your_api_user -export OS_PASSWORD=your_password -export OS_USER_DOMAIN_NAME=OTC-EU-DE-000000000010000XXXXX -export OS_PROJECT_NAME=eu-de_your_project -export OS_AUTH_URL=https://iam.eu-de.otc.t-systems.com/v3 -``` - - -## Configure `.ecs_settings.sh` -Here you can customize your Elastic Cloud Server (ECS): - - Password for the user `linux` (**you should definitely change that**) - You may have to adjust the `remote_user` in the Ansible Playbooks under [ansible](ansible) if you are using a normal/default Debian base image - - (Optional) For using a custom `ews.cfg` set to `true`; See here: [Optional: Custom `ews.cfg`](#ews-cfg) - - (Optional) Change the instance type (flavor) of the ECS. - `s2.medium.8` corresponds to 1 vCPU and 8GB of RAM and is the minimum required flavor. - A full list of flavors can be found [here](https://docs.otc.t-systems.com/en-us/usermanual/ecs/en-us_topic_0035470096.html). - - Change the OS (Don't touch; for T-Pot we need Debian 9) - - Specify the VPC, Subnet, Security Group and Key Pair you created before - - (Optional) Change the disk size - - You can choose from multiple Availibility Zones (AZ). For reference see [here](https://docs.otc.t-systems.com/en-us/endpoint/index.html). - -``` -# Set password for user linux -linuxpass=LiNuXuSeRPaSs# - -# Custom EWS config -custom_ews=false - -# Set ECS related stuff -instance=s2.medium.8 -imagename=Standard_Debian_9_latest -subnet=your-subnet -vpcname=your-vpc -secgroup=your-sg -keyname=your-KeyPair -disksize=128 -az=eu-de-03 -``` - - -## Configure `tpot.conf.dist` -The file is located in [`iso/installer/tpot.conf.dist`](../../iso/installer/tpot.conf.dist). -Here you can choose: - - between the various T-Pot editions - - a username for the web interface - - a password for the web interface (**you should definitely change that**) - -``` -# tpot configuration file -# myCONF_TPOT_FLAVOR=[STANDARD, SENSOR, INDUSTRIAL, COLLECTOR, NEXTGEN, LEGACY] -myCONF_TPOT_FLAVOR='STANDARD' -myCONF_WEB_USER='webuser' -myCONF_WEB_PW='w3b$ecret' -``` - - -## Optional: Custom `ews.cfg` and HPFEEDS -To enable these features, set `custom_ews=true` in `.ecs_settings.sh`; See here: [Configure `.ecs_settings.sh`](#ecs-settings) - -### ews.cfg -You can use a custom config file for `ewsposter`. -e.g. when you have your own credentials for delivering data to our [Sicherheitstacho](https://sicherheitstacho.eu/start/main). -You can find the `ews.cfg` template file here: [`ansible/roles/custom_ews/templates/ews.cfg`](ansible/roles/custom_ews/templates/ews.cfg) and adapt it for your needs. - -For setting custom credentials, these settings would be relevant for you (the rest of the file can stay as is): -``` -[MAIN] -... -contact = your_email_address -... - -[EWS] -... -username = your_username -token = your_token -... -``` - -### HPFEEDS -You can also specify HPFEEDS in [`ansible/roles/custom_ews/templates/hpfeeds.cfg`](ansible/roles/custom_ews/templates/hpfeeds.cfg). -That file constains the defaults (turned off) and you can adapt it for your needs, e.g. for SISSDEN: -``` -myENABLE=true -myHOST=hpfeeds.sissden.eu -myPORT=10000 -myCHANNEL=t-pot.events -myCERT=/opt/ewsposter/sissden.pem -myIDENT=your_user -mySECRET=your_secret -myFORMAT=json -``` - - - -# Deploying a T-Pot :honey_pot::honeybee: -Now, after configuring everything, we can finally start deploying T-Pots: -`./deploy_ansible_otc_t-pot.sh` -(Yes, it is as easy as that :smile:) - -The script will first create an Open Telekom Cloud ECS via the API. -After that, the Ansible Playbooks are executed on the newly created Host to install the T-Pot and configure everything. - -You can see the progress of every step in the console output. -If something should go wrong, you will be provided with an according error message, that you can hopefully act upon and retry. - - -# Further documentation -- [Ansible Documentation](https://docs.ansible.com/ansible/latest/) -- [Open Telekom Cloud Help Center](https://docs.otc.t-systems.com/) -- [Open Telekom Cloud API Overview](https://docs.otc.t-systems.com/en-us/api/wp/en-us_topic_0052070394.html) -- [otc-tools](https://github.com/OpenTelekomCloud/otc-tools) on GitHub diff --git a/cloud/open-telekom-cloud/ansible/custom_ews.yaml b/cloud/open-telekom-cloud/ansible/custom_ews.yaml deleted file mode 100644 index a9fc39e0..00000000 --- a/cloud/open-telekom-cloud/ansible/custom_ews.yaml +++ /dev/null @@ -1,10 +0,0 @@ -# This playbook sets a custom EWS config on the T-Pot - -- hosts: TPOT - remote_user: linux - become: yes - become_user: root - become_method: sudo - - roles: - - custom_ews diff --git a/cloud/open-telekom-cloud/ansible/install.yaml b/cloud/open-telekom-cloud/ansible/install.yaml deleted file mode 100644 index e0fe70ca..00000000 --- a/cloud/open-telekom-cloud/ansible/install.yaml +++ /dev/null @@ -1,13 +0,0 @@ -# This playbook deploys a T-Pot - -- hosts: TPOT - remote_user: linux - become: yes - become_user: root - become_method: sudo - gather_facts: no - - roles: - - install - - diff --git a/cloud/open-telekom-cloud/ansible/reboot.yaml b/cloud/open-telekom-cloud/ansible/reboot.yaml deleted file mode 100644 index f4fc4f3f..00000000 --- a/cloud/open-telekom-cloud/ansible/reboot.yaml +++ /dev/null @@ -1,12 +0,0 @@ -# This playbook reboots a T-Pot - -- hosts: TPOT - remote_user: linux - become: yes - become_user: root - become_method: sudo - - tasks: - - name: Finally rebooting t-pot in one minute - make sure your next login is on port 64295 or via https:// on port 64297 - shell: /sbin/shutdown -r -t 1 - become: true diff --git a/cloud/open-telekom-cloud/deploy_ansible_otc_t-pot.sh b/cloud/open-telekom-cloud/deploy_ansible_otc_t-pot.sh deleted file mode 100755 index 9f3756f9..00000000 --- a/cloud/open-telekom-cloud/deploy_ansible_otc_t-pot.sh +++ /dev/null @@ -1,122 +0,0 @@ -#!/bin/bash - -# Check if required packages are installed -if ! hash ansible 2>/dev/null; then - echo "### Package 'ansible' is missing. Please install it with:" - echo " sudo apt-add-repository --yes --update ppa:ansible/ansible" - echo " sudo apt install ansible" - exit 1 -fi - -if ! hash pwgen 2>/dev/null; then - echo "### Package 'pwgen' is missing. Please install it with:" - echo " sudo apt install pwgen" - exit 1 -fi - -if ! hash jq 2>/dev/null; then - echo "### Package 'jq' is missing. Please install it with:" - echo " sudo apt install jq" - exit 1 -fi - -# Check for Agent Forwarding -if ! printenv | grep SSH_AUTH_SOCK > /dev/null; then - echo "### Agent forwarding seems to be disabled." - echo "### In order to let Ansible do its work, please enable it." - exit 1 -fi - -# Import ECS settings -source .ecs_settings.sh - -# Import OTC authentication credentials -source .otc_env.sh - -# Password is later used by Ansible -export LINUX_PASS=$linuxpass - -# Ignore ssh host keys as they are new anyway -export ANSIBLE_HOST_KEY_CHECKING=False - -# Create hosts directory -mkdir -p hosts - -# Create random ID -HPNAME=t-pot-otc-$(pwgen -ns 6 -1) - -# Get otc-tools -echo "### Cloning otc-tools..." -git clone https://github.com/OpenTelekomCloud/otc-tools.git 2>/dev/null - -# Create ECS via OTC API -echo "### Creating new ECS host via OTC API..." -./otc-tools/otc.sh ecs create \ - --instance-type $instance\ - --instance-name $HPNAME\ - --image-name $imagename\ - --subnet-name $subnet\ - --vpc-name $vpcname\ - --security-group-name $secgroup\ - --admin-pass $linuxpass\ - --key-name $keyname\ - --public true\ - --disksize $disksize\ - --disktype SATA\ - --az $az\ - --wait \ -2> otc_tools.log - -if [ $? -eq 0 ]; then - - if [ "$(uname)" == "Darwin" ]; then - PUBIP=$(./otc-tools/otc.sh ecs list 2>/dev/null | grep $HPNAME|cut -d "," -f2 |cut -d "\"" -f 2) - else - PUBIP=$(./otc-tools/otc.sh ecs list 2>/dev/null | grep $HPNAME|cut -d " " -f17) - fi - - echo "[TPOT]" > ./hosts/$HPNAME - echo $PUBIP HPNAME=$HPNAME>> ./hosts/$HPNAME - echo "### NEW HOST $HPNAME ON IP $PUBIP" - - ansible-playbook -i ./hosts/$HPNAME ./ansible/install.yaml - - if [ $custom_ews = true ]; then - - ansible-playbook -i ./hosts/$HPNAME ./ansible/custom_ews.yaml - - fi - - ansible-playbook -i ./hosts/$HPNAME ./ansible/reboot.yaml - - echo "***********************************************" - echo "***** SSH TO TARGET: " - echo "***** ssh linux@$PUBIP -p 64295" - echo "***********************************************" - -else - - if grep '401 Unauthorized' otc_tools.log > /dev/null; then - echo "### API username or password is incorrect" - elif grep 'Flavor' otc_tools.log > /dev/null; then - echo "### Specified ECS Flavor not found" - elif grep 'No image found by name' otc_tools.log > /dev/null; then - echo "### Specified Image not found" - elif grep 'No subnet found by name' otc_tools.log > /dev/null; then - echo "### Specified Subnet not found" - elif grep 'No VPC found by name' otc_tools.log > /dev/null; then - echo "### Specified VPC not found" - elif grep 'No security-group found by name' otc_tools.log > /dev/null; then - echo "### Specified Security Group not found" - elif grep 'Invalid key_name provided' otc_tools.log > /dev/null; then - echo "### Specified Key Pair not found" - elif grep 'availability_zone' otc_tools.log > /dev/null; then - echo "### Specified Availability Zone not found" - elif grep 'quota' otc_tools.log > /dev/null; then - echo "### Quota exceeded. Please check your available quotas online" - echo "### You can either delete unused resources or apply for a higher quota" - fi - - echo "### ECS creation unsuccessful. Aborting..." - -fi