Merge pull request #371 from TheHADILP/hpfeeds-cmd

HPFEEDS cmdline option

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 20190626
+- **HPFEEDS Opt-In commandline option**
+  - Pass a hpfeeds config file as a commandline argument
+  - hpfeeds config is saved in `/data/ews/conf/hpfeeds.cfg`
+  - Update script restores hpfeeds config
+
 ## 20190604
 - **Finalize Fatt support**
   - Build visualizations, searches, dashboards

README.md

@@ -122,6 +122,10 @@ h apt-fast the downloads will be even faster by downloading packages not only in
 - **Added Ansible T-Pot Deployment on Open Telekom Cloud**
   - Reusable Ansible Playbooks for all cloud providers
   - Example Showcase with our Open Telekom Cloud
+- **HPFEEDS Opt-In commandline option**
+  - Pass a hpfeeds config file as a commandline argument
+  - hpfeeds config is saved in `/data/ews/conf/hpfeeds.cfg`
+  - Update script restores hpfeeds config
 
 <a name="concept"></a>
 # Technical Concept
@@ -375,11 +379,13 @@ T-Pot requires outgoing git, http, https connections for updates (Debian, Docker
 # Updates
 For the ones of you who want to live on the bleeding edge of T-Pot development we introduced an update feature which will allow you to update all T-Pot relevant files to be up to date with the T-Pot master branch.
 **If you made any relevant changes to the T-Pot relevant config files make sure to create a backup first.**
+
 The Update script will:
- - **merciless** overwrite local changes to be in sync with the T-Pot master branch
+ - **mercilessly** overwrite local changes to be in sync with the T-Pot master branch
  - upgrade the system to the packages available in Debian (Sid)
  - update all resources to be in-sync with the T-Pot master branch
  - ensure all T-Pot relevant system files will be patched / copied into the original T-Pot state
+ - restore your custom ews.cfg and HPFEED settings from `/data/ews/conf`
 
 You simply run the update script:
 ```
@@ -467,8 +473,17 @@ We encourage you not to disable the data submission as it is the main purpose of
 
 <a name="hpfeeds-optin"></a>
 ## Opt-In HPFEEDS Data Submission
-As an Opt-In it is now possible to also share T-Pot data with 3rd party HPFEEDS brokers, such as [SISSDEN](https://sissden.eu).
+As an Opt-In it is now possible to also share T-Pot data with 3rd party HPFEEDS brokers, such as [SISSDEN](https://sissden.eu).  
-If you want to share your T-Pot data you simply have to register an account with a 3rd party broker with its own benefits towards the community. Once registered you will receive your credentials to share events with the broker. In T-Pot you simply run `hpfeeds_optin.sh` which will ask for your credentials, in case of SISSDEN this is just `Ident` and `Secret`, everything else is pre-configured. It will automatically update `/opt/tpot/etc/tpot.yml` to deliver events to your desired broker.
+If you want to share your T-Pot data you simply have to register an account with a 3rd party broker with its own benefits towards the community. Once registered you will receive your credentials to share events with the broker. In T-Pot you simply run `hpfeeds_optin.sh` which will ask for your credentials, in case of SISSDEN this is just `Ident` and `Secret`, everything else is pre-configured.  
+It will automatically update `/opt/tpot/etc/tpot.yml` to deliver events to your desired broker.
+
+The script can accept a config file as an argument, e.g. `./hpfeeds_optin.sh --conf=hpfeeds.cfg`
+
+Your current config will also be stored in `/data/ews/conf/hpfeeds.cfg` where you can review or change it.  
+Be sure to apply any changes by running `./hpfeeds_optin.sh --conf=/data/ews/conf/hpfeeds.cfg`.  
+No worries: You old config gets backed up in `/data/ews/conf/hpfeeds.cfg.old`
+
+Of course you can also rerun the `hpfeeds_optin.sh` script to change and apply your settings interactively.
 
 <a name="roadmap"></a>
 # Roadmap

bin/hpfeeds_optin.sh

@@ -67,10 +67,28 @@ mySECRET="secret"
 myFORMAT="json"
 }
 
+function fuWRITETOFILE () {
+if [ -f '/data/ews/conf/hpfeeds.cfg' ]; then
+  echo "Creating backup of current config in /data/ews/conf/hpfeeds.cfg.old"
+  mv /data/ews/conf/hpfeeds.cfg /data/ews/conf/hpfeeds.cfg.old
+fi
+echo "Storing new config in /data/ews/conf/hpfeeds.cfg"
+cat >> /data/ews/conf/hpfeeds.cfg <<EOF
+myENABLE=$myENABLE
+myHOST=$myHOST
+myPORT=$myPORT
+myCHANNEL=$myCHANNEL
+myIDENT=$myIDENT
+mySECRET=$mySECRET
+myCERT=$myCERT
+myFORMAT=$myFORMAT
+EOF
+}
+
 function fuAPPLY () {
 echo "Now stopping T-Pot ..."
 systemctl stop tpot
-echo "Applying your settings ... "
+echo "Applying your settings to tpot.yml ... "
 sed --follow-symlinks -i "s/EWS_HPFEEDS_ENABLE.*/EWS_HPFEEDS_ENABLE=${myENABLE}/g" "$myTPOTYMLFILE"
 sed --follow-symlinks -i "s/EWS_HPFEEDS_HOST.*/EWS_HPFEEDS_HOST=${myHOST}/g" "$myTPOTYMLFILE"
 sed --follow-symlinks -i "s/EWS_HPFEEDS_PORT.*/EWS_HPFEEDS_PORT=${myPORT}/g" "$myTPOTYMLFILE"
@@ -81,10 +99,19 @@ sed --follow-symlinks -i "s/EWS_HPFEEDS_SECRET.*/EWS_HPFEEDS_SECRET=${mySECRET}/
 sed --follow-symlinks -i "s/EWS_HPFEEDS_FORMAT.*/EWS_HPFEEDS_FORMAT=${myFORMAT}/g" "$myTPOTYMLFILE"
 echo "Now starting T-Pot ..."
 systemctl start tpot
-echo "You can always change or review your settings in the ewsposter section of $myTPOTYMLFILE"
+echo "You can always change or review your settings in /data/ews/conf/hpfeeds.cfg and apply changes by"
+echo "running \"./hpfeeds_optin.sh --conf=/data/ews/conf/hpfeeds.cfg\""
 echo "Done."
 }
 
+# Check for cmdline argument and parse config file
+filename=$(echo $@ | cut -d= -f2)
+if [ $# == 1 ] && echo $@ | grep '\-\-conf=' > /dev/null && [ ! -z $filename ] && [ -f $filename ]
+  then
+    source $filename
+else
+
+# Proceed with interactive setup when no config file is found
 echo "HPFEEDS Delivery Opt-In for T-Pot"
 echo "---------------------------------"
 echo "By running this script you agree to share your data with a 3rd party and agree to their corresponding sharing terms."
@@ -120,5 +147,7 @@ while [ 1 != 2 ]
           ;;
       esac
 done
-fuAPPLY
 
+fi
+fuWRITETOFILE
+fuAPPLY

cloud/open-telekom-cloud/README.md

@@ -18,8 +18,7 @@ This example showcases the deployment on our own Public Cloud Offering [Open Tel
   - [Configure `.otc_env.sh`](#otc-env)
   - [Configure `.ecs_settings.sh`](#ecs-settings)
   - [Configure `tpot.conf.dist`](#tpot-conf)
-  - [Optional: Custom `ews.cfg`](#ews-cfg)
+  - [Optional: Custom `ews.cfg` and HPFEEDS](#ews-hpfeeds)
-  - [Optional: Configure `.hpfeeds_settings.sh`](#hpfeeds)
 - [Deploying a T-Pot](#deploy)
 - [Further documentation](#documentation)
 
@@ -171,11 +170,12 @@ myCONF_WEB_USER='webuser'
 myCONF_WEB_PW='w3b$ecret'
 ```
 
-<a name="ews-cfg"></a>
+<a name="ews-hpfeeds"></a>
-## Optional: Custom `ews.cfg`
+## Optional: Custom `ews.cfg` and HPFEEDS
-To enable this feature, set `custom_ews=true` in `.ecs_settings.sh`; See here:  [Configure `.ecs_settings.sh`](#ecs-settings)  
+To enable these features, set `custom_ews=true` in `.ecs_settings.sh`; See here:  [Configure `.ecs_settings.sh`](#ecs-settings)  
 
-Here you can create a custom config file for `ewsposter`.  
+### ews.cfg
+You can use a custom config file for `ewsposter`.  
 e.g. when you have your own credentials for delivering data to our [Sicherheitstacho](https://sicherheitstacho.eu/start/main).  
 You can find the `ews.cfg` template file here: [`ansible/roles/custom_ews/templates/ews.cfg`](ansible/roles/custom_ews/templates/ews.cfg) and adapt it for your needs.
 
@@ -193,21 +193,21 @@ token = your_token
 ...
 ```
 
-<a name="hpfeeds"></a>
+### HPFEEDS
-## Optional: Configure `.hpfeeds_settings.sh`
+You can also specify HPFEEDS in [`ansible/roles/custom_ews/templates/hpfeeds.cfg`](ansible/roles/custom_ews/templates/hpfeeds.cfg).  
-When the `custom_ews=true` setting is set in`.ecs_settings.sh` (see here:  [Configure `.ecs_settings.sh`](#ecs-settings)), you can also specify custom HPFEEDS in `.hpfeeds_settings.sh`.  
+That file constains the defaults (turned off) and you can adapt it for your needs, e.g. for SISSDEN:
-e.g. for SISSDEN: 
 ```
 myENABLE=true
 myHOST=hpfeeds.sissden.eu
 myPORT=10000
 myCHANNEL=t-pot.events
+myCERT=/opt/ewsposter/sissden.pem
 myIDENT=your_user
 mySECRET=your_secret
-myCERT=/opt/ewsposter/sissden.pem
 myFORMAT=json
 ```
 
+
 <a name="deploy"></a>
 # Deploying a T-Pot :honey_pot::honeybee:
 Now, after configuring everything, we can finally start deploying T-Pots:  

cloud/open-telekom-cloud/ansible/roles/custom_ews/tasks/main.yaml

@@ -12,29 +12,13 @@
     insertafter: '/opt/ewsposter/ews.ip'
     line: '     - /data/ews/conf/ews.cfg:/opt/ewsposter/ews.cfg'
 
-- name: Lookup HPFEED environment variables
+- name: Copy hpfeeds configuration file
-  set_fact:
+  template:
-    myENABLE: "{{ lookup('env', 'myENABLE') }}"
+    src: ../templates/hpfeeds.cfg
-    myHOST: "{{ lookup('env', 'myHOST') }}"
+    dest: /data/ews/conf
-    myPORT: "{{ lookup('env', 'myPORT') }}"
+    owner: root
-    myCHANNEL: "{{ lookup('env', 'myCHANNEL') }}"
+    group: root
-    myIDENT: "{{ lookup('env', 'myIDENT') }}"
+    mode: 0644
-    mySECRET: "{{ lookup('env', 'mySECRET') }}"
-    myCERT: "{{ lookup('env', 'myCERT') }}"
-    myFORMAT: "{{ lookup('env', 'myFORMAT') }}"
 
-- name: Apply HPFEED settings in tpot.yml
+- name: Applying hpfeeds settings
-  lineinfile:
+  command: /opt/tpot/bin/hpfeeds_optin.sh --conf=/data/ews/conf/hpfeeds.cfg
-    path: /opt/tpot/etc/tpot.yml
-    regexp: "{{ item.regexp }}"
-    line: "{{ item.line }}"
-  with_items:
-    - { regexp: 'EWS_HPFEEDS_ENABLE.*', line: '     - EWS_HPFEEDS_ENABLE={{ myENABLE | lower }}' }
-    - { regexp: 'EWS_HPFEEDS_HOST.*', line: '     - EWS_HPFEEDS_HOST={{ myHOST }}' }
-    - { regexp: 'EWS_HPFEEDS_PORT.*', line: '     - EWS_HPFEEDS_PORT={{ myPORT }}' }
-    - { regexp: 'EWS_HPFEEDS_CHANNELS.*', line: '     - EWS_HPFEEDS_CHANNELS={{ myCHANNEL }}' }
-    - { regexp: 'EWS_HPFEEDS_IDENT.*', line: '     - EWS_HPFEEDS_IDENT={{ myIDENT }}' }
-    - { regexp: 'EWS_HPFEEDS_SECRET.*', line: '     - EWS_HPFEEDS_SECRET={{ mySECRET }}' }
-    - { regexp: 'EWS_HPFEEDS_TLSCERT.*', line: '     - EWS_HPFEEDS_TLSCERT={{ myCERT }}' }
-    - { regexp: 'EWS_HPFEEDS_TLSCERT.False', line: '     - EWS_HPFEEDS_TLSCERT={{ myCERT | lower }}' }
-    - { regexp: 'EWS_HPFEEDS_FORMAT.*', line: '     - EWS_HPFEEDS_FORMAT={{ myFORMAT }}' }

cloud/open-telekom-cloud/ansible/roles/custom_ews/templates/hpfeeds.cfg

@@ -2,7 +2,7 @@ myENABLE=false
 myHOST=host
 myPORT=port
 myCHANNEL=channels
+myCERT=false
 myIDENT=user
 mySECRET=secret
-myCERT=false
 myFORMAT=json

cloud/open-telekom-cloud/deploy_ansible_otc_t-pot.sh

@@ -33,22 +33,9 @@ source .ecs_settings.sh
 # Import OTC authentication credentials
 source .otc_env.sh
 
-# Import HPFEED settings
-source .hpfeeds_settings.sh
-
 # Password is later used by Ansible
 export LINUX_PASS=$linuxpass
 
-# HPFEED settings are later used by Ansible
-export myENABLE=$myENABLE
-export myHOST=$myHOST
-export myPORT=$myPORT
-export myCHANNEL=$myCHANNEL
-export myIDENT=$myIDENT
-export mySECRET=$mySECRET
-export myCERT=$myCERT
-export myFORMAT=$myFORMAT
-
 # Ignore ssh host keys as they are new anyway
 export ANSIBLE_HOST_KEY_CHECKING=False
 

update.sh

@@ -253,18 +253,24 @@ echo "### https://raw.githubusercontent.com/dtag-dev-sec/tpotce/master/etc/objec
 echo "### Export and import the objects easily through the Kibana WebUI:"
 echo "### Go to Kibana > Management > Saved Objects > Export / Import"
 echo "### All objects will be overwritten upon import, make sure to run an export first."
-echo
-echo "### Please reboot."
-echo
 }
 
 function fuRESTORE_EWSCFG () {
 if [ -f '/data/ews/conf/ews.cfg' ] && ! grep 'ews.cfg' /opt/tpot/etc/tpot.yml > /dev/null; then
+    echo
     echo "### Restoring volume mount for ews.cfg in tpot.yml"
     sed -i '/\/opt\/ewsposter\/ews.ip/a\\ \ \ \ \ - /data/ews/conf/ews.cfg:/opt/ewsposter/ews.cfg' /opt/tpot/etc/tpot.yml
 fi
 }
 
+function fuRESTORE_HPFEEDS () {
+if [ -f '/data/ews/conf/hpfeeds.cfg' ]; then
+    echo
+    echo "### Restoring HPFEEDS in tpot.yml"
+    ./bin/hpfeeds_optin.sh --conf=/data/ews/conf/hpfeeds.cfg
+fi
+}
+
 
 ################
 # Main section #
@@ -297,3 +303,8 @@ fuBACKUP
 fuSELFUPDATE "$0" "$@"
 fuUPDATER
 fuRESTORE_EWSCFG
+fuRESTORE_HPFEEDS
+
+echo
+echo "### Please reboot."
+echo