Merge pull request #585 from dtag-dev-sec/dev

Prepare release 19.03.3

README.md

@@ -1,6 +1,6 @@
 ![T-Pot](doc/tpotsocial.png)
 
-T-Pot 19.03 runs on Debian (Sid), is based heavily on
+T-Pot 19.03 runs on Debian (Stable), is based heavily on
 
 [docker](https://www.docker.com/), [docker-compose](https://docs.docker.com/compose/)
 
@@ -43,7 +43,6 @@ Furthermore we use the following tools
 
 
 # Table of Contents
-- [Changelog](#changelog)
 - [Technical Concept](#concept)
 - [System Requirements](#requirements)
 - [Installation](#installation)
@@ -76,64 +75,10 @@ Furthermore we use the following tools
 - [Testimonial](#testimonial)
 - [Fun Fact](#funfact)
 
-<a name="changelog"></a>
-# Release Notes
-- **Move from Ubuntu 18.04 to Debian (Sid)**
-  - For almost 5 years Ubuntu LTS versions were our distributions of choice. Last year we made a design choice for T-Pot to be closer to a rolling release model and thus allowing us to issue smaller changes and releases in a more timely manner. The distribution of choice is Debian (Sid / unstable) which will provide us with the latest advancements in a Debian based distribution.
-- **Include HoneyPy honeypot**
-  - *HoneyPy* is now included in the NEXTGEN installation type
-- **Include Suricata 4.1.3**
-  - Building *Suricata 4.1.3* from scratch to enable JA3 and overall better protocol support.
-- **Update tools to the latest versions**
-  - ELK Stack 6.6.2
-  - CyberChef 8.27.0
-  - SpiderFoot v3.0
-  - Cockpit 188
-  - NGINX is now built to enforce TLS 1.3 on the T-Pot WebUI
-- **Update honeypots**
-  - Where possible / feasible the honeypots have been updated to their latest versions.
-  - *Cowrie* now supports *HASSH* generated hashes which allows for an easier identification of an attacker accross IP adresses.
-  - *Heralding* now supports *SOCKS5* emulation.
-- **Update Dashboards & Visualizations**
-  - *Offset Dashboard* added to easily spot changes in attacks on a single dashboard in 24h time window.
-  - *Cowrie Dashboard* modified to integrate *HASSH* support / visualizations.
-  - *HoneyPy Dashboard* added to support latest honeypot addition.
-  - *Suricata Dashboard* modified to integrate *JA3* support / visualizations.
-- **Debian mirror selection**
-  - During base install you now have to manually select a mirror.
-  - Upon T-Pot install the mirror closest to you will be determined automatically, `netselect-apt` requires you to allow ICMP outbound.
-  - This solves peering problems for most of the users speeding up installation and updates.
-- **Bugs**
-  - Fixed issue #298 where the import and export of objects on the shell did not work.
-  - Fixed issue #313 where Spiderfoot raised a KeyError, which was previously fixed in upstream.
-  - Fixed error in Suricata where path for reference.config changed.
-- **Release Cycle**
-  - As far as possible we will integrate changes now faster into the master branch, eliminating the need for monolithic releases. The update feature will be continuously improved on that behalf. However this might not account for all feature changes.
-- **HPFEEDS Opt-In**
-  - If you want to share your T-Pot data with a 3rd party HPFEEDS broker such as you can do so by runnning `hpfeeds_optin.sh` on T-Pot.
-- **Update Feature**
-  - For the ones who like to live on the bleeding edge of T-Pot development there is now an update script available in `/opt/tpot/update.sh`.
-  - This feature is beta and is mostly intended to provide you with the latest development advances without the need of reinstalling T-Pot.
-- **Deprecated tools**
-  - *ctop* will no longer be part of T-Pot.
-- **Fix #332**
-  - If T-Pot, opposed to the requirements, does not have full internet access netselect-apt fails to determine the fastest mirror as it needs ICMP and UDP outgoing. Should netselect-apt fail the default mirrors will be used.
-- **Improve install speed with apt-fast**
-  - Migrating from a stable base install to Debian (Sid) requires downloading lots of packages. Depending on your geo location the download speed was already improved by introducing netselect-apt to determine the fastest mirror. With apt-fast the downloads will be even faster by downloading packages not only in parallel but also with multiple connections per package.
-- **HPFEEDS Opt-In commandline option**
-  - Pass a hpfeeds config file as a commandline argument
-  - hpfeeds config is saved in `/data/ews/conf/hpfeeds.cfg`
-  - Update script restores hpfeeds config
-- **Ansible T-Pot Deployment**
-  - Transitioned from bash script to all Ansible
-  - Reusable Ansible Playbook for OpenStack clouds
-  - Example Showcase with our Open Telekom Cloud
-  - Adaptable for other cloud providers
-
 <a name="concept"></a>
 # Technical Concept
 
-T-Pot is based on the network installer Debian (Stretch). During installation the whole system will be updated to Debian (Sid).
+T-Pot is based on the network installer Debian (Stable).
 The honeypot daemons as well as other support components being used have been containerized using [docker](http://docker.io).
 This allows us to run multiple honeypot daemons on the same network interface while maintaining a small footprint and constrain each honeypot within its own environment.
 
@@ -302,7 +247,7 @@ In some cases it is necessary to install Debian 9.7 (Stretch) on your own:
  - Within your company you have to setup special policies, software etc.
  - You just like to stay on top of things.
 
-The T-Pot Universal Installer will upgrade the system to Debian (Sid) and install all required T-Pot dependencies.
+The T-Pot Universal Installer will upgrade the system and install all required T-Pot dependencies.
 
 Just follow these steps:
 
@@ -396,7 +341,7 @@ For the ones of you who want to live on the bleeding edge of T-Pot development w
 
 The Update script will:
  - **mercilessly** overwrite local changes to be in sync with the T-Pot master branch
- - upgrade the system to the packages available in Debian (Sid)
+ - upgrade the system to the packages available in Debian (Stable)
  - update all resources to be in-sync with the T-Pot master branch
  - ensure all T-Pot relevant system files will be patched / copied into the original T-Pot state
  - restore your custom ews.cfg and HPFEED settings from `/data/ews/conf`

docker/honeysap/Dockerfile

@@ -0,0 +1,39 @@
+FROM alpine:latest
+#
+# Include dist
+ADD dist/ /root/dist/
+#
+# Install packages
+RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
+    apk -U --no-cache add \
+            build-base \
+            git \
+            libcap \
+            python2 \
+            python2-dev \
+            py2-pip \
+            tcpdump && \
+#
+# Clone honeysap from git
+    git clone --depth=1 https://github.com/SecureAuthCorp/HoneySAP /opt/honeysap && \
+    cd /opt/honeysap && \
+    mkdir conf && \
+    cp /root/dist/* conf/ && \
+    python setup.py install && \
+    pip install -r requirements-optional.txt && \
+#
+# Setup user, groups and configs
+    addgroup -g 2000 honeysap && \
+    adduser -S -s /bin/ash -u 2000 -D -g 2000 honeysap && \
+    chown -R honeysap:honeysap /opt/honeysap && \
+#    setcap cap_net_bind_service=+ep /opt/honeypy/env/bin/python && \
+#
+# Clean up
+    apk del --purge git && \
+    rm -rf /root/* \
+           /var/cache/apk/*
+#
+# Set workdir and start honeysap
+USER honeysap:honeysap
+WORKDIR /opt/honeysap
+CMD ["/opt/honeysap/bin/honeysap", "--config-file", "/opt/honeysap/conf/honeysap.yml"]

docker/honeysap/dist/external_route_table.yml

@@ -0,0 +1,6 @@
+# HoneSAP default external profile route table
+# ============================================
+#
+
+# Allow any protocols to 10.0.0.100 port 3200
+- allow,any,10.0.0.100,3200,

docker/honeysap/dist/honeysap.yml

@@ -0,0 +1,103 @@
+# HoneSAP default external profile configuration
+# ==============================================
+
+# Console logging configuration
+# -----------------------------
+
+# Level of console logging
+verbose: 2
+
+# Use colored output
+colored_console: false
+
+
+# Miscellaneous configuration
+# ---------------------------
+
+# Enable reloading after a change in one of the configuration files
+reload: true
+
+# Address to listen for all services
+listener_address: 0.0.0.0
+
+
+# SAP instance configuration
+# --------------------------
+
+# Release version
+release: "720"
+
+
+# Services configuration
+# ----------------------
+
+services:
+    - 
+        # SAP Router configuration
+        # ------------------------
+        service: SAPRouterService
+        alias: ExternalSAPRouter
+        enabled: yes
+        listener_port: 3299
+
+        # Router version number
+        router_version: 40
+
+        # Router patch version
+        router_version_patch: 4
+
+        # Password for information requests. If present it will be required
+        info_password:
+
+        # Wether the external administration would be enabled on this SAP Router
+        external_admin: false
+
+        # Route table file
+        route_table: !include external_route_table.yml
+
+        # Hostname for the SAP Router
+        hostname: saprouter
+
+    -
+        # SAP Dispatcher configuration
+        # ----------------------------
+        service: SAPDispatcherService
+        alias: InternalDispatcherService
+        enabled: yes
+        virtual: yes
+        listener_port: 3200
+        listener_address: 10.0.0.100
+
+        # Name of the instance
+        instance: NSP
+        
+        # Client number
+        client_no: "001"
+        
+        # SID
+        sid: PRD
+        
+        # Hostname
+        hostname: uscasf-sap01
+
+
+# Feeds configuration
+# -------------------
+
+feeds:
+    -
+        feed: LogFeed
+        log_filename: log/honeysap-external.log
+        enabled: yes
+    -
+        feed: ConsoleFeed
+        enabled: yes
+    -
+        feed: HPFeed
+        channels:
+            - honeysap.events
+        feed_host: 10.250.250.20
+        feed_port: 20000
+        feed_ident: honeysap
+        feed_secret: password
+        enabled: no

docker/honeysap/docker-compose.yml

@@ -0,0 +1,20 @@
+version: '2.3'
+
+networks:
+  honeysap_local:
+
+services:
+
+# HoneySAP service
+  honeysap:
+    build: .
+    container_name: honeysap
+    restart: always
+    networks:
+     - honeysap_local
+    ports:
+     - "3299:3299"
+     - "8001:8001"
+    image: "dtagdevsec/honeysap:2006"
+    volumes:
+     - /data/honeysap/log:/opt/honeysap/log

iso/installer/install.sh

@@ -11,22 +11,21 @@ myPROGRESSBOXCONF=" --backtitle "$myBACKTITLE" --progressbox 24 80"
 mySITES="https://hub.docker.com https://github.com https://pypi.python.org https://debian.org"
 myTPOTCOMPOSE="/opt/tpot/etc/tpot.yml"
 myLSB_STABLE_SUPPORTED="stretch buster"
-myLSB_TESTING_SUPPORTED="sid"
+myLSB_TESTING_SUPPORTED="stable"
 myREMOTESITES="https://hub.docker.com https://github.com https://pypi.python.org https://debian.org"
 myPREINSTALLPACKAGES="aria2 apache2-utils cracklib-runtime curl dialog figlet fuse grc libcrack2 libpq-dev lsb-release netselect-apt net-tools software-properties-common toilet"
-myINSTALLPACKAGES="aria2 apache2-utils apparmor apt-transport-https aufs-tools bash-completion build-essential ca-certificates cgroupfs-mount cockpit cockpit-docker console-setup console-setup-linux cracklib-runtime curl debconf-utils dialog dnsutils docker.io docker-compose elasticsearch-curator ethtool fail2ban figlet genisoimage git glances grc haveged html2text htop iptables iw jq kbd libcrack2 libltdl7 libpam-google-authenticator man mosh multitail netselect-apt net-tools npm ntp openssh-server openssl pass pigz prips software-properties-common syslinux psmisc pv python3-pip toilet unattended-upgrades unzip vim wget wireless-tools wpasupplicant"
+myINSTALLPACKAGES="aria2 apache2-utils apparmor apt-transport-https aufs-tools bash-completion build-essential ca-certificates cgroupfs-mount cockpit console-setup console-setup-linux cracklib-runtime curl debconf-utils dialog dnsutils docker.io docker-compose elasticsearch-curator ethtool fail2ban figlet genisoimage git glances grc haveged html2text htop iptables iw jq kbd libcrack2 libltdl7 libpam-google-authenticator man mosh multitail netselect-apt net-tools npm ntp openssh-server openssl pass pigz prips software-properties-common syslinux psmisc pv python3-pip toilet unattended-upgrades unzip vim wget wireless-tools wpasupplicant"
 myINFO="\
-########################################
+###########################################
-### T-Pot Installer for Debian (Sid) ###
+### T-Pot Installer for Debian (Stable) ###
-########################################
+###########################################
 
 Disclaimer:
 This script will install T-Pot on this system.
 By running the script you know what you are doing:
 1. SSH will be reconfigured to tcp/64295.
-2. Your Debian installation will be upgraded to Sid / unstable.
+2. Please ensure other means of access to this system in case something goes wrong.
-3. Please ensure other means of access to this system in case something goes wrong.
+3. At best this script will be executed on the console instead through a SSH session.
-4. At best this script will be executed on the console instead through a SSH session.
 
 ########################################
 
@@ -283,15 +282,15 @@ function fuGET_DEPS {
   echo
   echo "### Determine fastest mirror for your location."
   echo
-  netselect-apt -n -a amd64 unstable && cp sources.list /etc/apt/
+  netselect-apt -n -a amd64 stable && cp sources.list /etc/apt/
-  mySOURCESCHECK=$(cat /etc/apt/sources.list | grep -c unstable)
+  mySOURCESCHECK=$(cat /etc/apt/sources.list | grep -c stable)
   if [ "$mySOURCESCHECK" == "0" ]
     then
       echo "### Automatic mirror selection failed, using main mirror."
-      # Point to Debian (Sid, unstable)
+      # Point to Debian (stable)
 tee /etc/apt/sources.list <<EOF
-deb http://deb.debian.org/debian unstable main contrib non-free
+deb http://deb.debian.org/debian stable main contrib non-free
-deb-src http://deb.debian.org/debian unstable main contrib non-free
+deb-src http://deb.debian.org/debian stable main contrib non-free
 EOF
   fi
   echo
@@ -403,7 +402,7 @@ for i in "$@"
         echo "  A configuration example is available in \"tpotce/iso/installer/tpot.conf.dist\"."
         echo
         echo "--type=<[user, auto, iso]>"
-	echo "  user, use this if you want to manually install a T-Pot on a Debian (testing) machine."
+	echo "  user, use this if you want to manually install a T-Pot on a Debian (Stable) machine."
         echo "  auto, implied if a configuration file is passed as an argument for automatic deployment."
         echo "  iso, use this if you are a T-Pot developer and want to install a T-Pot from a pre-compiled iso."
         echo
@@ -684,7 +683,7 @@ echo "UseRoaming no" | tee -a /etc/ssh/ssh_config
 
 # Installing elasticdump, yq
 fuBANNER "Installing pkgs"
-npm install https://github.com/taskrabbit/elasticsearch-dump -g
+npm install elasticdump -g
 pip3 install yq
 hash -r
 

iso/isolinux/txt.cfg

@@ -1,6 +1,6 @@
 default install
 label install
-  menu label ^T-Pot 19.03.1 (based on Debian Sid)
+  menu label ^T-Pot 19.03.3 (based on Debian Stable)
   menu default
   kernel linux
   append vga=788 initrd=initrd.gz console-setup/ask_detect=true --

update.sh

@@ -82,7 +82,7 @@ echo
 # Let's check for version
 function fuCHECK_VERSION () {
 local myMINVERSION="19.03.0"
-local myMASTERVERSION="19.03.2"
+local myMASTERVERSION="19.03.3"
 echo
 echo "### Checking for Release ID"
 myRELEASE=$(lsb_release -i | grep Debian -c)
@@ -183,7 +183,7 @@ function fuUPDATER () {
 export DEBIAN_FRONTEND=noninteractive
 echo "### Installing apt-fast"
 /bin/bash -c "$(curl -sL https://raw.githubusercontent.com/ilikenwf/apt-fast/master/quick-install.sh)"
-local myPACKAGES="aria2 apache2-utils apparmor apt-transport-https aufs-tools bash-completion build-essential ca-certificates cgroupfs-mount cockpit cockpit-docker console-setup console-setup-linux cracklib-runtime curl debconf-utils dialog dnsutils docker.io docker-compose elasticsearch-curator ethtool fail2ban figlet genisoimage git glances grc haveged html2text htop iptables iw jq kbd libcrack2 libltdl7 libpam-google-authenticator man mosh multitail netselect-apt net-tools npm ntp openssh-server openssl pass pigz prips software-properties-common syslinux psmisc pv python3-elasticsearch-curator python3-pip toilet unattended-upgrades unzip vim wget wireless-tools wpasupplicant"
+local myPACKAGES="aria2 apache2-utils apparmor apt-transport-https aufs-tools bash-completion build-essential ca-certificates cgroupfs-mount cockpit console-setup console-setup-linux cracklib-runtime curl debconf-utils dialog dnsutils docker.io docker-compose elasticsearch-curator ethtool fail2ban figlet genisoimage git glances grc haveged html2text htop iptables iw jq kbd libcrack2 libltdl7 libpam-google-authenticator man mosh multitail netselect-apt net-tools npm ntp openssh-server openssl pass pigz prips software-properties-common syslinux psmisc pv python3-elasticsearch-curator python3-pip toilet unattended-upgrades unzip vim wget wireless-tools wpasupplicant"
 echo "### Removing pip based install of elasticsearch-curator"
 pip3 uninstall elasticsearch-curator -y
 hash -r
@@ -199,7 +199,7 @@ echo "docker.io docker.io/restart       boolean true" | debconf-set-selections -
 echo "debconf debconf/frontend select noninteractive" | debconf-set-selections -v
 apt-fast -y dist-upgrade -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" --force-yes
 dpkg --configure -a
-npm install "https://github.com/taskrabbit/elasticsearch-dump" -g
+npm install elasticdump -g
 pip3 install --upgrade yq
 hash -r
 echo "### Removing and holding back problematic packages ..."

version

@@ -1 +1 @@
-19.03.2
+19.03.3