Add a new elasticsearch honeypot

adjust installer
adjust elasticpot configs to T-Pot's environment
create Dockerfile
adjust logstash config
update Readme

README.md

@@ -1,6 +1,6 @@
 ![T-Pot](doc/tpotsocial.png)
 
-T-Pot 19.03 runs on Debian (Stable), is based heavily on
+T-Pot 20.06 runs on Debian (Stable), is based heavily on
 
 [docker](https://www.docker.com/), [docker-compose](https://docs.docker.com/compose/)
 
@@ -12,7 +12,7 @@ and includes dockerized versions of the following honeypots
 * [conpot](http://conpot.org/),
 * [cowrie](https://github.com/cowrie/cowrie),
 * [dionaea](https://github.com/DinoTools/dionaea),
-* [elasticpot](https://github.com/schmalle/ElasticpotPY),
+* [elasticpot](https://gitlab.com/bontchev/elasticpot),
 * [glutton](https://github.com/mushorg/glutton),
 * [heralding](https://github.com/johnnykv/heralding),
 * [honeypy](https://github.com/foospidy/HoneyPy),
@@ -88,7 +88,7 @@ In T-Pot we combine the dockerized honeypots ...
 * [conpot](http://conpot.org/),
 * [cowrie](http://www.micheloosterhof.com/cowrie/),
 * [dionaea](https://github.com/DinoTools/dionaea),
-* [elasticpot](https://github.com/schmalle/ElasticpotPY),
+* [elasticpot](https://gitlab.com/bontchev/elasticpot),
 * [glutton](https://github.com/mushorg/glutton),
 * [heralding](https://github.com/johnnykv/heralding),
 * [honeypy](https://github.com/foospidy/HoneyPy),
@@ -179,7 +179,7 @@ Depending on your installation type, whether you install on [real hardware](#har
 # Installation
 The installation of T-Pot is straight forward and heavily depends on a working, transparent and non-proxied up and running internet connection. Otherwise the installation **will fail!**
 
-Firstly, decide if you want to download our prebuilt installation ISO image from [GitHub](https://github.com/dtag-dev-sec/tpotce/releases), [create it yourself](#createiso) ***or*** [post-install on an existing Debian 9.7 (Stretch)](#postinstall).
+Firstly, decide if you want to download our prebuilt installation ISO image from [GitHub](https://github.com/dtag-dev-sec/tpotce/releases), [create it yourself](#createiso) ***or*** [post-install on an existing Debian 10 (Buster)](#postinstall).
 
 Secondly, decide where you want to let the system run: [real hardware](#hardware) or in a [virtual machine](#vm)?
 
@@ -193,7 +193,7 @@ You can download the prebuilt installation image from [GitHub](https://github.co
 For transparency reasons and to give you the ability to customize your install, we provide you the [ISO Creator](https://github.com/dtag-dev-sec/tpotce) that enables you to create your own ISO installation image.
 
 **Requirements to create the ISO image:**
-- Debian 9.7 or newer as host system (others *may* work, but *remain* untested)
+- Debian 10 as host system (others *may* work, but *remain* untested)
 - 4GB of free memory  
 - 32GB of free storage
 - A working internet connection
@@ -240,7 +240,7 @@ Whereas most CD burning tools allow you to burn from ISO images, the procedure t
 
 <a name="postinstall"></a>
 ## Post-Install User
-In some cases it is necessary to install Debian 9.7 (Stretch) on your own:
+In some cases it is necessary to install Debian 10 (Buster) on your own:
  - Cloud provider does not offer mounting ISO images.
  - Hardware setup needs special drivers and / or kernels.
  - Within your company you have to setup special policies, software etc.
@@ -474,7 +474,7 @@ We hope you understand that we cannot provide support on an individual basis. We
 # Licenses
 The software that T-Pot is built on uses the following licenses.
 <br>GPLv2: [conpot](https://github.com/mushorg/conpot/blob/master/LICENSE.txt), [dionaea](https://github.com/DinoTools/dionaea/blob/master/LICENSE), [honeypy](https://github.com/foospidy/HoneyPy/blob/master/LICENSE), [honeytrap](https://github.com/armedpot/honeytrap/blob/master/LICENSE), [suricata](http://suricata-ids.org/about/open-source/)
-<br>GPLv3: [adbhoney](https://github.com/huuck/ADBHoney), [elasticpot](https://github.com/schmalle/ElasticpotPY), [ewsposter](https://github.com/dtag-dev-sec/ews/), [fatt](https://github.com/0x4D31/fatt/blob/master/LICENSE), [rdpy](https://github.com/citronneur/rdpy/blob/master/LICENSE), [heralding](https://github.com/johnnykv/heralding/blob/master/LICENSE.txt), [snare](https://github.com/mushorg/snare/blob/master/LICENSE), [tanner](https://github.com/mushorg/snare/blob/master/LICENSE)
+<br>GPLv3: [adbhoney](https://github.com/huuck/ADBHoney), [elasticpot](https://gitlab.com/bontchev/elasticpot/-/blob/master/LICENSE), [ewsposter](https://github.com/dtag-dev-sec/ews/), [fatt](https://github.com/0x4D31/fatt/blob/master/LICENSE), [rdpy](https://github.com/citronneur/rdpy/blob/master/LICENSE), [heralding](https://github.com/johnnykv/heralding/blob/master/LICENSE.txt), [snare](https://github.com/mushorg/snare/blob/master/LICENSE), [tanner](https://github.com/mushorg/snare/blob/master/LICENSE)
 <br>Apache 2 License: [cyberchef](https://github.com/gchq/CyberChef/blob/master/LICENSE), [elasticsearch](https://github.com/elasticsearch/elasticsearch/blob/master/LICENSE.txt), [logstash](https://github.com/elasticsearch/logstash/blob/master/LICENSE), [kibana](https://github.com/elasticsearch/kibana/blob/master/LICENSE.md), [docker](https://github.com/docker/docker/blob/master/LICENSE), [elasticsearch-head](https://github.com/mobz/elasticsearch-head/blob/master/LICENCE)
 <br>MIT license: [ciscoasa](https://github.com/Cymmetria/ciscoasa_honeypot/blob/master/LICENSE), [glutton](https://github.com/mushorg/glutton/blob/master/LICENSE)
 <br> Other: [citrixhoneypot](https://github.com/MalwareTech/CitrixHoneypot#licencing-agreement-malwaretech-public-licence), [cowrie](https://github.com/micheloosterhof/cowrie/blob/master/LICENSE.md), [mailoney](https://github.com/awhitehatter/mailoney), [Debian licensing](https://www.debian.org/legal/licenses/)
@@ -495,7 +495,7 @@ Without open source and the fruitful development community (we are proud to be a
 * [debian](http://www.debian.org/)
 * [dionaea](https://github.com/DinoTools/dionaea/graphs/contributors)
 * [docker](https://github.com/docker/docker/graphs/contributors)
-* [elasticpot](https://github.com/schmalle/ElasticpotPY/graphs/contributors)
+* [elasticpot](https://gitlab.com/bontchev/elasticpot/-/project_members)
 * [elasticsearch](https://github.com/elastic/elasticsearch/graphs/contributors)
 * [elasticsearch-head](https://github.com/mobz/elasticsearch-head/graphs/contributors)
 * [ewsposter](https://github.com/armedpot/ewsposter/graphs/contributors)

bin/hptest.sh

@@ -83,6 +83,7 @@ fuCHECKFORARGS
 echo "Starting scans ..."
 echo "$myMEDPOTPACKET" | nc "$myHOST" 2575 &
 curl -XGET "http://$myHOST:9200/logstash-*/_search" &
+curl -XPOST -H "Content-Type: application/json" -d '{"name":"test","email":"test@test.com"}' "http://$myHOST:9200/test" &
 echo "
I20100" | timeout --foreground 3 nc "$myHOST" 10001 &
 fuSCAN "180" "7,8,102,135,161,1025,1080,5000,9200" "$myHOST" "-sC -sS -sU -sV"
 fuSCAN "180" "2048,4096,5432" "$myHOST" "-sC -sS -sU -sV --version-light"

docker/docker-compose.yml

@@ -37,6 +37,11 @@ services:
     build: dionaea/.
     image: "dtagdevsec/dionaea:2006"
 
+# ElasticPot service
+  elasticpot:
+    build: elasticpot/.
+    image: "dtagdevsec/elasticpot:2006"
+
 # Glutton service
   glutton:
     build: glutton/.

docker/elasticpot.old/Dockerfile

@@ -0,0 +1,34 @@
+FROM alpine:latest
+#
+# Include dist
+ADD dist/ /root/dist/
+#
+# Install packages
+RUN apk -U --no-cache add \
+             git \
+	     py3-pip \
+             python3 && \
+    pip3 install --no-cache-dir bottle \
+                                configparser \
+                                datetime \
+                                requests && \
+    mkdir -p /opt && \
+    cd /opt/ && \
+    git clone --depth=1 https://github.com/schmalle/ElasticpotPY.git && \
+#
+# Setup user, groups and configs
+    addgroup -g 2000 elasticpot && \
+    adduser -S -H -s /bin/ash -u 2000 -D -g 2000 elasticpot && \
+    mv /root/dist/elasticpot.cfg /opt/ElasticpotPY/ && \
+    mkdir /opt/ElasticpotPY/log && \
+#
+# Clean up
+    apk del --purge git && \
+    rm -rf /root/* && \
+    rm -rf /var/cache/apk/*
+#
+# Start elasticpot
+STOPSIGNAL SIGINT
+USER elasticpot:elasticpot
+WORKDIR /opt/ElasticpotPY/
+CMD ["/usr/bin/python3","main.py"]

docker/elasticpot.old/docker-compose.yml

@@ -0,0 +1,20 @@
+version: '2.3'
+
+networks:
+  elasticpot_local:
+
+services:
+
+# Elasticpot service
+  elasticpot:
+    build: .
+    container_name: elasticpot
+    restart: always
+    networks:
+     - elasticpot_local
+    ports:
+     - "9200:9200"
+    image: "dtagdevsec/elasticpot:2006"
+    read_only: true
+    volumes:
+     - /data/elasticpot/log:/opt/ElasticpotPY/log

docker/elasticpot/Dockerfile

@@ -4,31 +4,41 @@ FROM alpine:latest
 ADD dist/ /root/dist/
 #
 # Install packages
-RUN apk -U --no-cache add \
+RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
+    apk -U add \
+             build-base \
+	     ca-certificates \
              git \
+             libffi-dev \
+	     openssl \
+             openssl-dev \
+             py3-mysqlclient \
+             py3-requests \
 	     py3-pip \
-             python3 && \
-    pip3 install --no-cache-dir bottle \
-                                configparser \
-                                datetime \
-                                requests && \
+             python3 \
+             python3-dev && \
     mkdir -p /opt && \
     cd /opt/ && \
-    git clone --depth=1 https://github.com/schmalle/ElasticpotPY.git && \
+    git clone --depth=1 https://gitlab.com/bontchev/elasticpot.git/ && \
+    cd elasticpot && \
+    pip3 install -r requirements.txt && \
 #
 # Setup user, groups and configs
     addgroup -g 2000 elasticpot && \
     adduser -S -H -s /bin/ash -u 2000 -D -g 2000 elasticpot && \
-    mv /root/dist/elasticpot.cfg /opt/ElasticpotPY/ && \
-    mkdir /opt/ElasticpotPY/log && \
+    mv /root/dist/honeypot.cfg /opt/elasticpot/etc/ && \
 #
 # Clean up
-    apk del --purge git && \
+    apk del --purge build-base \
+                    git \
+                    libffi-dev \
+		    openssl-dev \
+		    python3-dev && \
     rm -rf /root/* && \
     rm -rf /var/cache/apk/*
 #
 # Start elasticpot
 STOPSIGNAL SIGINT
 USER elasticpot:elasticpot
-WORKDIR /opt/ElasticpotPY/
-CMD ["/usr/bin/python3","main.py"]
+WORKDIR /opt/elasticpot/
+CMD ["/usr/bin/python3","elasticpot.py"]

docker/elasticpot/dist/honeypot.cfg

@@ -0,0 +1,301 @@
+# ============================================================================
+# General Honeypot Options
+# ============================================================================
+[honeypot]
+
+# Sensor name is used to identify this honeypot instance. Used by the database
+# logging modules such as JSON.
+#
+# If not specified, the logging modules will instead use the IP address of the
+# server as the sensor name.
+#
+# (default: the name of the local machine)
+sensor_name = t-pot
+
+# The version of Elasticsearch reported by the honeypot.
+#
+# (default: 1.4.1)
+#spoofed_version = 1.4.1
+
+# The Elasticsearch instance name reported by the honeypot.
+#
+# (default = Green Goblin)
+instance_name = USNYES01
+
+# The name of the simulated Elasticsearch cluster
+#
+# (default = elasticsearch
+#cluster_name = elasticsearch
+
+# The name of the simulated host running Elasticsearch
+#
+# (default = elk)
+host_name = usnyes01
+
+# The build number of the simulated Elasticsearch instance
+# Use something realistic or simply don't touch this value
+#
+# (default = 89d3241)
+#build = 89d3241
+
+# The number of processors on the simulated host
+#
+# (default = 12)
+#total_processors = 12
+
+# The total number of CPU cores on the simulated host
+# Use a multiple of total_processors
+#
+# (default = 24)
+#total_cores = 24
+
+# The total number of sockets on the simulated host
+# Use a multiple of total_cores
+#
+# (default = 48)
+#total_sockets = 48
+
+# The MAC address of the networking card of the simulated host
+#
+# (default = 08:01:c7:3F:15:DD)
+#mac_address = 08:01:c7:3F:15:DD
+
+# Directory where to save log files in.
+# Log files are <log_filename>.YYYY-MM-DD in that directory
+#
+# (default: log)
+log_path = log
+
+# Log file name
+#
+# (default: stdout)
+#log_filename =
+
+# Directory containing the response files
+#
+# (default: responses)
+#responses_dir = responses
+
+# ============================================================================
+# Network Specific Options
+# ============================================================================
+
+# Port to listen for incoming connections.
+#
+# (default: 9200)
+#listen_port = 9200
+
+# Site to query for one's public IP address
+#
+# (default: https://ident.me)
+#public_ip_url = https://ident.me
+
+# Enable to log the public IP of the honeypot (useful if listening on 127.0.0.1)
+# IP address is obtained by querying public_ip_url
+#
+# (default: false)
+#report_public_ip = false
+
+
+# ============================================================================
+# Output Plugins
+# These provide an extensible mechanism to send audit log entries to third
+# parties. The audit entries contain information on clients connecting to
+# the honeypot.
+#
+# Output entries need to start with 'output_' and have the 'enabled' entry.
+# ============================================================================
+
+# JSON based logging module
+#
+[output_jsonlog]
+enabled = true
+logfile = log/elasticpot.json
+epoch_timestamp = false
+
+# MySQL logging module
+# Database structure for this module is supplied in docs/sql/mysql.sql
+#
+# MySQL logging requires extra software: sudo apt-get install libmysqlclient-dev
+# MySQL logging requires an extra Python module: pip install mysql-python
+#
+#[output_mysql]
+#enabled = false
+#host = localhost
+#database = elasticpot
+#username = elasticpot
+#password = secret
+#port = 3306
+#debug = false
+# Whether to store geolocation data in the database
+#geoip = true
+# Location of the databases used for geolocation
+#geoip_citydb = data/GeoLite2-City.mmdb
+#geoip_asndb = data/GeoLite2-ASN.mmdb
+
+# Text output
+# This writes audit log entries to a text file
+#
+#[output_textlog]
+#enabled = false
+#logfile = log/elasticpot.txt
+
+# HPFeeds
+#
+# Note the lack of "s" at the end:
+[output_hpfeed]
+enabled = false
+#server = hpfeeds.mysite.org
+#tlscert = /path/to/tls/cert/file
+#port = 10000
+#identifier = abc123
+#secret = secret
+#channel = elasticpot
+
+# MongoDB logging module
+#
+#[output_mongodb]
+#enabled = false
+#host = 127.0.0.1
+#port = 27017
+#username = elasticpot
+#password = secret
+#database = elasticpot
+# Note: .format(username, password, host, port, database) is done
+#  on the following string; make sure that there are 5 placeholders ({}) in it
+#connection_string = mongodb://{}:{}@{}:{}/{}
+# Whether to store geolocation data in the database
+#geoip = true
+# Location of the databases used for geolocation
+#geoip_citydb = data/GeoLite2-City.mmdb
+#geoip_asndb = data/GeoLite2-ASN.mmdb
+
+# RedisDB logging module
+#
+#[output_redisdb]
+#enabled = false
+#host = 127.0.0.1
+#port = 6379
+# DB of the redis server. Defaults to 0
+#db = 0
+# Password of the redis server. Defaults to None
+#password = secret
+# Name of the list to push to or the channel to publish to. Required
+#keyname = elasticpot
+# Method to use when sending data to redis.
+# Can be one of [lpush, rpush, publish]. Defaults to lpush
+#send_method = lpush
+
+# Rethinkdb output module
+#
+#[output_rethinkdblog]
+#enabled = false
+#host = 127.0.0.1
+#port = 28015
+#table = events
+#db = elasticpot
+#password =
+
+# InfluxDB logging module
+#
+#[output_influx]
+#enabled = false
+#host = 127.0.0.1
+#port = 8086
+#database_name = elasticpot
+#retention_policy_duration = 12w
+
+# InfluxDB 2.0 logging module
+#
+#[output_influx2]
+#enabled = false
+#host = hostname
+#token = token
+#org = organization
+#bucket = elasticpot
+
+# CouchDB logging module
+#
+#[output_couch]
+#enabled = false
+#host = localhost
+#port = 5984
+#username = elasticpot
+#password = secret
+#database = elasticpot
+#geoip = true
+# Location of the databases used for geolocation
+#geoip_citydb = data/GeoLite2-City.mmdb
+#geoip_asndb = data/GeoLite2-ASN.mmdb
+
+# SQLite3 logging module
+#
+# Logging to SQLite3 database. To init the database, use the script
+# docs/sql/sqlite3.sql:
+#     sqlite3 <db_file> < docs/sql/sqlite3.sql
+#
+#[output_sqlite]
+#enabled = false
+#debug = false
+#db_file = data/elasticpot.db
+# Whether to store geolocation data in the database
+#geoip = true
+# Location of the databases used for geolocation
+#geoip_citydb = data/GeoLite2-City.mmdb
+#geoip_asndb = data/GeoLite2-ASN.mmdb
+
+# Elasticsearch logging module
+#
+#[output_elastic]
+#enabled = false
+#host = localhost
+#port = 9200
+#index = elasticpot
+#
+# type has been deprecated since ES 6.0.0
+# use _doc which is the default type. See
+# https://stackoverflow.com/a/53688626 for
+# more information
+#
+#type = _doc
+#
+# set pipeline = geoip to map src_ip to
+# geo location data. You can use a custom
+# pipeline but you must ensure it exists
+# in elasticsearch.
+#
+#pipeline = geoip
+#
+# Authentication. When x-pack.security is enabled
+# in ES, default users have been created and requests
+# must be authenticated.
+#
+# Credentials
+#
+#username = elasticpot
+#password = secret
+#
+# TLS encryption. Communications between the client (elasticpot) 
+# and the ES server should naturally be protected by encryption
+# if requests are authenticated (to prevent from man-in-the-middle 
+# attacks). The following options are then paramount
+# if username and password are provided.
+#
+# use ssl/tls
+#ssl = true
+# verify SSL certificates
+#verify_certs = true
+# Path to trusted CA certs on disk
+#ca_certs = /path/to/cert/file/elastic_ca.crt
+
+
+# TODO:
+
+# Kafka logging module
+#
+#[output_kafka]
+#enabled = false
+#host = 127.0.0.1
+#port = 9092
+#topic = elasticpot
+

docker/elasticpot/docker-compose.yml

@@ -17,4 +17,4 @@ services:
     image: "dtagdevsec/elasticpot:2006"
     read_only: true
     volumes:
-     - /data/elasticpot/log:/opt/ElasticpotPY/log
+     - /data/elasticpot/log:/opt/elasticpot/log

docker/elk/logstash/dist/logstash.conf

@@ -64,9 +64,9 @@ input {
     type => "Dionaea"
   }
 
-# Elasticpot
+# ElasticPot
   file {
-    path => ["/data/elasticpot/log/elasticpot.log"]
+    path => ["/data/elasticpot/log/elasticpot.json"]
     codec => json
     type => "ElasticPot"
   }
@@ -296,6 +296,17 @@ filter {
     date {
       match => [ "timestamp", "ISO8601" ]
     }
+    mutate {
+      rename => {
+        "content_type" => "http.http_content_type"
+        "dst_port" => "dest_port"
+        "dst_ip" => "dest_ip"
+        "message" => "event_type"
+        "request" => "request_method"
+        "user_agent" => "http_user_agent"
+	"url" => "http.url"
+      }
+    }
   }
 
 # Glutton

etc/compose/nextgen.yml

@@ -11,6 +11,7 @@ networks:
   conpot_local_kamstrup_382:
   cowrie_local:
   cyberchef_local:
+  elasticpot_local:
   heralding_local:
   honeypy_local:
   mailoney_local:
@@ -210,6 +211,19 @@ services:
      - /data/dionaea/log:/opt/dionaea/var/log
      - /data/dionaea/rtp:/opt/dionaea/var/dionaea/rtp
 
+# ElasticPot service
+  elasticpot:
+    container_name: elasticpot
+    restart: always
+    networks:
+     - elasticpot_local
+    ports:
+     - "9200:9200"
+    image: "dtagdevsec/elasticpot:2006"
+    read_only: true
+    volumes:
+     - /data/elasticpot/log:/opt/elasticpot/log
+
 # Glutton service
   glutton:
     container_name: glutton
@@ -268,7 +282,7 @@ services:
      - "2323:2323"
      - "2324:2324"
      - "4096:4096"
-     - "9200:9200"
+    # - "9200:9200"
     image: "dtagdevsec/honeypy:2006"
     read_only: true
     volumes:

etc/compose/sensor.yml

@@ -195,7 +195,7 @@ services:
      - /data/dionaea/log:/opt/dionaea/var/log
      - /data/dionaea/rtp:/opt/dionaea/var/dionaea/rtp
 
-# Elasticpot service
+# ElasticPot service
   elasticpot:
     container_name: elasticpot
     restart: always
@@ -206,7 +206,7 @@ services:
     image: "dtagdevsec/elasticpot:2006"
     read_only: true
     volumes:
-     - /data/elasticpot/log:/opt/ElasticpotPY/log
+     - /data/elasticpot/log:/opt/elasticpot/log
 
 # Heralding service
   heralding:

etc/compose/standard.yml

@@ -196,7 +196,7 @@ services:
      - /data/dionaea/log:/opt/dionaea/var/log
      - /data/dionaea/rtp:/opt/dionaea/var/dionaea/rtp
 
-# Elasticpot service
+# ElasticPot service
   elasticpot:
     container_name: elasticpot
     restart: always
@@ -207,7 +207,7 @@ services:
     image: "dtagdevsec/elasticpot:2006"
     read_only: true
     volumes:
-     - /data/elasticpot/log:/opt/ElasticpotPY/log
+     - /data/elasticpot/log:/opt/elasticpot/log
 
 # Heralding service
   heralding:

etc/logrotate/logrotate.conf

@@ -11,6 +11,7 @@
 /data/dionaea/log/dionaea.sqlite
 /data/dionaea/dionaea-errors.log
 /data/elasticpot/log/elasticpot.log
+/data/elasticpot/log/elasticpot.json
 /data/elk/log/*.log
 /data/fatt/log/fatt.log
 /data/glutton/log/*.log