Merge branch 'master' into hpfeeds-cmd

This commit is contained in:
Sebastian Haderecker 2019-06-26 11:17:49 +02:00 committed by GitHub
commit b505d560e7
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
23 changed files with 193 additions and 58 deletions

View file

@ -1,11 +1,38 @@
# Changelog
## 20190516
## 20190626
- **HPFEEDS Opt-In commandline option**
- Pass a hpfeeds config file as a commandline argument
- hpfeeds config is saved in `/data/ews/conf/hpfeeds.cfg`
- Update script restores hpfeeds config
## 20190604
- **Finalize Fatt support**
- Build visualizations, searches, dashboards
- Rebuild index patterns
- Some finishing touches
## 20190601
- **Start supporting Fatt, remove Glastopf**
- Build Dockerfile, Adjust logstash, installer, update and such.
- Glastopf is no longer supported within T-Pot
## 20190528+20190531
- **Increase total number of fields**
- Adjust total number of fileds for logstash templae from 1000 to 2000.
## 20190526
- **Fix build for Cowrie**
- Upstream changes required a new package `py-bcrypt`.
## 20190525
- **Fix build for RDPY**
- Building was prevented due to cache error which occurs lately on Alpine if `apk` is using `--no-ache' as options.
## 20190520
- **Adjust permissions for /data folder**
- Now it is possible to download files from `/data` using SCP, WINSCP or CyberDuck.
## 20190513
- **Added Ansible T-Pot Deployment on Open Telekom Cloud**
- Reusable Ansible Playbooks for all cloud providers

View file

@ -12,7 +12,6 @@ and includes dockerized versions of the following honeypots
* [cowrie](https://github.com/cowrie/cowrie),
* [dionaea](https://github.com/DinoTools/dionaea),
* [elasticpot](https://github.com/schmalle/ElasticpotPY),
* [glastopf](http://mushmush.org/),
* [glutton](https://github.com/mushorg/glutton),
* [heralding](https://github.com/johnnykv/heralding),
* [honeypy](https://github.com/foospidy/HoneyPy),
@ -30,6 +29,7 @@ Furthermore we use the following tools
* [Cyberchef](https://gchq.github.io/CyberChef/) a web app for encryption, encoding, compression and data analysis.
* [ELK stack](https://www.elastic.co/videos) to beautifully visualize all the events captured by T-Pot.
* [Elasticsearch Head](https://mobz.github.io/elasticsearch-head/) a web front end for browsing and interacting with an Elastic Search cluster.
* [Fatt](https://github.com/0x4D31/fatt) a pyshark based script for extracting network metadata and fingerprints from pcap files and live network traffic.
* [Spiderfoot](https://github.com/smicallef/spiderfoot) a open source intelligence automation tool.
* [Suricata](http://suricata-ids.org/) a Network Security Monitoring engine.
@ -141,7 +141,6 @@ In T-Pot we combine the dockerized honeypots ...
* [cowrie](http://www.micheloosterhof.com/cowrie/),
* [dionaea](https://github.com/DinoTools/dionaea),
* [elasticpot](https://github.com/schmalle/ElasticPot),
* [glastopf](http://mushmush.org/),
* [glutton](https://github.com/mushorg/glutton),
* [heralding](https://github.com/johnnykv/heralding),
* [honeypy](https://github.com/foospidy/HoneyPy),
@ -157,6 +156,7 @@ In T-Pot we combine the dockerized honeypots ...
* [Cyberchef](https://gchq.github.io/CyberChef/) a web app for encryption, encoding, compression and data analysis.
* [ELK stack](https://www.elastic.co/videos) to beautifully visualize all the events captured by T-Pot.
* [Elasticsearch Head](https://mobz.github.io/elasticsearch-head/) a web front end for browsing and interacting with an Elastic Search cluster.
* [Fatt](https://github.com/0x4D31/fatt) a pyshark based script for extracting network metadata and fingerprints from pcap files and live network traffic.
* [Spiderfoot](https://github.com/smicallef/spiderfoot) a open source intelligence automation tool.
* [Suricata](http://suricata-ids.org/) a Network Security Monitoring engine.
@ -220,7 +220,7 @@ Depending on your installation type, whether you install on [real hardware](#har
##### NextGen Installation (Glutton replacing Honeytrap, HoneyPy replacing Elasticpot)
- Honeypots: adbhoney, ciscoasa, conpot, cowrie, dionaea, glutton, heralding, honeypy, mailoney, rdpy, snare & tanner
- Tools: cockpit, cyberchef, ELK, elasticsearch head, ewsposter, NGINX, spiderfoot, p0f and suricata
- Tools: cockpit, cyberchef, ELK, elasticsearch head, ewsposter, fatt, NGINX, spiderfoot, p0f and suricata
- 6-8 GB RAM (less RAM is possible but might introduce swapping)
- 128 GB SSD (smaller is possible but limits the capacity of storing events)
@ -389,6 +389,7 @@ The Update script will:
You simply run the update script:
```
sudo su -
cd /opt/tpot/
./update.sh -y
```
@ -513,7 +514,7 @@ We hope you understand that we cannot provide support on an individual basis. We
# Licenses
The software that T-Pot is built on uses the following licenses.
<br>GPLv2: [conpot](https://github.com/mushorg/conpot/blob/master/LICENSE.txt), [dionaea](https://github.com/DinoTools/dionaea/blob/master/LICENSE), [honeypy](https://github.com/foospidy/HoneyPy/blob/master/LICENSE), [honeytrap](https://github.com/armedpot/honeytrap/blob/master/LICENSE), [suricata](http://suricata-ids.org/about/open-source/)
<br>GPLv3: [adbhoney](https://github.com/huuck/ADBHoney), [elasticpot](https://github.com/schmalle/ElasticPot), [ewsposter](https://github.com/dtag-dev-sec/ews/), [glastopf](https://github.com/glastopf/glastopf/blob/master/GPL), [rdpy](https://github.com/citronneur/rdpy/blob/master/LICENSE), [heralding](https://github.com/johnnykv/heralding/blob/master/LICENSE.txt), [snare](https://github.com/mushorg/snare/blob/master/LICENSE), [tanner](https://github.com/mushorg/snare/blob/master/LICENSE)
<br>GPLv3: [adbhoney](https://github.com/huuck/ADBHoney), [elasticpot](https://github.com/schmalle/ElasticPot), [ewsposter](https://github.com/dtag-dev-sec/ews/), [fatt](https://github.com/0x4D31/fatt/blob/master/LICENSE), [rdpy](https://github.com/citronneur/rdpy/blob/master/LICENSE), [heralding](https://github.com/johnnykv/heralding/blob/master/LICENSE.txt), [snare](https://github.com/mushorg/snare/blob/master/LICENSE), [tanner](https://github.com/mushorg/snare/blob/master/LICENSE)
<br>Apache 2 License: [cyberchef](https://github.com/gchq/CyberChef/blob/master/LICENSE), [elasticsearch](https://github.com/elasticsearch/elasticsearch/blob/master/LICENSE.txt), [logstash](https://github.com/elasticsearch/logstash/blob/master/LICENSE), [kibana](https://github.com/elasticsearch/kibana/blob/master/LICENSE.md), [docker](https://github.com/docker/docker/blob/master/LICENSE), [elasticsearch-head](https://github.com/mobz/elasticsearch-head/blob/master/LICENCE)
<br>MIT license: [ciscoasa](https://github.com/Cymmetria/ciscoasa_honeypot/blob/master/LICENSE), [glutton](https://github.com/mushorg/glutton/blob/master/LICENSE)
<br> Other: [cowrie](https://github.com/micheloosterhof/cowrie/blob/master/LICENSE.md), [mailoney](https://github.com/awhitehatter/mailoney), [Debian licensing](https://www.debian.org/legal/licenses/)
@ -537,7 +538,7 @@ Without open source and the fruitful development community (we are proud to be a
* [elasticsearch](https://github.com/elastic/elasticsearch/graphs/contributors)
* [elasticsearch-head](https://github.com/mobz/elasticsearch-head/graphs/contributors)
* [ewsposter](https://github.com/armedpot/ewsposter/graphs/contributors)
* [glastopf](https://github.com/mushorg/glastopf/graphs/contributors)
* [fatt](https://github.com/0x4D31/fatt/graphs/contributors)
* [glutton](https://github.com/mushorg/glutton/graphs/contributors)
* [heralding](https://github.com/johnnykv/heralding/graphs/contributors)
* [honeypy](https://github.com/foospidy/HoneyPy/graphs/contributors)

View file

@ -129,12 +129,12 @@ fuELK () {
chown tpot:tpot /data/elk -R
}
# Let's create a function to clean up and prepare glastopf data
fuGLASTOPF () {
if [ "$myPERSISTENCE" != "on" ]; then rm -rf /data/glastopf/*; fi
mkdir -p /data/glastopf/db /data/glastopf/log
chmod 770 /data/glastopf -R
chown tpot:tpot /data/glastopf -R
# Let's create a function to clean up and prepare fatt data
fuFATT () {
if [ "$myPERSISTENCE" != "on" ]; then rm -rf /data/fatt/*; fi
mkdir -p /data/fatt/log
chmod 770 -R /data/fatt
chown tpot:tpot -R /data/fatt
}
# Let's create a function to clean up and prepare glastopf data
@ -262,7 +262,7 @@ if [ "$myPERSISTENCE" = "on" ];
fuDIONAEA
fuELASTICPOT
fuELK
fuGLASTOPF
fuFATT
fuGLUTTON
fuHERALDING
fuHONEYPY

View file

@ -82,6 +82,8 @@ fuCHECKFORARGS
echo "Starting scans ..."
echo "$myMEDPOTPACKET" | nc "$myHOST" 2575 &
curl -XGET "http://$myHOST:9200/logstash-*/_search" &
echo "I20100" | timeout --foreground 3 nc "$myHOST" 10001 &
fuSCAN "180" "7,8,102,135,161,1025,1080,5000,9200" "$myHOST" "-sC -sS -sU -sV"
fuSCAN "180" "2048,4096,5432" "$myHOST" "-sC -sS -sU -sV --version-light"
fuSCAN "120" "20,21" "$myHOST" "--script=ftp* -sC -sS -sV"

View file

@ -60,6 +60,25 @@ curl -s XPUT ''$myES'_template/.*' -H 'Content-Type: application/json' -d'
}'
echo
# Set logstash template
echo -n $myCOL1"### Setting up logstash template: "$myCOL0
curl -s XPUT ''$myES'_template/logstash' -H 'Content-Type: application/json' -d'
{
"index_patterns": "logstash-*",
"settings" : {
"index" : {
"number_of_shards": 1,
"number_of_replicas": 0,
"mapping" : {
"total_fields" : {
"limit" : "2000"
}
}
}
}
}'
echo
# Restore indices
curl -s -X DELETE ''$myES'.kibana*' > /dev/null
for i in $myINDICES;

Binary file not shown.

Before

Width:  |  Height:  |  Size: 374 KiB

After

Width:  |  Height:  |  Size: 258 KiB

View file

@ -17,6 +17,7 @@ RUN apk -U --no-cache add \
openssl-dev \
python \
python-dev \
py-bcrypt \
py-mysqldb \
py-pip \
py-requests \

View file

@ -4,7 +4,12 @@
"settings" : {
"index.refresh_interval" : "5s",
"index.number_of_shards" : "1",
"index.number_of_replicas" : "0"
"index.number_of_replicas" : "0",
"mapping" : {
"total_fields" : {
"limit" : "2000"
}
}
},
"mappings" : {
"_default_" : {

View file

@ -4,7 +4,8 @@
"settings" : {
"index.refresh_interval" : "5s",
"index.number_of_shards" : "1",
"index.number_of_replicas" : "0"
"index.number_of_replicas" : "0",
"index.mapping.total_fields.limit": "2000"
},
"mappings" : {
"_default_" : {

View file

@ -1,6 +1,13 @@
# Input section
input {
# Fatt
file {
path => ["/data/fatt/log/fatt.log"]
codec => json
type => "Fatt"
}
# Suricata
file {
path => ["/data/suricata/log/eve.json"]
@ -57,12 +64,6 @@ input {
type => "ElasticPot"
}
# Glastopf
file {
path => ["/data/glastopf/log/glastopf.log"]
type => "Glastopf"
}
# Glutton
file {
path => ["/data/glutton/log/glutton.log"]
@ -128,6 +129,27 @@ input {
# Filter Section
filter {
# Fatt
if [type] == "Fatt" {
date {
match => [ "timestamp", "ISO8601" ]
}
mutate {
rename => {
"sourceIp" => "src_ip"
"destinationIp" => "dest_ip"
"sourcePort" => "src_port"
"destinationPort" => "dest_port"
"gquic" => "fatt_gquic"
"http" => "fatt_http"
"rdp" => "fatt_rdp"
"ssh" => "fatt_ssh"
"tls" => "fatt_tls"
}
}
}
# Suricata
if [type] == "Suricata" {
date {
@ -243,17 +265,6 @@ filter {
}
}
# Glastopf
if [type] == "Glastopf" {
grok {
match => [ "message", "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE}%{SPACE}%{IP:src_ip}%{SPACE}%{WORD}%{SPACE}%{URIPROTO:http_method}%{SPACE}%{NOTSPACE:http_uri}%{SPACE}%{NOTSPACE}%{SPACE}%{HOSTNAME}:%{NUMBER:dest_port:integer}" ]
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
remove_field => ["timestamp"]
}
}
# Glutton
if [type] == "Glutton" {
date {
@ -406,7 +417,7 @@ if "_grokparsefailure" in [tags] { drop {} }
}
# Add T-Pot hostname and external IP
if [type] == "Adbhoney" or [type] == "Ciscoasa" or [type] == "ConPot" or [type] == "Cowrie" or [type] == "Dionaea" or [type] == "ElasticPot" or [type] == "Glastopf" or [type] == "Glutton" or [type] == "Honeytrap" or [type] == "Heralding" or [type] == "Honeypy" or [type] == "Mailoney" or [type] == "Medpot" or [type] == "P0f" or [type] == "Rdpy" or [type] == "Suricata" or [type] == "Tanner" {
if [type] == "Adbhoney" or [type] == "Ciscoasa" or [type] == "ConPot" or [type] == "Cowrie" or [type] == "Dionaea" or [type] == "ElasticPot" or [type] == "Fatt" or [type] == "Glutton" or [type] == "Honeytrap" or [type] == "Heralding" or [type] == "Honeypy" or [type] == "Mailoney" or [type] == "Medpot" or [type] == "P0f" or [type] == "Rdpy" or [type] == "Suricata" or [type] == "Tanner" {
mutate {
add_field => {
"t-pot_ip_ext" => "${MY_EXTIP}"

41
docker/fatt/Dockerfile Normal file
View file

@ -0,0 +1,41 @@
FROM alpine
#
# Include dist
#ADD dist/ /root/dist/
#
# Get and install dependencies & packages
RUN apk -U add \
git \
py3-libxml2 \
py3-lxml \
python3 \
python3-dev && \
apk -U add tshark --repository http://dl-3.alpinelinux.org/alpine/edge/community/ && \
#
# Setup user
addgroup -g 2000 fatt && \
adduser -S -s /bin/ash -u 2000 -D -g 2000 fatt && \
#
# Install fatt
mkdir -p /opt && \
cd /opt && \
git clone --depth=1 https://github.com/0x4D31/fatt && \
cd fatt && \
mkdir -p log && \
pip3 install --upgrade pip && \
pip3 install pyshark==0.4.2.2 && \
#
# Setup configs
chown fatt:fatt -R /opt/fatt/* && \
#
# Clean up
apk del --purge git \
python3-dev && \
rm -rf /root/* && \
rm -rf /var/cache/apk/*
#
# Start fatt
STOPSIGNAL SIGINT
ENV PYTHONPATH /opt/fatt
WORKDIR /opt/fatt
CMD python3 fatt.py -i $(/sbin/ip address | grep '^2: ' | awk '{ print $2 }' | tr -d [:punct:]) --print_output --json_logging -o log/fatt.log

View file

@ -0,0 +1,17 @@
version: '2.3'
services:
# Fatt service
fatt:
build: .
container_name: fatt
restart: always
network_mode: "host"
cap_add:
- NET_ADMIN
- SYS_NICE
- NET_RAW
image: "dtagdevsec/fatt:1903"
volumes:
- /data/fatt/log:/opt/fatt/log

View file

@ -1,6 +1,6 @@
[![](https://images.microbadger.com/badges/version/dtagdevsec/glastopf:1903.svg)](https://microbadger.com/images/dtagdevsec/glastopf:1903 "Get your own version badge on microbadger.com") [![](https://images.microbadger.com/badges/image/dtagdevsec/glastopf:1903.svg)](https://microbadger.com/images/dtagdevsec/glastopf:1903 "Get your own image badge on microbadger.com")
# glastopf
# glastopf (deprecated)
[glastopf](https://github.com/mushorg/glastopf) is a python web application honeypot.

View file

@ -4,7 +4,7 @@ FROM alpine
ADD dist/ /root/dist/
# Get and install dependencies & packages
RUN apk -U --no-cache add \
RUN apk -U add \
build-base \
git \
libffi-dev \
@ -14,8 +14,6 @@ RUN apk -U --no-cache add \
python-dev \
py-pip \
py-setuptools && \
apk -U --no-cache add --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ \
py-qt && \
# Setup user
addgroup -g 2000 rdpy && \
@ -52,8 +50,7 @@ RUN apk -U --no-cache add \
libffi-dev \
openssl-dev \
python-dev \
py-pip \
py-qt && \
py-pip && \
rm -rf /var/cache/apk/*
# Start rdpy

View file

@ -1,11 +1,11 @@
FROM alpine
#
# Include dist
ADD dist/ /root/dist/
#
# Install packages
RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
apk -U --no-cache add \
#RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
RUN apk -U --no-cache add \
ca-certificates \
curl \
file \
@ -51,16 +51,16 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
py2-pip \
rust \
yaml-dev && \
#
# Upgrade pip, install virtualenv
pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir suricata-update && \
#
# Get and build Suricata
mkdir -p /opt/builder/ && \
wget https://www.openinfosecfoundation.org/download/suricata-4.1.3.tar.gz && \
tar xvfz suricata-4.1.3.tar.gz --strip-components=1 -C /opt/builder/ && \
rm suricata-4.1.3.tar.gz && \
wget https://www.openinfosecfoundation.org/download/suricata-4.1.4.tar.gz && \
tar xvfz suricata-4.1.4.tar.gz --strip-components=1 -C /opt/builder/ && \
rm suricata-4.1.4.tar.gz && \
cd /opt/builder && \
./configure \
--prefix=/usr \
@ -80,7 +80,7 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
make check && \
make install && \
make install-full && \
#
# Setup user, groups and configs
addgroup -g 2000 suri && \
adduser -S -H -u 2000 -D -g 2000 suri && \
@ -89,12 +89,12 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
cp /root/dist/*.bpf /etc/suricata/ && \
mkdir -p /etc/suricata/rules && \
cp /opt/builder/rules/* /etc/suricata/rules/ && \
#
# Download the latest EmergingThreats ruleset, replace rulebase and enable all rules
cp /root/dist/update.sh /usr/bin/ && \
chmod 755 /usr/bin/update.sh && \
update.sh OPEN && \
#
# Clean up
apk del --purge \
automake \
@ -124,7 +124,7 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
rm -rf /opt/builder && \
rm -rf /root/* && \
rm -rf /var/cache/apk/*
#
# Start suricata
STOPSIGNAL SIGINT
CMD SURICATA_CAPTURE_FILTER=$(update.sh $OINKCODE) && exec suricata -v -F $SURICATA_CAPTURE_FILTER -i $(/sbin/ip address | grep '^2: ' | awk '{ print $2 }' | tr -d [:punct:])

View file

@ -1,4 +1,4 @@
not (host sicherheitstacho.eu or community.sicherheitstacho.eu) and
not (host archive.ubuntu.com or security.ubuntu.com) and
not (host deb.debian.org) and
not (host index.docker.io or docker.io) and
not (host hpfeeds.sissden.eu)

View file

@ -406,6 +406,20 @@ services:
#### NSM
##################
# Fatt service
fatt:
build: .
container_name: fatt
restart: always
network_mode: "host"
cap_add:
- NET_ADMIN
- SYS_NICE
- NET_RAW
image: "dtagdevsec/fatt:1903"
volumes:
- /data/fatt/log:/opt/fatt/log
# P0f service
p0f:
container_name: p0f

View file

@ -16,8 +16,7 @@
/data/dionaea/dionaea-errors.log
/data/elasticpot/log/elasticpot.log
/data/elk/log/*.log
/data/glastopf/log/glastopf.log
/data/glastopf/db/glastopf.db
/data/fatt/log/fatt.log
/data/glutton/log/*.log
/data/glutton/log/*.err
/data/heralding/log/*.log
@ -38,7 +37,7 @@
{
su tpot tpot
copytruncate
create 760 tpot tpot
create 770 tpot tpot
daily
missingok
notifempty

Binary file not shown.

Binary file not shown.

Binary file not shown.

View file

@ -771,7 +771,7 @@ mkdir -p /data/adbhoney/downloads /data/adbhoney/log \
/data/dionaea/log /data/dionaea/bistreams /data/dionaea/binaries /data/dionaea/rtp /data/dionaea/roots/ftp /data/dionaea/roots/tftp /data/dionaea/roots/www /data/dionaea/roots/upnp \
/data/elasticpot/log \
/data/elk/data /data/elk/log \
/data/glastopf/log /data/glastopf/db \
/data/fatt/log \
/data/honeytrap/log/ /data/honeytrap/attacks/ /data/honeytrap/downloads/ \
/data/glutton/log \
/data/heralding/log \

View file

@ -218,7 +218,7 @@ mkdir -p /data/adbhoney/downloads /data/adbhoney/log \
/data/dionaea/log /data/dionaea/bistreams /data/dionaea/binaries /data/dionaea/rtp /data/dionaea/roots/ftp /data/dionaea/roots/tftp /data/dionaea/roots/www /data/dionaea/roots/upnp \
/data/elasticpot/log \
/data/elk/data /data/elk/log \
/data/glastopf/log /data/glastopf/db \
/data/fatt/log \
/data/honeytrap/log/ /data/honeytrap/attacks/ /data/honeytrap/downloads/ \
/data/glutton/log \
/data/heralding/log \