Suricata: improve setup/config for ETPRO ruleset

PROBLEM (see #487)
- ET rule files start with the "emerging-*" prefix;
- ETPRO rule files do not start with that prefix.

SOLUTION
Concatenate all rule files, with the exception of "*disabled.rules",
into a single rule file at "/etc/suricata/rules/tpotce.rules"

I have left as-is the "sed" command that enables all commented-out
rules. Since that is usually done for performance reasons, maybe it
could be turned into a configuration option like the OINKCODE.

Another thing worth considering is to use "suricata-update" instead:
https://suricata-update.readthedocs.io/en/latest/quickstart.html

docker/suricata/Dockerfile

@@ -20,7 +20,7 @@ RUN apk -U --no-cache add \
     cp /root/dist/suricata.yaml /etc/suricata/suricata.yaml && \
     cp /root/dist/*.bpf /etc/suricata/ && \
 #
-# Download the latest EmergingThreats ruleset, replace rulebase and enable all rules
+# Download the latest EmergingThreats ruleset, replace rulebase and enable all rules except disabled ones
     cp /root/dist/update.sh /usr/bin/ && \
     chmod 755 /usr/bin/update.sh && \
     update.sh OPEN && \

docker/suricata/dist/suricata.yaml

@@ -1815,68 +1815,21 @@ napatech:
 default-rule-path: /etc/suricata/rules
 
 rule-files:
- - botcc.rules
- - botcc.portgrouped.rules
- - ciarmy.rules
- - compromised.rules
- - drop.rules
- - dshield.rules
- - emerging-activex.rules
- - emerging-adware_pup.rules
- - emerging-attack_response.rules
- - emerging-chat.rules
- - emerging-coinminer.rules
- - emerging-current_events.rules
- - emerging-dns.rules
- - emerging-dos.rules
- - emerging-exploit.rules
- - emerging-exploit_kit.rules
- - emerging-ftp.rules
- - emerging-games.rules
- - emerging-hunting.rules
- - emerging-icmp_info.rules
- - emerging-icmp.rules
- - emerging-imap.rules
- - emerging-inappropriate.rules
- - emerging-info.rules
- - emerging-ja3.rules
- - emerging-malware.rules
- - emerging-misc.rules
- - emerging-mobile_malware.rules
- - emerging-netbios.rules
- - emerging-p2p.rules
- - emerging-phishing.rules
- - emerging-policy.rules
- - emerging-pop3.rules
- - emerging-rpc.rules
- - emerging-scada.rules
- - emerging-scan.rules
- - emerging-shellcode.rules
- - emerging-smtp.rules
- - emerging-snmp.rules
- - emerging-sql.rules
- - emerging-telnet.rules
- - emerging-tftp.rules
-# - emerging-trojan.rules
- - emerging-user_agents.rules
- - emerging-voip.rules
- - emerging-web_client.rules
- - emerging-web_server.rules
- - emerging-web_specific_apps.rules
- - emerging-worm.rules
- - tor.rules
- - decoder-events.rules # available in suricata sources under rules dir
- - stream-events.rules  # available in suricata sources under rules dir
- - http-events.rules    # available in suricata sources under rules dir
- - smtp-events.rules    # available in suricata sources under rules dir
- - dns-events.rules     # available in suricata sources under rules dir
- - tls-events.rules     # available in suricata sources under rules dir
- - modbus-events.rules  # available in suricata sources under rules dir
- - app-layer-events.rules  # available in suricata sources under rules dir
- - dnp3-events.rules       # available in suricata sources under rules dir
- - ntp-events.rules       # available in suricata sources under rules dir
- - ipsec-events.rules       # available in suricata sources under rules dir
- - kerberos-events.rules       # available in suricata sources under rules dir
+# generated by update.sh, contains ET/ETPRO ruleset
+ - tpotce.rules
+# available in suricata sources under rules dir
+ - app-layer-events.rules
+ - decoder-events.rules
+ - dnp3-events.rules
+ - dns-events.rules
+ - http-events.rules
+ - ipsec-events.rules
+ - kerberos-events.rules
+ - modbus-events.rules
+ - ntp-events.rules
+ - smtp-events.rules
+ - stream-events.rules
+ - tls-events.rules
 
 ##
 ## Auxiliary configuration files.

docker/suricata/dist/update.sh

@@ -27,6 +27,22 @@ if [ "$myOINKCODE" != "" ] && [ "$myOINKCODE" == "OPEN" ];
 fi
 }
 
+function fuENRULES {
+  # Cleanup old files and extract new files.
+  rm -rf /tmp/rules /tmp/tpotce.rules
+  tar xfz /tmp/rules.tar.gz -C /tmp/ 2>&1 > /dev/null
+  # Create the new ruleset by:
+  # - looping through rule files, except deleted ones;
+  # - enabling all disabled rules (performance should be OK);
+  # - removing unnecessary empty/comment lines.
+  ls /tmp/rules/*.rules | grep -v deleted.rules | while read f;
+    do
+      cat $f | sed "s/^#alert/alert/" | grep -Ev "^(#|$)" >> /tmp/tpotce.rules
+    done
+  # Copy the new ruleset and config to where they belong.
+  cp -f /tmp/tpotce.rules /tmp/rules/classification.config /etc/suricata/rules
+}
+
 # Check internet availability 
 function fuCHECKINET () {
 mySITES=$1
@@ -47,8 +63,7 @@ myCHECK=$(fuCHECKINET "rules.emergingthreatspro.com rules.emergingthreats.net")
 if [ "$myCHECK" == "0" ];
   then
     fuDLRULES 2>&1 > /dev/null
-    tar xvfz /tmp/rules.tar.gz -C /etc/suricata/ 2>&1 > /dev/null
-    sed -i s/^#alert/alert/ /etc/suricata/rules/*.rules 2>&1 > /dev/null
+    fuENRULES 2>&1 > /dev/null
     echo "/etc/suricata/capture-filter.bpf"
   else
     echo "/etc/suricata/null.bpf"