Infoblox + Palo Alto: How to format TIDE threat feeds for Palo Alto EDLs [ Edited ]

[#1]

This post is meant to cover formatting a TIDE threat feed into an ingestable format for any Palo Alto series firewall.  The following document is referenced:

https://www.infoblox.com/wp-content/uploads/infoblox-deployment-guide-implementing-infoblox-tide-fee...

The part this document will elaborate upon is this line in the integration doc:

• Access to the Infoblox TIDE website to download the data feeds.
• A VM (virtual machine) or workstation to modify the feeds per the Palo Alto Networks data formats. Per
the ‘Formatting Guidelines for an External Dynamic List’ section in the PAN OS administrators guide for
formatting information:
o Remove the quotes.
o Remove the field headers (i.e. IP, URL, host).
o Remove HTTP:// and HTTPS:// from the URLs

As far as I can tell, no documentation is available yet that elaborates on this process, which is what I will attempt to do here.  Disclaimer now, I'm not a programmer, my work is probably not optimal.  The point here is that this does work - if you're like me and writing API calls are the enemy this document is for you.  

What this document will not cover:  

The palo alto integration piece - the above reference doc covers how to feed this stuff in as an EDL to the firewall

Requirements for the following procedure:

- Basic linux knowledge, how to use VI, navigate directories, etc.

- CentOS7 (or your preferred flavor of linux, this was written using Centos7)

- Already configured repo's for your linux flavor, all packages required are available via yum

- Apache already installed/working on your Linux machine, this doc will not cover that scope.

- A TIDE login to generate an API key

Ok finally, the technical stuff.  The first place we will start is this reference doc on the TIDE webpage: https://platform.activetrust.net/APIGettingStartedGuide.pdf

Alright now that we have our 2 reference docs available, lets get started in CentOS.  Before we do anything with the reference material, we need to install python-pip in our linux environment.  Here is a quick and dirty reference page, all credit to the author listed.  https://linuxize.com/post/how-to-install-pip-on-centos-7/

From elevated linux privileges run this set of commands:

sudo yum install epel-release

sudo yum install python-pip

pip --version 

yum update python-pip

These commands will enable the EPEL repository, which is required for PIP.  The rest of the commands are just installing python then updating as required.  In my example my python-pip version is 19.0.2.  From my output:

[root@librenms ~]# pip --version
pip 19.0.2 from /usr/lib/python2.7/site-packages/pip (python 2.7)

Alright now that Python is prep'd and ready, navigate to /root/scripts and we'll start actually getting some work done.  In the above reference TIDE document you'll find an API call template that looks exactly like this:

#note: install the 'requests' library first:

#pip install -U requests

import requests

from pprint import pprint

#note: replace this api_key value with your api key!

api_key = 'YOUR_API_KEY'

api_endpoint = 'https://api.activetrust.net:8000'

api_path = '/api/data/threats/state/host'

url = '%s%s' % (api_endpoint,api_path)

params = {'rlimit': 2}

r = requests.get(url,params=params,auth=(api_key,''),verify=True)

print r.status_code

pprint(r.json())

From here the first thing we have to do is replace the api_key field with an actual API key.  From your TIDE dashboard, click on your account in the top right, account settings.  API keys can be generated here and just copy/pasted into the script.  From /root/scripts created a new file with your preferred text editor.  I'm using VI just because its native.  What I generally do is write everything in notepad++ and just paste it into my VI session.  

From /root/scripts:

- vi infoblox_threat_ingest.py

- i (for vi Insert mode)

- paste the above script with an active API key

- escape once, then :wq!

Once out of VI editor and back in /root/scripts:

- chmod 775 infoblox_threat_ingest.py

- chmod +x infoblox_threat_ingest.py

From here you can run your first test - just execute the script with "python infoblox_threat_ingest.py".  The output is restricted to 2 lines of output - this is specified in the script with params = {'rlimit': 2}.  This example is using the 'host' threatfeed from TIDE, to modify this to URL or IP, in the base python script modify the api_path field to the following examples:

IP - api_path = '/api/data/threats/state/ip'

URL - api_path = '/api/data/threats/state/url'

In my /root/scripts I have 3 of these scripts - one for each host/url/IP ingests.  

The output of the hosts script restricted to 2 looks something like this (ignore the title testscript.py, replace with your name as applicable):

[root@librenms scripts]# python testscript.py
200
{u'record_count': 2,
u'threat': [{u'batch_id': u'3da53757-b6de-11e4-a4da-f5f60c55f379',
u'class': u'APT',
u'detected': u'2015-02-17T19:47:01.000Z',
u'dga': False,
u'domain': u'tropiccritics.com',
u'expiration': u'2035-02-17T19:47:01.000Z',
u'host': u'tropiccritics.com',
u'id': u'3db144ae-b6de-11e4-a4da-f5f60c55f379',
u'imported': u'2015-02-17T19:47:01.000Z',
u'profile': u'IID',
u'property': u'APT_MalwareC2',
u'received': u'2015-02-17T19:47:01.000Z',
u'threat_level': 100,
u'tld': u'com',
u'type': u'HOST',
u'up': True},
{u'batch_id': u'3da53757-b6de-11e4-a4da-f5f60c55f379',
u'class': u'APT',
u'detected': u'2015-02-17T19:47:01.000Z',
u'dga': False,
u'domain': u'afkarehroshan.com',
u'expiration': u'2035-02-17T19:47:01.000Z',
u'host': u'afkarehroshan.com',
u'id': u'3db11e3f-b6de-11e4-a4da-f5f60c55f379',
u'imported': u'2015-02-17T19:47:01.000Z',
u'profile': u'IID',
u'property': u'APT_MalwareC2',
u'received': u'2015-02-17T19:47:01.000Z',
u'threat_level': 100,
u'tld': u'com',
u'type': u'HOST',
u'up': True}]}

Now while this is technically... something, its useless for the Palo Alto ingest.  Palo Alto EDL's require raw text with no formatting + linebreaks.  Now is when we have to write some home-brew formatting into our scripts.  The following example is a straight copy/paste from my production scripts.  This example uses my IP threat feed ingest and formats it appropriately.   We're just taking that original tempalate and adding a bunch of new lines.  Modifications are bolded.  

#!/usr/bin/python

#note: install the 'requests' library first:

#pip install -U requests

import json

import requests

from pprint import pprint

#note: replace this api_key value with your api key!

api_key = 'redacted'

api_endpoint = 'https://api.activetrust.net:8000'

api_path = '/api/data/threats/state/ip'

url = '%s%s' % (api_endpoint,api_path)

#params = {'rlimit': 5}

params = {}

r = requests.get(url,params=params,auth=(api_key,''),verify=True)

#print r.status_code

data = json.loads(r.text);

#print data

for i in range(data['record_count']):

    print data["threat"][i]['ip']

I dont want to get too deep on these changes, so this is the quick view.  Import json is required for parsing output, the params field restriction is lifted from 5 to infinite, and json is parsing out just the IP feed which is what we want.  While this example uses the IP threat feed, update the api_path fields as needed for URL or Host.  In the final lines filtering as well, update ['ip'] to URL or Host as well.  The sample output will look something like this:

212.117.188.23
36.81.53.34
103.60.223.105
220.176.25.152
197.251.92.100
201.41.206.196
125.36.166.118
193.225.63.140
83.96.71.178
190.177.146.93
27.44.127.58
189.11.112.147
27.44.127.196
121.32.236.160
37.107.104.230
95.218.218.28
120.244.118.182
42.90.148.178

Boom, all the IP's we need for our threat feed extracted from that gross formatting.  URL scripts will give an output that looks like this:

https://mainteanance-upgraed.secure1-appsservices.net/IDMSWebAuth?appIdKey=VuwePaYPEAB0qSiEKfPz09007...
https://ax6dmuh8li7order-store-apprefund.cf/account/
http://etherchange.tech/
https://webs.apple.com-auth.signwe43mailreports2345.catawcta.com/IDMSWebAuth?appIdKey=UJqcnIpOkGTNXo...
https://iforgot.support.account.foxhoostqiluem.com/_
https://apple-appleid.com.bsdata.info/account/?view=login&appIdKey=7d8aeabab23541050654be1fd&country...
http://www.updater.com.nvpphaytech.duckdns.org/pp/hmtc2mzi%3D/signin

Alright we've got filtered output, now we have to get this into a webpage since that is how the Palo Alto EDL's work is referencing a source page.  This is where if you are an advanced user - do whatever you feel comfortable with.  This is how I did it.  

We'll schedule all of your threat feed scripts to run in cron, every 30 minutes, then dump to a target directory.  From any directory, enter:

crontab -e

i (ala VI insert mode)

*/30 * * * * /root/scripts/infoblox-threat-ip.py > /var/www/html/infoblox/infoblox_threat_ip_filtered
*/30 * * * * /root/scripts/infoblox-threat-url.py > /var/www/html/infoblox/infoblox_threat_url_filtered

escape

:wq!

Change your script names and directory names as applicable, however what this does is execute the scripts in /root/scripts every 30 minutes, redirects this output to /var/www/html/infoblox/infoblox_threat_ip_filtered and also overrwrite any previous files present in the target directory.  

From here we take those output files in /var/www/html and present them with Apache.  This document is already way too long so I cant get too indepth on apache setup and install, so I'm assuming Apache packages are already present and running.  Navigate to /etc/httpd/conf and open httpd.conf with VI or your favorite text editor.  In my case, this server is also running my monitoring software so theres already a full httpd.conf and I just added these lines to create an alias.

<VirtualHost *:80>

        DocumentRoot "/var/www/html"

        ServerName www.librenms.domain.net

        Alias /threat "/var/www/html/infoblox"

        <Directory "/var/www/html/infoblox">

        Options +Indexes

        AllowOverride None

        Order allow,deny

        Allow from all

        </Directory>

</VirtualHost>

All this is doing is directing the Apache server to answer queries on "www.librenms.domain/threat" and directing them to the /var/www/html/infoblox directory internally.  From here you will get presented a very basic web page where you can just click on the filtered threat output.  Voila, now you have a web-hosted, filtered, threat feed for Palo Alto ingest.  

Disclaimer again:  I'm not a programmer, I'm a hack who can use google.  This is meant for people who have very basic knowledge to complete a relatively complex task.  Please disseminate this information at your leisuire, just reference this post as a source.  I've stolen enough from stackoverflow and everywhere else, its time to give back.  

Please keep any discussion specific to ingesting + formatting for EDL's.  Specific EDL integration is covered thoroughly in the infoblox documentation, it's very straight forward if you have used EDL's or Palo Alto firewalls before at all.  

[#2]

Thank you for posting this!

Note that you can use Field parameters to limit the verbose field output on TIDE API Calls. For example:

api_path = '/api/data/threats/state/ip&field=ip'

instead of:

api_path = '/api/data/threats/state/ip'

I haven't tested this out in py but just wanted to point out that you can limit the field output. Also, :8000 is not required to perform these calls. Hope this is helpful.

[#3]

update on this one.

You can also specify the parameter:

threat_level_from=80 to get threats of 80 and above.