PyMISP - Python Library to Access MISP

Last modified: Wed Oct 02 2024 16:09:21 GMT+0200 (Central European Summer Time)

PyMISP - Python Library to access MISP

PyMISP is a Python library to access MISP platforms via their REST API.

PyMISP allows you to fetch events, add or update events/attributes, add or update samples or search for attributes.

Note that you need to have Auth Key access in your MISP instance to use PyMISP

Capabilities

Installation

You can install PyMISP by either using pip or by getting the last version from the GitHub repository

Install from pip

pip install pymisp

Install the latest version from the repository

git clone https://github.com/MISP/PyMISP.git && cd PyMISP
python setup.py install

Note that you will also need to install requests if you don't have it already.

Getting started

You now need to get your automation key. You can find it on the automation page:

https://<misp url>/events/automation

or on your profile

https://<misp url>/users/view/me

If you did not install using the repository, you can still fetch it to get examples to work on:

git clone https://github.com/MISP/PyMISP.git

In order to use these, you need to create a file named keys.py in the examples folder and edit it to put the url of your MISP instance and your automation key.

cd examples
cp keys.py.sample keys.py
vim keys.py

Once you are done with it, you are ready to start.

This is how keys.py looks:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

misp_url = 'https:///'
misp_key = 'Your MISP auth key' # The MISP auth key can be found on the MISP web interface under the automation section
misp_verifycert = True

Using PyMISP

To have a better understanding of how to use PyMISP, we will have a look at one of the existing examples: add_named_attribute.py This script allow us to add an attribute to an existing event while knowing only its type (the category is determined by default).

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from pymisp import PyMISP
from keys import misp_url, misp_key
import argparse

First of all, it is obvious that we need to import PyMISP. Then we also need to know both the instance with which we will work and the API key to use: Both should be stored in the keys.py file. Finally we import argparse library so the script can handle arguments.

# For python2 & 3 compat, a bit dirty, but it seems to be the least bad one
try:
    input = raw_input
except NameError:
    pass

Just a few lines to be sure that python 2 and 3 are supported

def init(url, key):
    return PyMISP(url, key, True, 'json', debug=True)

This function will create a PyMISP object that will be used later to interact with the MISP instance. As seen in the api.py, a PyMISP object need to know both the URL of the MISP instance and the API key to use. It can also take additional and not mandatory data, such as the use or not of SSL or the name of the export format.

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Create an event on MISP.')
    parser.add_argument("-e", "--event", type=int, help="The id of the event to update.")
    parser.add_argument("-t", "--type", help="The type of the added attribute")
    parser.add_argument("-v", "--value", help="The value of the attribute")
    args = parser.parse_args()

Then the function starts by preparing the awaited arguments:

Existing examples

As the name implies you will find several example scripts in the examples folder. For each you can get help if you do scriptname.py -h

Let us have a look at some of these examples:

add_named_attribute.py

Allow to add an argument to an existing event by giving only the type of the attribute. The category will be set with a default value.

Arguments:

add_user.py

Allow to add a user by giving the mandatory fields as entries.

Arguments:

add_user_json.py

Add the user described in the given json. If no file is provided, returns a json listing all the fields used to describe a user.

Arguments:

create_events.py

Allow a user to create a new event on the MISP instance.

Arguments:

del.py

Delete an event or an attribute from a MISP instance. The event has the priority: if both are set, only the event will be deleted.

Arguments:

delete_user.py

Delete the user with the given id. Keep in mind that disabling users (by setting the disabled flag via an edit) is always preferred to keep user associations to events intact.

Arguments:

edit_user.py

Edit the email of the user designed by the user_id.

Arguments:

edit_user_json.py

Edit the user designed by the user_id. If no file is provided, returns a json listing all the fields used to describe a user.

Arguments:

get.py

Get an event from a MISP instance in json format.

Arguments:

last.py

Download latest events from a MISP instance. A output file can be created to store these events.

Arguments:

searchall.py

Get all the events matching a value.

Arguments:

sharing_groups.py

Get a list of the sharing groups from the MISP instance. No argument.

sighting.py

Add sighting.

Arguments:

stats.py

Output attributes statistics from a MISP instance. No argument.

suricata.py

Download Suricata events.

Arguments:

tags.py

Get tags from MISP instance. No argument.

tagstatistics.py

Get statistics from tags.

Arguments:

up.py

Update an existing event regarding the data inside a given json file.

Arguments:

upload.py

Send malware sample to MISP.

Arguments:

users_list.py

Get a list of the sharing groups from the MISP instance. No argument.

Going further

feed-generator

It is used to generate the CIRCL OSINT feed. This script export the events as json, based on tags, organisation, events, ... It automatically update the dumps and the metadata file.

Here is an example of a config file:

url = ''
key = ''
ssl = True
outputdir = 'output'
# filters = {'tag' : 'tlp : white|feed-export|!privint', 'org':'CIRCL'}
filters = {}

valid_attribute_distribution_levels = ['0', '1', '2', '3', '4', '5']

Consuming feed

As the feed is a simple set of MISP json files, the files can be easily imported directly into any MISP instance. The script below processes the manifest file of an OSINT feed and reimport them in a MISP directly.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from pymisp import PyMISP
import requests

url = 'https://www.circl.lu/doc/misp/feed-osint/'
osintcircl = requests.get('{}manifest.json'.format(url))

misp = PyMISP('http://misp.test/', 'key', False, 'json')
for uri in osintcircl.json():
        req = requests.get('{}{}.json'.format(url,uri))
        misp.add_event(req.json())

ioc-2-misp

Allow to import OpenIOC files into MISP easily. It is also possible to set specific tags on these events.

Situational Awareness

Simple example on fetching the last events

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from pymisp import PyMISP
from keys import misp_url, misp_key, misp_verifycert
import argparse
import os
import json


# Usage for pipe masters: ./last.py -l 5h | jq .


def init(url, key):
    return PyMISP(url, key, misp_verifycert, 'json')


def download_last(m, last, out=None):
    result = m.download_last(last)
    if out is None:
        if 'response' in result:
            print(json.dumps(result['response']))
        else:
            print('No results for that time period')
            exit(0)
    else:
        with open(out, 'w') as f:
            f.write(json.dumps(result['response']))

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Download latest events from a MISP instance.')
    parser.add_argument("-l", "--last", required=True, help="can be defined in days, hours, minutes (for example 5d or 12h or 30m).")
    parser.add_argument("-o", "--output", help="Output file")

    args = parser.parse_args()

    if args.output is not None and os.path.exists(args.output):
        print('Output file already exists, abord.')
        exit(0)

    misp = init(misp_url, misp_key)

    download_last(misp, args.last, args.output)