Automation and MISP API

Last modified: Wed Oct 02 2024 16:09:21 GMT+0200 (Central European Summer Time)

Automation API

Automation functionality is designed to automatically generate signatures for intrusion detection systems. To enable signature generation for a given attribute, Signature field of this attribute must be set to Yes. Note that not all attribute types are applicable for signature generation, currently we only support NIDS signature generation for IP, domains, host names, user agents etc., and hash list generation for MD5/SHA1 values of file artefacts. Support for more attribute types is planned. To make this functionality available for automated tools an authentication key is used. This makes it easier for your tools to access the data without further form-based-authentication. The API key can be found and managed under My Profile page (/users/view/me) on a MISP instance.

General

Automation URL

The documentation will include a default MISP URL in the examples. Don't forget to replace it with your MISP URL.

Default MISP URL in the documentation:

https://<misp url>/

Automation key

The authentication of the automation is performed via a secure key available in the MISP UI interface. Make sure you keep that key secret as it gives access to the entire database! The API key is available in the event actions menu under automation.

Since version 2.2 the usage of the authentication key in the URL is deprecated. Instead, pass the auth key in an Authorization header in the request. The legacy option of having the auth key in the URL is temporarily still supported but not recommended.

The authorization is performed by using the following header:

Authorization: YOUR API KEY

Creating an automation key (using advanced authkeys)

Using the menu, go to Global Actions > My Profile and click "Auth keys" to show the auth keys view.

Screenshot of My Profile view with Auth keys expanded

The following form will be displayed:

Screenshot of add authkey form

You can add an optional comment to indicate what the key will be used for.

You can also limit the usage of the key to specific IPs or subnets (one per line), by adding them in the Allowed IPs field. On some instances it is mandatory to set an IP allowlist. When adding subnets, please note that you need to use the format network_ip/subnet_mask .

You can optionally set an expiration time for the key.

Finally, it is also possible to make this key read-only, meaning that it will not be possible to do any changes on this instance using this automation key.

After clicking submit you will get a confirmation that the auth key was created, the key will be shown only one time.

Screenshot showing success message that is displayed when an ip was successfully pinned for an authkey

The same fields are available when editing an automation key.

Pinning an allowed IP for an automation key (using advanced authkeys)

MISP will keep track of the unique IPs that were seen for a specific automation key. You can easily limit future usage of an automation key to one of the IPs that was seen in the past. To do so, using the menu, go to Global Actions > My Profile and click "Auth keys" to show the auth keys view. If the automation key was used in the past, you will see the "Seen IPs" listed per key. Click on the pin button next to the IP you want to limit usage to.

Screenshot showing auth keys view with the pin button available for seen IPs
You will get a pop up requesting confirmation that you want to pin this IP for the key:

Screenshot showing pop up which is displayed, requesting user confirmation after clicking the pin IP button
After confirmation, if all goes well, you will get a confirmation that the allowed IP was set for the automation key:
Screenshot showing success message that is displayed when an ip was successfully pinned for an authkey

Accept and Content-Type headers

When performing your request, depending on the type of request, you might need to explicitly specify in what content type you want to get your results. This is done by setting one of the below Accept headers:

Accept: application/json
Accept: application/xml

When submitting data in a POST, PUT or DELETE operation you also need to specify in what content-type you encoded the payload. This is done by setting one of the below Content-Type headers:

Content-Type: application/json
Content-Type: application/xml

Example:

curl --header "Authorization: YOUR API KEY" --header "Accept: application/json" --header "Content-Type: application/json" https://<misp url>/

By appending .json or .xml the content type can also be set without the need for a header.

Automation using PyMISP

PyMISP is a Python library to access MISP platforms via their REST API.

PyMISP allows you to fetch events, add or update events/attributes, add or update samples or search for attributes.

PyMISP is available including a documentation with various examples.

Status Codes

To be done

Error Handling

Wrong endpoint chosen

Example

curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" http://10.50.13.60/servers/gaaa
{"name":"Not Found","message":"Not Found","url":"\/servers\/gaaa"}

It is possible to search in the database for a list of attributes or events based on a list of criteria.

To return attributes or events in a desired format, use the following URL and header settings:

URL:

YOUR_MISP_URL/attributes/restSearch
YOUR_MISP_URL/events/restSearch

Headers:

Accept: application/json
Content-type: application/json
Authorization: YOUR_API_KEY

The next feature to take care of then is the body of the query. This is where you are going to put your filters.



As an example, if we want to export all the IP addresses that have a TLP marking and not marked as TLP:red, you can find below the corresponding filters to use:

{
    "returnFormat": "json",
    "type": {
        "OR": [
            "ip-src",
            "ip-dst"
        ]
    },
    "tags": {
        "NOT": [
            "tlp:red"
        ],
        "OR": [
            "tlp:%"
        ]
    }
}

Find below a non exhaustive list of parameters that can be used to filter data in your search (some parameters specific to given export formats are not mentioned):

Events management

/events

Accepted Methods

Description

Receive, update or delete Events. There is also a good amount of special output formats that can be triggered.

GET /events

Description

Receive events based on criteria

URL Arguments

Output

[{"id":"1","org_id":"1","date":"2014-12-10","info":"OSINT - F-Secure W32\/Regin, Stage #1","uuid":"54884656-2da8-4625-bf07-43ef950d210b","published":true,"analysis":"2","attribute_count":"39","orgc_id":"2","timestamp":"1418217625","distribution":"3","sharing_group_id":"0","proposal_email_lock":false,"locked":false,"threat_level_id":"1","publish_timestamp":"1515749192","disable_correlation":false,"Org":{"id":"1","name":"ORGNAME"},"Orgc":{"id":"2","name":"CIRCL"},"EventTag":[{"id":"1","event_id":"1","tag_id":"1","Tag":{"id":"1","name":"Type:OSINT","colour":"#1eed40","exportable":true}}],"SharingGroup":{"id":null,"name":null}}]

Example

curl --header "Authorization: YOUR API KEY" --header "Accept: application/json" --header "Content-Type: application/json" https://<misp url>/

POST /events

Example

curl -i -H "Accept: application/json" -H "content-type: application/json" -H "Authorization: YOUR API KEY" --data "@event.json" -X POST http://10.50.13.60/events

That is how an event JSON object should look like

{"Event":{"date":"2015-01-01","threat_level_id":"1","info":"testevent","published":false,"analysis":"0","distribution":"0","Attribute":[{"type":"domain","category":"Network activity","to_ids":false,"distribution":"0","comment":"","value":"test.com"}]}}

DELETE /events

Description

Delete events based on criteria

URL Arguments

Output

{
    "name": "Event deleted.",
    "message": "Event deleted.",
    "url": "\/events\/delete\/1"
}

Example

curl --header "Authorization: YOUR API KEY" --header "Accept: application/json" --header "Content-Type: application/json" https:///

curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" -X "DELETE" http://10.50.13.60/events/1

GET /events/index

Description

Return the event index. - Warning, there's a limit on the number of results

Output

[{"id":"1","org_id":"1","date":"2014-12-10","info":"OSINT - F-Secure W32\/Regin, Stage #1","uuid":"54884656-2da8-4625-bf07-43ef950d210b","published":true,"analysis":"2","attribute_count":"39","orgc_id":"2","timestamp":"1418217625","distribution":"3","sharing_group_id":"0","proposal_email_lock":false,"locked":false,"threat_level_id":"1","publish_timestamp":"1515749192","disable_correlation":false,"Org":{"id":"1","name":"ORGNAME"},"Orgc":{"id":"2","name":"CIRCL"},"EventTag":[{"id":"1","event_id":"1","tag_id":"1","Tag":{"id":"1","name":"Type:OSINT","colour":"#1eed40","exportable":true}}],"SharingGroup":{"id":null,"name":null}}]

Example

curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" http://10.50.13.60/events/index

POST /events/AddTag

Add a tag or a tag collection to an existing event.

URL Arguments

Examples:

curl \
 -d '{"event":"1210","tag":"tlp:green"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/events/addTag
curl \
 -d '{"event":"1210","tag":"383"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/events/addTag
curl \
 -d '{"event":"1210","tag":"collection_1"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/events/addTag

POST /events/removeTag

Remove a tag from an existing event. Note that removing a tag collection in one go is not possible.

URL Arguments

Examples:

curl \
 -d '{"event":"1210","tag":"tlp:amber"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/events/removeTag
curl \
 -d '{"event":"1210","tag":"987"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/events/removeTag

GET /events/pushEventToZMQ/

Description

Will push an Event to ZMQ

URL Arguments

Example

curl --header "Authorization: YOUR API KEY" --header "Accept: application/json" --header "Content-Type: application/json" https://<misp url>events/pushEventToZMQ/223

GET /events/nids NIDS rules export

Automatic export of all network related attributes is available under the Snort or Suricata rule format. Only published events and attributes marked as IDS Signature are exported.

You can configure your tools to automatically download the following file:

https://<misp url>/events/nids/suricata/download
https://<misp url>/events/nids/snort/download

The full API syntax is as follows:

https://<misp url>/events/nids/[format]/download/[eventid]/[frame]/[tags]/[from]/[to]/[last]
format
The export format, can be "suricata" or "snort"
eventid
Restrict the download to a single event
frame
Some commented out explanation framing the data. The reason to disable this would be if you would like to concatenate a list of exports from various select events in order to avoid unnecessary duplication of the comments.
tags
To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). For example, to include tag1 and tag2 but exclude tag3 you would use:
https://<misp url>/events/nids/snort/download/false/false/tag1&&tag2&&!tag3
from
Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to
Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last
Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 6d or 12h or 30m). This filter will use the published timestamp of the event.

The keywords false or null should be used for optional empty parameters in the URL.

An example for a Suricata export for all events excluding those tagged tag1, without all of the commented information at the start of the file would look like this:

https://<misp url>/events/nids/suricata/download/null/true/!tag1

Administration is able to maintain an allowedlist containing host, domain name and IP numbers to exclude from the NIDS export.

GET /events/hids Hash - HIDS database export

Automatic export of MD5/SHA1 checksums contained in file-related attributes. This list can be used to feed forensic software when searching for suspicious files. Only published events and attributes marked as IDS Signature are exported.

You can configure your tools to automatically download all the MD5 hashes from MISP:

https://<misp url>/events/hids/md5/download

Or the SHA1 hashes:

https://<misp url>/events/hids/sha1/download

The API's full format is as follow:

https://<misp url>/events/hids/[format]/download/[tags]/[from]/[to]/[last]
format
The export format, can be "md5" or "sha1"
tags
To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead). For example, to include tag1 and tag2 but exclude tag3 you would use:
https://<misp url>/events/hids/md5/download/tag1&&tag2&&!tag3
from
Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to
Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last
Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.

The keywords false or null should be used for optional empty parameters in the URL.

For example, to only show sha1 values from events tagged tag1, use:

https://<misp url>/events/hids/sha1/download/tag1

GET /events/stix STIX export

You can export MISP events in MITRE's STIX format (to read more about STIX). The STIX XML export is currently very slow and can lead to timeouts with larger events or collections of events. The STIX JSON return format does not suffer from this issue.

Usage of the API:

https://<misp url>/events/stix/download

Search parameters can be passed to the function via URL parameters or by POSTing an xml or json object (depending on the return type). The following parameters can be passed to the STIX export tool: id, withAttachments, tags. Both id and tags can use the && (and) and ! (not) operators to build queries. Using the URL parameters, the syntax is as follows:

https://<misp url>/events/stix/download/[id]/[withAttachments]/[tags]/[from]/[to]/[last]
id
The event's ID
withAttachments
Encode attachments where applicable
tags
To include a tag in the results just write its names into this parameter. To exclude a tag prepend it with a '!'. You can also chain several tag commands together with the '&&' operator. Please be aware the colons (:) cannot be used in the tag search. Use semicolons instead (the search will automatically search for colons instead).

For example, to include tag1 and tag2 but exclude tag3 you would use:

https://<misp url>/events/stix/download/false/true/tag1&&tag2&&!tag3
from
Events with the date set to a date after the one specified in the from field (format: 2015-02-15). This filter will use the date of the event.
to
Events with the date set to a date before the one specified in the to field (format: 2015-02-15). This filter will use the date of the event.
last
Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m). This filter will use the published timestamp of the event.

You can post an XML or JSON object containing additional parameters in the following formats.

If you use JSON query objects:

https://<misp url>/events/stix/download.json
{"request": {"id":["!51","!62"],"withAttachment":false,"tags":["APT1","!OSINT"],"from":false,"to":"2015-02-15"}}

If you use XML query objects:

https://<misp url>/events/stix/download
<request><id>!51</id><id>!62</id><withAttachment>false</withAttachment><tags>APT1</tags><tags>!OSINT</tags><from>false</from><to>2015-02-15</to></request>

Various ways to narrow down the search results of the STIX export

For example, to retrieve all events tagged "APT1" but excluding events tagged "OSINT" and excluding events #51 and #62 without any attachments:

https://<misp url>/events/stix/download/!51&&!62/false/APT1&&!OSINT/2015-02-15

To export the same events using a POST request use:

https://<misp url>/events/stix/download.json

Together with this JSON object in the POST message:

{"request": {"id":["!51","!62"],"tags":["APT1","!OSINT"],"from":"2015-02-15"}}

XML is automatically assumed when using the STIX export:

https://<misp url>/events/stix/download

The same search could be accomplished using the following POSTed XML object (note that ampersands need to be escaped, or alternatively separate id and tag elements can be used):

<request><id>!51</id><id>!62</id><tags>APT1</tags><tags>!OSINT</tags><from>2015-02-15</from></request>

Tag management

POST /tags/add

Description

Add a tag on the instance

URL Arguments

Mandatory

Description

Attaches a tag to an object by a given UUID. Note that adding a tag collection via this endpoint is not possible. Please refer to /events/addTag and /attributes/addTag for that functionality.

This endpoint exists for convenience reasons and might be slightly less performant than /events/addTag and /attributes/addTag.

URL Arguments

Response

{
    "name": "Tag tlp3Awhite(7) successfully attached to Attribute(153).",
    "message": "Tag tlp3Awhite(7) successfully attached to Attribute(153).",
    "url": "\/tags\/attachTagToObject"
}

Example

curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" -X POST http://10.50.13.60/tags/attachTagToObject/5a0d68b3-6da0-4ced-8233-77bb950d210f/tlp3Awhite
curl --header "Authorization: YOUR API KEY " -d "{"uuid"="5a0d68b3-6da0-4ced-8233-77bb950d210f" "tag"="tlp:white"}" --header "Accept: application/json" --header "Content-Type: application/json" -X POST http://10.50.13.60/tags/attachTagToObject/
curl \
 -d '{"uuid":"e76949e6-5ccb-4483-bef2-0e4cac73d236","tag":"6"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/tags/attachTagToObject

POST /tags/removeTagFromObject

Description

Removes a tag from an object (attribute or event) with given UUID.

This endpoint exists for convenience reasons and might be slightly less performant than /events/removeTag and /attributes/removeTag.

URL Arguments

Response

{
    "name": "Tag tlp3Awhite(7) successfully removed from Attribute(153).",
    "message": "Tag tlp3Awhite(7) successfully removed from Attribute(153).",
    "url": "\/tags\/removeTagFromObject"
}

Example

curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" -X POST http://10.50.13.60/tags/removeTagFromObject/5a0d68b3-6da0-4ced-8233-77bb950d210f/tlp3Awhite
curl \
 -d '{"uuid":"85752e06-7644-40c8-8190-f8bbe9e7b2c7","tag":"tlp:white"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/tags/removeTagFromObject
curl \
 -d '{"uuid":"85752e06-7644-40c8-8190-f8bbe9e7b2c7","tag":7}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/tags/removeTagFromObject

GET /tags/tagStatistics/

Description

Will give an overview of the used attribute tags

Output

{
    "tags": {
        "Type:OSINT": "1",
        "tlp:white": "1",
        "osint:source-type=\"technical-report\"": "1",
        "misp-galaxy:threat-actor=\"Lazarus Group\"": "1",
        "misp-galaxy:rat=\"FALLCHILL\"": "1"
    },
    "taxonomies": []
}

Example

curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" -X GET http://10.50.13.60/tags/tagStatistics/

Attribute management

POST /attributes/add/

Adds an Attribute to an event

URL Arguments

Output

Example

curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" -d "{"event_id":"3542","value":"1.2.3.4","category":"Network activity","type":"ip-dst"}" http://10.50.13.60/attributes/add/3542

GET /attributes

Get an attribute

URL Arguments

URL Attributes

Output

{"Attribute":{"id":"39","event_id":"1","object_id":"0","object_relation":null,"category":"Payload installation","type":"md5","to_ids":true,"uuid":"548847db-060c-4275-a0c7-15bb950d210b","timestamp":"1418217435","distribution":"3","sharing_group_id":"0","comment":"Regin samples collected.","deleted":false,"disable_correlation":false,"value":"049436bb90f71cf38549817d9b90e2da","event_uuid":"54884656-2da8-4625-bf07-43ef950d210b"}}

Example

curl --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" http://10.50.13.60/attributes/548847db-060c-4275-a0c7-15bb950d210b

POST /attributes/delete/

Description

Delete attributes.

URL Arguments

Output

{"message":"Attribute deleted."}

Example

curl -X POST --header "Authorization: YOUR API KEY" --header "Accept: application/json" --header "Content-Type: application/json" https://<misp url>/attributes/delete/12345
curl -X POST --header "Authorization: YOUR API KEY " --header "Accept: application/json" --header "Content-Type: application/json" http://10.50.13.60/attributes/delete/548847db-060c-4275-a0c7-15bb950d210b

Hard delete:

curl -X POST --header "Authorization: YOUR API KEY" --header "Accept: application/json" --header "Content-Type: application/json" https://<misp url>/attributes/delete/12345/1

POST /attributes/addTag

Add a tag or a tag collection to an existing attribute.

URL Arguments

Examples:

curl \
 -d '{"attribute":"256919","tag":"tlp:green"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/attributes/addTag
curl \
 -d '{"attribute":"256919","tag":"987"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/attributes/addTag
curl \
 -d '{"attribute":"256919","tag":"collection_1"}' \
 -H "Authorization: YOUR API KEY" \
 -H "Accept: application/json" \
 -H "Content-type: application/json" \
 -X POST https://192.168.0.223/attributes/addTag