Add comprehensive DNS record enumeration and pivot enrichment

README.md

@@ -81,6 +81,7 @@ For further Information see the [license file](https://misp.github.io/misp-modul
 * [CVE Advanced Lookup](https://misp.github.io/misp-modules/expansion/#cve-advanced-lookup) - An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE).
 * [Cytomic Orion Lookup](https://misp.github.io/misp-modules/expansion/#cytomic-orion-lookup) - An expansion module to enrich attributes in MISP by quering the Cytomic Orion API
 * [DBL Spamhaus Lookup](https://misp.github.io/misp-modules/expansion/#dbl-spamhaus-lookup) - Checks Spamhaus DBL for a domain name.
+* [DNS Record Enumeration](https://misp.github.io/misp-modules/expansion/#dns-record-enumeration) - Query DNS data types including SOA and TXT, preserve complete resource records, and extract linked domain, IP, email and URL pivots. [JSON documentation](documentation/expansion/dns_query.json).
 * [DNS Resolver](https://misp.github.io/misp-modules/expansion/#dns-resolver) - Simple DNS expansion service to resolve IP address from MISP attributes
 * [DOCX Enrich](https://misp.github.io/misp-modules/expansion/#docx-enrich) - Module to extract freetext from a .docx document.
 * [DomainTools Lookup](https://misp.github.io/misp-modules/expansion/#domaintools-lookup) - DomainTools MISP expansion module.

documentation/expansion/dns_query.json

@@ -0,0 +1,119 @@
+{
+  "name": "DNS Record Enumeration",
+  "module": "dns_query",
+  "module-type": [
+    "expansion"
+  ],
+  "description": "Actively query DNS data types and collect complete resource records and typed correlation pivots in MISP.",
+  "requirements": [
+    "dnspython",
+    "PyMISP",
+    "DNS access to a recursive resolver"
+  ],
+  "features": [
+    "Query every data type known to the installed dnspython version explicitly; dnspython 2.8.0 enumerates 74 types. Common pivot-rich types are scheduled first. Results depend on resolver support and the time budget.",
+    "Include SOA, TXT, SPF, MX, NS, CNAME, DNAME, PTR, SRV, NAPTR, CAA, HTTPS/SVCB, DNSSEC and legacy data types. record_types can select a comma-separated subset or private types such as TYPE65400.",
+    "Use absolute DNS names with resolver search disabled. Normalize IDNA and case. Domains, hostnames, domain|ip, hostname|port, URLs, email/email-src/email-dst/dns-soa-email and ip-src/ip-dst (including |port) are supported. IPs and URL IP literals use the reverse DNS name.",
+    "Retain each resource record's complete presentation-format RDATA, actual owner, class and first observed TTL in a dns-resource-record object. Object first_seen and last_seen mark this run's observation time.",
+    "Keep records from answer, authority and additional sections, including SOA and glue in negative replies. queried-name and section preserve context; an authority SOA describes its own zone, not a positive SOA answer at the input name.",
+    "Deduplicate by owner, class, type and RDATA across queries; preserve all query names and response sections on the object. Aliases and canonical answers retain separate owner names.",
+    "Extract domain targets, IPs, MX/SRV/URI priorities, SRV weights and ports, SOA timings and responsible mailbox, RP mailbox, HTTPS/SVCB address hints and ports, and URLs from URI/CAA/TXT data. All other data remains available in rdata.",
+    "Concatenate TXT/SPF chunks for reading while retaining quoting and chunk boundaries in rdata. Parse explicit SPF include/redirect/a/mx/exists domains and IPv4/IPv6 networks. Extract email addresses and HTTP(S) reporting URLs; no web request is made.",
+    "Query TXT at _dmarc, _mta-sts and _smtp._tls under the input name by default. One bounded follow-up stage queries A and AAAA for up to 16 discovered domain targets. Follow-ups share the original time budget and are not recursively expanded.",
+    "Eight worker threads share immutable resolver configuration and a total DNS time budget. Preserve successful results when other queries time out, are refused, return NXDOMAIN, or cannot start within the budget. Report limitations in the first object's comment. No returned records produces an error.",
+    "Set EDNS DNSSEC OK to request available DNSSEC data. This module does not independently validate DNSSEC signatures.",
+    "Return the original attribute with its UUID and metadata; every record object is derived-from that UUID. All generated attributes use to_ids=false. The bundled template provides strict validation even before a PyMISP template update."
+  ],
+  "input": {
+    "format": "misp_standard",
+    "types": [
+      "domain",
+      "hostname",
+      "domain|ip",
+      "hostname|port",
+      "url",
+      "email",
+      "email-src",
+      "email-dst",
+      "dns-soa-email",
+      "ip-src",
+      "ip-dst",
+      "ip-src|port",
+      "ip-dst|port"
+    ],
+    "required_attribute_fields": [
+      "type",
+      "value",
+      "uuid"
+    ]
+  },
+  "output": {
+    "format": "misp_standard",
+    "objects": [
+      "dns-resource-record"
+    ],
+    "max_records": 1000,
+    "errors": "Malformed requests, unsupported attributes, invalid values/configuration or no resource records return an error. Individual DNS failures preserve successful records with diagnostic comments."
+  },
+  "config": {
+    "nameserver": {
+      "default": "System resolver",
+      "description": "Optional recursive DNS server IP address (IPv4 or IPv6)."
+    },
+    "record_types": {
+      "default": "all",
+      "description": "All dnspython-known data types or a comma-separated list such as A,AAAA,SOA,TXT,MX,NS,CAA,HTTPS. Up to 128 explicit types. Excludes ANY, AXFR, IXFR, OPT, TKEY, TSIG, MAILA, MAILB and reserved values. A/AAAA follow-ups and optional mail-policy TXT queries are controlled separately."
+    },
+    "query_timeout": {
+      "default": 2,
+      "range": "0.1–10 seconds",
+      "description": "Per-query lifetime, including resolver retries, capped by the remaining total budget."
+    },
+    "max_time": {
+      "default": 30,
+      "range": "1–120 seconds",
+      "description": "Total budget shared by all initial DNS queries and address follow-ups. Serialization adds some processing time."
+    },
+    "max_pivots": {
+      "default": 16,
+      "range": "0–64 names",
+      "description": "Maximum discovered domain names to resolve in a single A/AAAA follow-up stage. Set 0 to disable."
+    },
+    "mail_policy": {
+      "default": true,
+      "description": "Boolean or true/false/1/0 string. Query _dmarc, _mta-sts and _smtp._tls TXT names for non-IP input. This uses the exact input domain, without guessing an organizational domain."
+    }
+  },
+  "limitations": [
+    "This enumerates record types at known names, not all records in a zone. DNS has no general query that reveals every owner name. It does not guess subdomains, DKIM selectors or SRV service labels; supply those names explicitly.",
+    "It does not send ANY, perform zone transfers, scan all 65,536 numeric type codes, walk NSEC chains, or recursively expand newly found names. Private numeric types can be selected explicitly.",
+    "A recursive resolver may filter types, return cached answers or omit records. No claim of complete zone coverage or DNSSEC authentication is made. A TTL is the resolver-returned remaining TTL.",
+    "Some obsolete or specialized types have no dnspython-specific decoder; full generic RDATA is still retained, while extracted pivot fields depend on the decoder.",
+    "Queries go to the configured recursive resolver. Eight workers and bounded time, result count and follow-up count limit each expansion's workload."
+  ],
+  "installation": [
+    "Install the module through the normal misp-modules package deployment; dnspython is already in the minimal/all dependency extras and PyMISP is a core dependency.",
+    "Restart misp-modules and enable dns_query in MISP Enrichment plugin settings. No API key is required.",
+    "Import/update the dns-resource-record template in MISP from MISP/misp-objects. Until available upstream, the identical definition is bundled at misp_modules/data/dns-resource-record/definition.json; import it into the MISP object-template store with your normal administrator workflow.",
+    "Expand an existing supported attribute using its original UUID. Both input and results use misp_standard format."
+  ],
+  "verification": {
+    "command": "python -m pytest -q -o log_cli=false tests/test_dns_query.py tests/test_dns.py tests/test_url_dns.py",
+    "coverage": "Record types, SOA/TXT and policy pivots, unknown types, aliases and deduplication, negative authority/glue records, supported inputs, configuration, partial failures, budgets, limits, metadata and strict object validation. A local UDP DNS server exercises real dnspython wire parsing and EDNS.",
+    "limitations": "Public DNS and deployment into a running MISP instance have not been tested."
+  },
+  "example": {
+    "module": "dns_query",
+    "attribute": {
+      "type": "domain",
+      "value": "example.org",
+      "uuid": "5b582d80-7a7e-4b6a-9f22-77656e72bb3b"
+    },
+    "config": {
+      "record_types": "all",
+      "max_time": 30,
+      "max_pivots": 16,
+      "mail_policy": true
+    }
+  }
+}

misp_modules/data/dns-resource-record/definition.json

@@ -0,0 +1,166 @@
+{
+  "attributes": {
+    "domain": {
+      "description": "Domain name extracted from the resource record data.",
+      "misp-attribute": "domain",
+      "multiple": true,
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "email": {
+      "description": "Email address extracted from resource record data or mail policy text.",
+      "misp-attribute": "email",
+      "multiple": true,
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "expire": {
+      "description": "SOA expiry interval in seconds.",
+      "disable_correlation": true,
+      "misp-attribute": "counter",
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "ip": {
+      "description": "IP address or network extracted from address records, SPF or service binding hints.",
+      "misp-attribute": "ip-dst",
+      "multiple": true,
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "minimum": {
+      "description": "SOA minimum negative-cache TTL in seconds.",
+      "disable_correlation": true,
+      "misp-attribute": "counter",
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "port": {
+      "description": "Service port extracted from SRV or service binding data.",
+      "disable_correlation": true,
+      "misp-attribute": "port",
+      "multiple": true,
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "priority": {
+      "description": "MX preference, SRV/URI priority, or service binding priority.",
+      "disable_correlation": true,
+      "misp-attribute": "counter",
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "queried-name": {
+      "description": "Absolute DNS query name, without the final root dot. May differ from rrname when aliases are followed.",
+      "misp-attribute": "domain",
+      "multiple": true,
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "rdata": {
+      "description": "Complete resource record data in DNS presentation format; preserves TXT quoting, SOA fields and unknown types.",
+      "misp-attribute": "text",
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "refresh": {
+      "description": "SOA secondary refresh interval in seconds.",
+      "disable_correlation": true,
+      "misp-attribute": "counter",
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "retry": {
+      "description": "SOA retry interval in seconds.",
+      "disable_correlation": true,
+      "misp-attribute": "counter",
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "rrclass": {
+      "description": "DNS resource record class, normally IN.",
+      "disable_correlation": true,
+      "misp-attribute": "text",
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "rrname": {
+      "description": "Owner name in DNS presentation form, without the final root dot except for the DNS root itself (.). Text permits root, wildcard and escaped owner names.",
+      "misp-attribute": "text",
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "rrtype": {
+      "description": "DNS resource record type mnemonic, or TYPE followed by its numeric code.",
+      "disable_correlation": true,
+      "misp-attribute": "text",
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "section": {
+      "description": "DNS response section where this record was observed. Authority and additional records describe their own rrname, not necessarily the queried name.",
+      "disable_correlation": true,
+      "misp-attribute": "text",
+      "multiple": true,
+      "to_ids": false,
+      "ui-priority": 0,
+      "values_list": [
+        "answer",
+        "authority",
+        "additional"
+      ]
+    },
+    "serial": {
+      "description": "SOA zone serial number.",
+      "disable_correlation": true,
+      "misp-attribute": "counter",
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "soa-email": {
+      "description": "SOA responsible mailbox converted from DNS RNAME notation to an email address.",
+      "misp-attribute": "dns-soa-email",
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "text": {
+      "description": "Decoded TXT/SPF character strings concatenated in order; rdata preserves original chunk boundaries.",
+      "misp-attribute": "text",
+      "multiple": true,
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "ttl": {
+      "description": "Remaining time to live in seconds as returned by the resolver.",
+      "disable_correlation": true,
+      "misp-attribute": "counter",
+      "to_ids": false,
+      "ui-priority": 0
+    },
+    "url": {
+      "description": "URL extracted from resource record data; no HTTP request is implied.",
+      "misp-attribute": "url",
+      "multiple": true,
+      "to_ids": false,
+      "ui-priority": 1
+    },
+    "weight": {
+      "description": "SRV or URI relative weight.",
+      "disable_correlation": true,
+      "misp-attribute": "counter",
+      "to_ids": false,
+      "ui-priority": 0
+    }
+  },
+  "description": "A single DNS resource record observed through active resolution, preserving its query name, owner, type, class, TTL and complete RDATA alongside extracted correlation pivots. Object first_seen and last_seen record observation time, not zone history.",
+  "meta-category": "network",
+  "name": "dns-resource-record",
+  "required": [
+    "queried-name",
+    "rrname",
+    "rrtype",
+    "rdata"
+  ],
+  "uuid": "c6138676-09a5-4c3d-b73e-654df15e3242",
+  "version": 1
+}