p0f Fingerprint Database and NAT Detection Explained

p0f identifies operating systems passively, by matching the traffic it observes against its fingerprint database, and the same signals can reveal NAT. This is part 3 of the OS detection techniques guide; part 2 covers Nmap OS detection and TCP/IP fingerprinting.

Os Detection Techniques – P0f Fingerprint database

Whenever p0f obtains a fingerprint from the observed traffic, it defers to the data read from p0f.fp to identify the operating system and obtain some ancillary data needed for other analysis tasks. The fingerprint database is a simple text file where lines starting with ; are ignored.

== Module specification ==

The file is split into sections based on the type of traffic the fingerprints apply to. Section identifiers are enclosed in square brackets, like so:

[module:direction]

module – the name of the fingerprinting module (e.g. ‘tcp’ or ‘http’).

direction – the direction of fingerprinted traffic: ‘request’ (from client to server) or ‘response’ (from server to client).

For the TCP module, ‘client’ matches the initial SYN; and ‘server’ matches SYN+ACK.

The ‘direction’ part is omitted for MTU signatures, as they work equally well both ways.

== Signature groups ==

The actual signatures must be preceeded by an ‘label’ line, describing the fingerprinted software:

label = type:class:name:flavor

type – some signatures in p0f.fp offer broad, last-resort matching for less researched corner cases. The goal there is to give an answer slightly better than “unknown”, but less precise than what the user may be expecting.

Normal, reasonably specific signatures that can’t be radically improved should have their type specified as ‘s’; while generic, last-resort ones should be tagged with ‘g’.

Note that generic signatures are considered only if no specific matches are found in the database.

class – the tool needs to distinguish between OS-identifying signatures (only one of which should be matched for any given host) and signatures that just identify user applications (many of which may be seen concurrently).

To assist with this, OS-specific signatures should specify the OS architecture family here (e.g., ‘win’, ‘unix’, ‘cisco’); while application-related sigs (NMap, MSIE, Apache) should use a special value of ‘!’.

Most TCP signatures are OS-specific, and should have OS family defined. Other signatures, such as HTTP, should use ‘!’ unless the fingerprinted component is deeply intertwined with the platform (e.g., Windows Update).

NOTE: To avoid variations (e.g. ‘win’ and ‘windows’ or ‘unix’ and ‘linux’), all classes need to be pre-registered using a ‘classes’ directive, seen near the beginning of p0f.fp.

name – a human-readable short name for what the fingerprint actually helps identify – say, ‘Linux’, ‘Sendmail’, or ‘NMap’. The tool doesn’t care about the exact value, but requires consistency – so don’t switch between ‘Internet Explorer’ and ‘MSIE’, or ‘MacOS’ and ‘Mac OS’.

flavor – anything you want to say to further qualify the observation. Can be the version of the identified software, or a description of what the application seems to be doing (e.g. ‘SYN scan’ for NMap).

NOTE: Don’t be too specific: if you have a signature for Apache 2.2.16, but have no reason to suspect that other recent versions behave in a radically different way, just say ‘2.x’.

P0f uses labels to group similar signatures that may be plausibly generated by the same system or application, and should not be considered a strong signal for NAT Os Detection Techniques.

To further assist the tool in deciding which OS and application combinations are reasonable, and which ones are indicative of foul play, any ‘label’ line for applications (class ‘!’) should be followed by a comma-delimited list of OS names or @-prefixed OS architecture classes on which this software is known to be used on. For example:

label = s:!:Uncle John’s Networked ls Utility:2.3.0.1 sys = Linux,FreeBSD,OpenBSD

…or:

label = s:!:Mom’s Homestyle Browser:1.x sys = @unix,@win

The label can be followed by any number of module-specific signatures; all of them will be linked to the most recent label, and will be reported the same way.

All sections except for ‘name’ are omitted for [mtu] signatures, which do not convey any OS-specific information, and just describe link types.

== MTU signatures ==

Many operating systems derive the maximum segment size specified in TCP options from the MTU of their network interface; that value, in turn, normally depends on the design of the link-layer protocol. A different MTU is associated with PPPoE, a different one with IPSec, and a different one with Juniper VPN.

The format of the signatures in the [mtu] section is exceedingly simple, consisting just of a description and a list of values:

label = Ethernet sig = 1500

These will be matched for any wildcard MSS TCP packets (see below) not generated by userspace TCP tools.

== TCP Os Detection Techniques signatures ==

For TCP traffic, signature layout is as follows:

sig = ver:ittl:olen:mss:wsize,scale:olayout:quirks:pclass

ver – signature for IPv4 (‘4’), IPv6 (‘6’), or both (‘*’).

NEW SIGNATURES: P0f documents the protocol observed on the wire, but you should replace it with ‘*’ unless you have observed some actual differences between IPv4 and IPv6 traffic, or unless the software supports only one of these versions to begin with.

ittl – initial TTL used by the OS. Almost all operating systems use 64, 128, or 255; ancient versions of Windows sometimes used 32, and several obscure systems sometimes resort to odd values such as 60.

NEW SIGNATURES: P0f will usually suggest something, using the format of ‘observed_ttl+distance’ (e.g. 54+10). Consider using traceroute to check that the distance is accurate, then sum up the values. If initial TTL can’t be guessed, p0f will output ‘nnn+?’, and you need to use traceroute to estimate the ‘?’.

A handful of userspace tools will generate random TTLs. In these cases, determine maximum initial TTL and then add a – suffix to the value to avoid confusion.

olen – length of IPv4 options or IPv6 extension headers. Usually zero for normal IPv4 traffic; always zero for IPv6 due to the limitations of libpcap.

NEW SIGNATURES: Copy p0f output literally.

mss – maximum segment size, if specified in TCP options. Special value of ‘*’ can be used to denote that MSS varies depending on the parameters of sender’s network link, and should not be a part of the signature. In this case, MSS will be used to guess the type of network hookup according to the [mtu] rules.

NEW SIGNATURES: Use ‘*’ for any commodity OSes where MSS is around 1300 – 1500, unless you know for sure that it’s fixed. If the value is outside that range, you can probably copy it literally.

wsize – window size. Can be expressed as a fixed value, but many operating systems set it to a multiple of MSS or MTU, or a multiple of some random integer. P0f automatically detects these cases, and allows notation such as ‘mss*4’, ‘mtu*4’, or ‘%8192’ to be used. Wilcard (‘*’) is possible too.

NEW SIGNATURES: Copy p0f output literally. If frequent variations are seen, look for obvious patterns. If there are no patterns, ‘*’ is a possible alternative.

scale – window scaling factor, if specified in TCP options. Fixed value or ‘*’.

NEW SIGNATURES: Copy literally, unless the value varies randomly. Many systems alter between 2 or 3 scaling factors, in which case, it’s better to have several ‘sig’ lines, rather than a wildcard.

olayout – comma-delimited layout and ordering of TCP options, if any. This is one of the most valuable TCP fingerprinting signals. Supported values:

eol+n – explicit end of options, followed by n bytes of padding nop – no-op option mss – maximum segment size ws – window scaling sok – selective ACK permitted sack – selective ACK (should not be seen) ts – timestamp ?n – unknown option ID n

NEW SIGNATURES: Copy this string literally.

quirks – comma-delimited properties and quirks observed in IP or TCP headers:

df – “don’t fragment” set (probably PMTUD); ignored for IPv6 id+ – DF set but IPID non-zero; ignored for IPv6 id- – DF not set but IPID is zero; ignored for IPv6 ecn – explicit congestion notification support 0+ – “must be zero” field not zero; ignored for IPv6 flow – non-zero IPv6 flow ID; ignored for IPv4

seq- – sequence number is zero ack+ – ACK number is non-zero, but ACK flag not set ack- – ACK number is zero, but ACK flag set uptr+ – URG pointer is non-zero, but URG flag not set urgf+ – URG flag used pushf+ – PUSH flag used

ts1- – own timestamp specified as zero ts2+ – non-zero peer timestamp on initial SYN opt+ – trailing non-zero data in options segment exws – excessive window scaling factor (> 14) bad – malformed TCP options

If a signature scoped to both IPv4 and IPv6 contains quirks valid for just one of these protocols, such quirks will be ignored for on packets using the other protocol. For example, any combination of ‘df’, ‘id+’, and ‘id-‘ is always matched by any IPv6 packet.

NEW SIGNATURES: Copy literally.

pclass – payload size classification: ‘0’ for zero, ‘+’ for non-zero, ‘*’ for any. The packets we fingerprint right now normally have no payloads, but some corner cases exist.

NEW SIGNATURES: Copy literally.

NOTE: The TCP module allows some fuzziness when an exact match can’t be found: ‘df’ and ‘id+’ quirks are allowed to disappear; ‘id-‘ or ‘ecn’ may appear; and TTLs can change.

To gather new SYN (‘request’) signatures, simply connect to the fingerprinted system, and p0f will provide you with the necessary data. To gather SYN+ACK (‘response’) signatures, you should use the bundled p0f-sendsyn utility while p0f is running in the background; creating them manually is not advisable.

== HTTP signatures ==

A special directive should appear at the beginning of the [https:request] section, structured the following way:

ua_os = Linux,Windows,iOS=[iPad],iOS=[iPhone],Mac OS X,…

This list should specify OS names that should be looked for within the User-Agent string if the string is otherwise deemed to be honest. This input is not used for fingerprinting, but aids NAT detection in some useful ways.

The names have to match the names used in ‘sig’ specifiers across p0f.fp. If a particular name used by p0f differs from what typically appears in User-Agent, the name=[string] syntax may be used to define any number of aliases.

Other than that, HTTP signatures for GET and HEAD requests have the following layout:

sig = ver:horder:habsent:expsw

ver – 0 for HTTP/1.0, 1 for HTTP/1.1, or ‘*’ for any.

NEW SIGNATURES: Copy the value literally, unless you have a specific reason to do otherwise.

horder – comma-separated, ordered list of headers that should appear in matching traffic. Substrings to match within each of these headers may be specified using a name=[value] notation.

The signature will be matched even if other headers appear in between, as long as the list itself is matched in the specified sequence.

Headers that usually do appear in the traffic, but may go away (e.g. Accept-Language if the user has no languages defined, or Referer if no referring site exists) should be prefixed with ‘?’, e.g. “?Referer”. P0f will accept their disappearance, but will not allow them to appear at any other location.

NEW SIGNATURES: Review the list and remove any headers that appear to be irrelevant to the fingerprinted software, and mark transient ones with ‘?’. Remove header values that do not add anything to the signature, or are request- or user-specific. In particular, pay attention to Accept, Accept-Language, and Accept-Charset, as they are highly specific to request type and user settings.

P0f automatically removes some headers, prefixes others with ‘?’, and inhibits the value of fields such as ‘Referer’ or ‘Cookie’ – but this is not a substitute for manual review.

NOTE: Server signatures may differ depending on the request (HTTP/1.1 versus 1.0, keep-alive versus one-shot, etc) and on the returned resource (e.g., CGI versus static content). Play around, browse to several URLs, also try curl and wget.

habsent – comma-separated list of headers that must *not* appear in matching traffic. This is particularly useful for noting the absence of standard headers (e.g. ‘Host’), or for differentiating between otherwise very similar signatures.

NEW SIGNATURES: P0f will automatically highlight the absence of any normally present headers; other entries may be added where necessary.

expsw – expected substring in ‘User-Agent’ or ‘Server’. This is not used to match traffic, and merely serves to detect dishonest software. If you want to explicitly match User-Agent, you need to do this in the ‘horder’ section, e.g.:

User-Agent=[Firefox]

Any of these sections sections except for ‘ver’ may be blank.

There are many protocol-level quirks that p0f could be detecting – for example, the use of non-standard newlines, or missing or extra spacing between header field names and values. There is also some information to be gathered from responses to OPTIONS or POST. That said, it does not seem to be worth the effort: the protocol is so verbose, and implemented so arbitrarily, that we are getting more than enough information just with a simple GET / HEAD fingerprint.

== SMTP signatures ==

*** NOT IMPLEMENTED YET ***

== FTP signatures ==

*** NOT IMPLEMENTED YET ***

6. Os Detection Techniques – NAT detection

In addition to fairly straightforward measurements of intrinsic properties of a single TCP session, p0f also tries to compare signatures across sessions to detect client-side connection sharing (NAT, HTTP proxies) or server-side load balancing.

This is done in two steps: the first significant deviation usually prompts a “host change” entry (which may be also indicative of multi-boot, address reuse, or other one-off events); and a persistent pattern of changes prompts an “ip sharing” notification later on.

All of these messages are accompanied by a set of reason codes:

os_sig – the OS detected right now doesn’t match the OS detected earlier on.

sig_diff – no definite OS detection data available, but protocol-level characteristics have changed drastically (e.g., different TCP option layout).

app_vs_os – the application detected running on the host is not supposed to work on the host’s operating system.

x_known – the signature progressed from known to unknown, or vice versa.

The following additional codes are specific to TCP:

tstamp – TCP timestamps went back or jumped forward.

ttl – TTL values have changed.

port – source port number has decreased.

mtu – system MTU has changed.

fuzzy – the precision with which a TCP signature is matched has changed.

The following code is also issued by the HTTP module:

via – data explicitly includes Via / X-Forwarded-For.

us_vs_os – OS fingerprint doesn’t match User-Agent data, and the User-Agent value otherwise looks honest.

app_srv_lb – server application signatures change, suggesting load balancing.

date – server-advertised date changes inconsistently.

Different reasons have different weights, balanced to keep p0f very sensitive even to very homogenous environments behind NAT. If you end up seeing false positives or other detection problems in your environment, please let me know!

sources:

p0f man page nmap man page RFC 791 (IP) https://tools.ietf.org/html/rfc791 RFC 793 (TCP) https://tools.ietf.org/html/rfc793 RFC 826 (ARP) https://tools.ietf.org/html/rfc826

https://en.wikipedia.org/wiki/MAC_address Ali, S., Heriyanto, T. and Allen, L. (2014) Kali Linux: assuring security by penetration testing. Birmingham: Packt Publ. Allen, J. M. (2019) SANS Institute pp.1–49. Anderson, B. and McGrew, D. (2017) OS Fingerprinting: New Techniques and a Study of Information Gain and Obfuscation. arXiv.org. Burroni, J. and Sarraute, C. (2010) Outrepasser les limites des techniques classiques de Prise d’Empreintes grace aux Reseaux de Neurones. arXiv.org. Fyodor (2013) Nmap Network Scanning Official Nmap Project Guide to Network Discovery and Security Scanning pp.1–465. Fyodor (2019a) nmap fingerprinting format [Online]. Available: https://nmap.org/book/osdetect-fingerprint-format.html [Accessed 2 Jun 2019a]. Fyodor (2019b) nmap os detection methods [Online]. Available: https://nmap.org/book/osdetect-methods.html [Accessed 2 Jun 2019b]. Kollmann, E. (2005) Chatter on the Wire: pp.1–46. Kollmann, E. (2019) bh-japan-laporte-kollmann-v8. In:. Tao, K., Li, J. and Sampalli, S. (2009) Detection of Spoofed MAC Addresses in 802.11 Wireless Networks. E-business and Telecommunications. Vol.2 (Chapter 15), pp.201–.

https://www.netresec.com/?page=Blog&month=2011-11&post=Passive-OS-Fingerprinting https://en.wikipedia.org/wiki/TCP/IP_stack_fingerprinting https://phrack.org/issues/54/9.html#article https://www.ietf.org/rfc/rfc2132.txt https://www.juniper.net/documentation/en_US/junose15.1/topics/concept/dhcp-relay-option-60-strings.html

https://subinsb.com/default-device-ttl-values/ https://securitytrails.com/blog/cybersecurity-fingerprinting https://forensicswiki.org/wiki/OS_fingerprinting https://en.wikipedia.org/wiki/Zero-configuration_networking#Service_discovery https://en.wikipedia.org/wiki/Service_discovery https://en.wikipedia.org/wiki/Server_Message_Block https://en.wikipedia.org/wiki/List_of_products_that_support_SMB https://en.wikipedia.org/wiki/NetBIOS https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol https://en.wikipedia.org/wiki/User_agent