Nmap OS Detection: How TCP/IP Fingerprinting Works

Nmap OS detection works by sending a series of specially crafted TCP, UDP and ICMP probes and comparing the replies against a database of known operating systems. This is part 2 of the OS detection techniques guide; part 3 covers the p0f fingerprint database and NAT detection.

TCP/IP Fingerprinting Methods Supported by Nmap – Os Detection Techniques

Nmap Os Detection Techniques employ fingerprinting which works by sending up to 16 TCP, UDP, and ICMP probes to known open and closed ports of the target machine. These probes are specially designed to exploit various ambiguities in the standard protocol RFCs. Then Nmap listens for responses. Dozens of attributes in those responses are analyzed and combined to generate a fingerprint. Every probe packet is tracked and resent at least once if there is no response. All of the packets are IPv4 with a random IP ID value. Probes to an open TCP port are skipped if no such port has been found. For closed TCP or UDP ports, Nmap will first check if such a port has been found. If not, Nmap will just pick a port at random and hope for the best.

closed ports of the target machine. These probes are specially designed to exploit various ambiguities in the standard protocol RFCs. Then Nmap listens for responses. Dozens of attributes in those responses are analyzed and combined to generate a fingerprint. Every probe packet is tracked and resent at least once if there is no response. All of the packets are IPv4 with a random IP ID value. Probes to an open TCP port are skipped if no such port has been found. For closed TCP or UDP ports, Nmap will first check if such a port has been found. If not, Nmap will just pick a port at random and hope for the best.

The following sections are highly technical and reveal the hidden workings of Nmap Os Detection Techniques. Nmap can be used effectively without understanding this, though the material can help you better understand remote networks and also detect and explain certain anomalies. Plus, some of the Os Detection Techniques are pretty cool. Readers in a hurry may skip to the section called “Dealing with Misidentified and Unidentified Hosts”. But for those of you who are ready for a journey through TCP explicit congestion notification, reserved UDP header bits, initial sequence numbers, bogus flags, and Christmas tree packets: read on!

Even the best of us occasionally forget byte offsets for packet header fields and flags. For quick reference, the IPv4, TCP, UDP, and ICMP header layouts can be found in the section called “TCP/IP Reference”. The layout for ICMP echo request and destination unreachable packets are shown in Figure 8.1 and Figure 8.2.

**Figure 8.1. ICMP echo request or reply header layout

ICMP echo request or reply header layout

**Figure 8.2. ICMP destination unreachable header layout

ICMP destination unreachable header layout

Probes Sent This section describes each IP probe sent by Nmap as part of TCP/IP fingerprinting. It refers to Nmap response tests and TCP options which are explained in the following section.

Sequence generation (SEQ, OPS, WIN, and T1) A series of six TCP probes is sent to generate these four test response lines. The probes are sent exactly 100 milliseconds apart so the total time taken is 500 ms. Exact timing is important as some of the sequence algorithms we detect (initial sequence numbers, IP IDs, and TCP timestamps) are time dependent. This timing value was chosen to take 500 ms so that we can reliably detect the common 2 Hz TCP timestamp sequences.

Each probe is a TCP SYN packet to a detected open port on the remote machine. The sequence and acknowledgment numbers are random (but saved so Nmap can differentiate responses). Detection accuracy requires probe consistency, so there is no data payload even if the user requested one with –data-length.

These packets vary in the TCP options they use and the TCP window field value. The following list provides the options and values for all six packets. The listed window field values do not reflect window scaling. EOL is the end-of-options-list option, which many sniffing tools don’t show by default.

Packet #1: window scale (10), NOP, MSS (1460), timestamp (TSval: 0xFFFFFFFF; TSecr: 0), SACK permitted. The window field is 1.

Packet #2: MSS (1400), window scale (0), SACK permitted, timestamp (TSval: 0xFFFFFFFF; TSecr: 0), EOL. The window field is 63.

Packet #3: Timestamp (TSval: 0xFFFFFFFF; TSecr: 0), NOP, NOP, window scale (5), NOP, MSS (640). The window field is 4.

Packet #4: SACK permitted, Timestamp (TSval: 0xFFFFFFFF; TSecr: 0), window scale (10), EOL. The window field is 4.

Packet #5: MSS (536), SACK permitted, Timestamp (TSval: 0xFFFFFFFF; TSecr: 0), window scale (10), EOL. The window field is 16.

Packet #6: MSS (265), SACK permitted, Timestamp (TSval: 0xFFFFFFFF; TSecr: 0). The window field is 512.

The results of these tests include four result category lines. The first, SEQ, contains results based on sequence analysis of the probe packets. These test results are GCD, SP, ISR, TI, II, TS, and SS. The next line, OPS contains the TCP options received for each of the probes (the test names are O1 through 06). Similarly, the WIN line contains window sizes for the probe responses (named W1 through W6). The final line related to these probes, T1, contains various test values for packet #1. Those results are for the R, DF, T, TG, W, S, A, F, O, RD, and Q tests. These tests are only reported for the first probe since they are almost always the same for each probe.

ICMP echo (IE) The IE test involves sending two ICMP echo request packets to the target. The first one has the IP DF bit set, a type-of-service (TOS) byte value of zero, a code of nine (even though it should be zero), the sequence number 295, a random IP ID and ICMP request identifier, and 120 bytes of 0x00 for the data payload.

The second ping query is similar, except a TOS of four (IP_TOS_RELIABILITY) is used, the code is zero, 150 bytes of data is sent, and the ICMP request ID and sequence numbers are incremented by one from the previous query values.

The results of both of these probes are combined into a IE line containing the R, DFI, T, TG, and CD tests. The R value is only true (Y) if both probes elicit responses. The T, and CD values are for the response to the first probe only, since they are highly unlikely to differ. DFI is a custom test for this special dual-probe ICMP case.

These ICMP probes follow immediately after the TCP sequence probes to ensure valid results of the shared IP ID sequence number test (see the section called “Shared IP ID sequence Boolean (SS)”).

TCP explicit congestion notification (ECN) This probe tests for explicit congestion notification (ECN) support in the target TCP stack. ECN is a method for improving Internet performance by allowing routers to signal congestion problems before they start having to drop packets. It is documented in RFC 3168. Nmap tests this by sending a SYN packet which also has the ECN CWR and ECE congestion control flags set. For an unrelated (to ECN) test, the urgent field value of 0xF7F5 is used even though the urgent flag is not set. The acknowledgment number is zero, sequence number is random, window size field is three, and the reserved bit which immediately precedes the CWR bit is set. TCP options are WScale (10), NOP, MSS (1460), SACK permitted, NOP, NOP. The probe is sent to an open port.

If a response is received, the R, DF, T, TG, W, O, CC, and Q tests are performed and recorded.

TCP (T2–T7) The six T2 through T7 tests each send one TCP probe packet. With one exception, the TCP options data in each case is (in hex) 03030A0102040109080AFFFFFFFF000000000402. Those 20 bytes correspond to window scale (10), NOP, MSS (265), Timestamp (TSval: 0xFFFFFFFF; TSecr: 0), then SACK permitted. The exception is that T7 uses a Window scale value of 15 rather than 10. The variable characteristics of each probe are described below:

T2 sends a TCP null (no flags set) packet with the IP DF bit set and a window field of 128 to an open port.

T3 sends a TCP packet with the SYN, FIN, URG, and PSH flags set and a window field of 256 to an open port. The IP DF bit is not set.

T4 sends a TCP ACK packet with IP DF and a window field of 1024 to an open port.

T5 sends a TCP SYN packet without IP DF and a window field of 31337 to a closed port.

T6 sends a TCP ACK packet with IP DF and a window field of 32768 to a closed port.

T7 sends a TCP packet with the FIN, PSH, and URG flags set and a window field of 65535 to a closed port. The IP DF bit is not set.

In each of these cases, a line is added to the fingerprint with results for the R, DF, T, TG, W, S, A, F, O, RD, and Q tests.

UDP (U1) This probe is a UDP packet sent to a closed port. The character ‘C’ (0x43) is repeated 300 times for the data field. The IP ID value is set to 0x1042 for operating systems which allow us to set this. If the port is truly closed and there is no firewall in place, Nmap expects to receive an ICMP port unreachable message in return. That response is then subjected to the R, DF, T, TG, IPL, UN, RIPL, RID, RIPCK, RUCK, and RUD tests.

Response Tests The previous section describes probes sent by Nmap, and this one completes the puzzle by describing the barrage of tests performed on responses. The short names (such as DF, R, and RIPCK) are those used in the nmap-os-db fingerprint database to save space. All numerical test values are given in hexadecimal notation, without leading zeros, unless noted otherwise. The tests are documented in roughly the order they appear in fingerprints.

TCP ISN greatest common divisor (GCD) The SEQ test sends six TCP SYN packets to an open port of the target machine and collects SYN/ ACK packets back. Each of these SYN/ACK packets contains a 32-bit initial sequence number ( ISN). This test attempts to determine the smallest number by which the target host increments these values. For example, many hosts (especially old ones) always increment the ISN in multiples of 64,000.

The first step in calculating this is creating an array of differences between probe responses. The first element is the difference between the 1st and 2nd probe response ISNs. The second element is the difference between the 2nd and 3rd responses. There are five elements if Nmap receives responses to all six probes. Since the next couple of sections reference this array, we will call it diff1. If an ISN is lower than the previous one, Nmap looks at both the number of values it would have to subtract from the first value to obtain the second, and the number of values it would have to count up (including wrapping the 32-bit counter back to zero). The smaller of those two values is stored in diff1. So the difference between 0x20000 followed by 0x15000 is 0xB000. The difference between 0xFFFFFF00 and 0xC000 is 0xC0FF. This test value then records the greatest common divisor of all those elements. This GCD is also used for calculating the SP result.

TCP ISN counter rate (ISR) This value reports the average rate of increase for the returned TCP initial sequence number. Recall that a difference is taken between each two consecutive probe responses and stored in the previously discussed diff1 array. Those differences are each divided by the amount of time elapsed (in seconds-will generally be about 0.1) between sending the two probes which generated them. The result is an array, which we’ll call seq_rates containing the rates of ISN counter increases per second. The array has one element for each diff1 value. An average is taken of the array values. If that average is less than one (e.g. a constant ISN is used), ISR is zero. Otherwise ISR is eight times the binary logarithm (log base-2) of that average value, rounded to the nearest integer.

TCP ISN sequence predictability index (SP) While the ISR test measures the average rate of initial sequence number increments, this value measures the ISN variability. It roughly estimates how difficult it would be to predict the next ISN from the known sequence of six probe responses. The calculation uses the difference array (seq_rates) and GCD values discussed in the previous section.

This test is only performed if at least four responses were seen. If the previously computed GCD value is greater than nine, the elements of the previously computed seq_rates array are divided by that value. We don’t do the division for smaller GCD values because those are usually caused by chance. A standard deviation of the array of the resultant values is then taken. If the result is one or less, SP is zero. Otherwise the binary logarithm of the result is computed, then it is multiplied by eight, rounded to the nearest integer, and stored as SP.

Please keep in mind that this test is only done for Os Detection Techniques and is not a full-blown audit of the target ISN generator. There are many algorithm weaknesses that lead to easy predictability even with a high SP value.

IP ID sequence generation algorithm (TI, CI, II) There are three tests that examine the IP header ID field of responses. TI is based on responses to the TCP SEQ probes. CI is from the responses to the three TCP probes sent to a closed port: T5, T6, and T7. II comes from the ICMP responses to the two IE ping probes. For TI, at least three responses must be received for the test to be included; for CI, at least two responses are required; and for II, both ICMP responses must be received.

For each of these tests, the target’s IP ID generation algorithm is classified based on the algorithm below. Minor differences between tests are noted. Note that difference values assume that the counter can wrap. So the difference between an IP ID of 65,100 followed by a value of 700 is 1,136. The difference between 2,000 followed by 1,100 is 64,636. Here are the calculation details:

If all of the ID numbers are zero, the value of the test is Z.

If the IP ID sequence ever increases by at least 20,000, the value is RD (random). This result isn’t possible for II because there are not enough samples to support it.

If all of the IP IDs are identical, the test is set to that value in hex.

If any of the differences between two consecutive IDs exceeds 1,000, and is not evenly divisible by 256, the test’s value is RI (random positive increments). If the difference is evenly divisible by 256, it must be at least 256,000 to cause this RI result.

If all of the differences are divisible by 256 and no greater than 5,120, the test is set to BI (broken increment). This happens on systems like Microsoft Windows where the IP ID is sent in host byte order rather than network byte order. It works fine and isn’t any sort of RFC violation, though it does give away host architecture details which can be useful to attackers.

If all of the differences are less than ten, the value is I (incremental). We allow difference up to ten here (rather than requiring sequential ordering) because traffic from other hosts can cause sequence gaps.

If none of the previous steps identify the generation algorithm, the test is omitted from the fingerprint.

Shared IP ID sequence Boolean (SS) This Boolean value records whether the target shares its IP ID sequence between the TCP and ICMP protocols. If our six TCP IP ID values are 117, 118, 119, 120, 121, and 122, then our ICMP results are 123 and 124, it is clear that not only are both sequences incremental, but they are both part of the same sequence. If, on the other hand, the TCP IP ID values are 117–122 but the ICMP values are 32,917 and 32,918, two different sequences are being used.

This test is only included if II is RI, BI, or I and TI is the same. If SS is included, the result is S if the sequence is shared and O (other) if it is not. That determination is made by the following algorithm:

Let avg be the final TCP sequence response IP ID minus the first TCP sequence response IP ID, divided by the difference in probe numbers. If probe #1 returns an IP ID of 10,000 and probe #6 returns 20,000, avg would be (20,000 − 10,000) / (6 − 1), which equals 2,000.

If the first ICMP echo response IP ID is less than the final TCP sequence response IP ID plus three times avg, the SS result is S. Otherwise it is O.

TCP timestamp option algorithm (TS) TS is another test which attempts to determine target OS characteristics based on how it generates a series of numbers. This one looks at the TCP timestamp option (if any) in responses to the SEQ probes. It examines the TSval (first four bytes of the option) rather than the echoed TSecr (last four bytes) value. It takes the difference between each consecutive TSval and divides that by the amount of time elapsed between Nmap sending the two probes which generated those responses. The resultant value gives a rate of timestamp increments per second. Nmap computes the average increments per second over all consecutive probes and then calculates the TS as follows:

If any of the responses have no timestamp option, TS is set to U (unsupported).

If any of the timestamp values are zero, TS is set to 0.

If the average increments per second falls within the ranges 0-5.66, 70-150, or 150-350, TS is set to 1, 7, or 8, respectively. These three ranges get special treatment because they correspond to the 2 Hz, 100 Hz, and 200 Hz frequencies used by many hosts.

In all other cases, Nmap records the binary logarithm of the average increments per second, rounded to the nearest integer. Since most hosts use 1,000 Hz frequencies, A is a common result.

TCP options (O, O1–O6) This test records the TCP header options in a packet. It preserves the original ordering and also provides some information about option values. Because RFC 793 doesn’t require any particular ordering, implementations often come up with unique orderings. Some platforms don’t implement all options (they are, of course, optional). When you combine all of those permutations with the number of different option values that implementations use, this test provides a veritable trove of information. The value for this test is a string of characters representing the options being used. Several options take arguments that come immediately after the character. Supported options and arguments are all shown in Table 8.1.

Table 8.1. O test values

Option Name Character Argument (if any) End of Options List (EOL) L No operation (NOP) N Maximum Segment Size (MSS) M The value is appended. Many systems echo the value used in the corresponding probe. Window Scale (WS) W The actual value is appended. Timestamp (TS) T The T is followed by two binary characters representing the TSval and TSecr values respectively. The characters are 0 if the field is zero and 1 otherwise. Selective ACK permitted (SACK) S

As an example, the string M5B4NW3NNT11 means the packet includes the MSS option (value 0x5B4) followed by a NOP. Next comes a window scale option with a value of three, then two more NOPs. The final option is a timestamp, and neither of its two fields were zero. If there are no TCP options in a response, the test will exist but the value string will be empty. If no probe was returned, the test is omitted.

While this test is generally named O, the six probes sent for sequence generation purposes are a special case. Those are inserted into the special OPS test line and take the names O1 through O6 to distinguish which probe packet they relate to. The “O” stands for “options”. Despite the different names, each test O1 through O6 is processed exactly the same way as the other O tests.

TCP initial window size (W, W1–W6) This test simply records the 16-bit TCP window size of the received packet. It is quite effective, since there are more than 80 values that at least one OS is known to send. A down side is that some operating systems have more than a dozen possible values by themselves. This leads to false negative results until we collect all of the possible window sizes used by an operating system.

While this test is generally named W, the six probes sent for sequence generation purposes are a special case. Those are inserted into a special WIN test line and take the names W1 through W6. The window size is recorded for all of the sequence number probes because they differ in TCP MSS option values, which causes some operating systems to advertise a different window size. Despite the different names, each test is processed exactly the same way.

Responsiveness (R) This test simply records whether the target responded to a given probe. Possible values are Y and N. If there is no reply, remaining fields for the test are omitted.

A risk with this test involves probes that are dropped by a firewall. This leads to R=N in the subject fingerprint. Yet the reference fingerprint in nmap-os-db may have R=Y if the target OS usually replies. Thus the firewall could prevent proper OS detection. To reduce this problem, reference fingerprints generally omit the R=Y test from the IE and U1 probes, which are the ones most likely to be dropped. In addition, if Nmap is missing a closed TCP port for a target, it will not set R=N for the T5, T6, or T7 tests even if the port it tries is non-responsive. After all, the lack of a closed port may be because they are all filtered.

IP don’t fragment bit (DF) The IP header contains a single bit which forbids routers from fragmenting a packet. If the packet is too large for routers to handle, they will just have to drop it (and ideally return a “destination unreachable, fragmentation needed” response). This test records Y if the bit is set, and N if it isn’t.

Don’t fragment (ICMP) (DFI) This is simply a modified version of the DF test that is used for the special IE probes. It compares results of the don’t fragment bit for the two ICMP echo request probes sent. It has four possible values, which are enumerated in Table 8.2.

Table 8.2. DFI test values

Value Description N Neither of the ping responses have the DF bit set. S Both responses echo the DF value of the probe. Y Both of the response DF bits are set. O The one remaining other combination-both responses have the DF bit toggled.

IP initial time-to-live (T) IP packets contain a field named time-to-live (TTL) which is decremented every time they traverse a router. If the field reaches zero, the packet must be discarded. This prevents packets from looping endlessly. Because operating systems differ on which TTL they start with, it can be used for Os Detection Techniques. Nmap determines how many hops away it is from the target by examining the ICMP port unreachable response to the U1 probe. That response includes the original IP packet, including the already-decremented TTL field, received by the target. By subtracting that value from our as-sent TTL, we learn how many hops away the machine is. Nmap then adds that hop distance to the probe response TTL to determine what the initial TTL was when that ICMP probe response packet was sent. That initial TTL value is stored in the fingerprint as the T result.

Even though an eight-bit field like TTL can never hold values greater than 0xFF, this test occasionally results in values of 0x100 or higher. This occurs when a system (could be the source, a target, or a system in between) corrupts or otherwise fails to correctly decrement the TTL. It can also occur due to asymmetric routes.

Nmap can also learn from the system interface and routing tables when the hop distance is zero (localhost scan) or one (on the same network segment). This value is used when Nmap prints the hop distance for the user, but it is not used for T result computation.

IP initial time-to-live guess (TG) It is not uncommon for Nmap to receive no response to the U1 probe, which prevents Nmap from learning how many hops away a target is. Firewalls and NAT devices love to block unsolicited UDP packets. But since common TTL values are spread well apart and targets are rarely more than 20 hops away, Nmap can make a pretty good guess anyway. Most systems send packets with an initial TTL of 32, 60, 64, 128, or 255. So the TTL value received in the response is rounded up to the next value out of 32, 64, 128, or 255. 60 is not in that list because it cannot be reliably distinguished from 64. It is rarely seen anyway. The resulting guess is stored in the TG field. This TTL guess field is not printed in a subject fingerprint if the actual TTL (T) value was discovered.

Explicit congestion notification (CC) This test is only used for the ECN probe. That probe is a SYN packet which includes the CWR and ECE congestion control flags. When the response SYN/ACK is received, those flags are examined to set the CC (congestion control) test value as described in Table 8.3.

Table 8.3. CC test values

Value Description Y Only the ECE bit is set (not CWR). This host supports ECN. N Neither of these two bits is set. The target does not support ECN. S Both bits are set. The target does not support ECN, but it echoes back what it thinks is reserved bit. O The one remaining combination of these two bits (other).

TCP miscellaneous quirks (Q) This tests for two quirks that a few implementations have in their TCP stack. The first is that the reserved field in the TCP header (right after the header length) is nonzero. This is particularly likely to happen in response to the ECN test as that one sets a reserved bit in the probe. If this is seen in a packet, an “R” is recorded in the Q string.

The other quirk Nmap tests for is a nonzero urgent pointer field value when the URG flag is not set. This is also particularly likely to be seen in response to the ECN probe, which sets a non-zero urgent field. A “U” is appended to the Q string when this is seen.

The Q string must always be generated in alphabetical order. If no quirks are present, the Q test is empty but still shown.

TCP sequence number (S) This test examines the 32-bit sequence number field in the TCP header. Rather than record the field value as some other tests do, this one examines how it compares to the TCP acknowledgment number from the probe that elicited the response. It then records the appropriate value as shown in Table 8.4.

Table 8.4. S test values

Value Description Z Sequence number is zero. A Sequence number is the same as the acknowledgment number in the probe. A+ Sequence number is the same as the acknowledgment number in the probe plus one. O Sequence number is something else (other).

TCP acknowledgment number (A) This test is the same as S except that it tests how the acknowledgment number in the response compares to the sequence number in the respective probe. The four possible values are given in Table 8.5.

Table 8.5. A test values

Value Description Z Acknowledgment number is zero. S Acknowledgment number is the same as the sequence number in the probe. S+ Acknowledgment number is the same as the sequence number in the probe plus one. O Acknowledgment number is something else (other).

TCP flags (F) This field records the TCP flags in the response. Each letter represents one flag, and they occur in the same order as in a TCP packet (from high-bit on the left, to the low ones). So the value AS represents the ACK and SYN bits set, while the value SA is illegal (wrong order). The possible flags are shown in Table 8.6.

Table 8.6. F test values

Character Flag name Flag byte value E ECN Echo (ECE) 64 U Urgent Data (URG) 32 A Acknowledgment (ACK) 16 P Push (PSH) 8 R Reset (RST) 4 S Synchronize (SYN) 2 F Final (FIN) 1

TCP RST data checksum (RD) Some operating systems return ASCII data such as error messages in reset packets. This is explicitly allowed by section 4.2.2.12 of RFC 1122. When Nmap encounters such data, it performs a CRC32 checksum and reports the results. When there is no data, RD is set to zero. Some of the few operating systems that may return data in their reset packets are HP-UX and versions of Mac OS prior to Mac OS X.

IP total length (IPL) This test records the total length (in octets) of an IP packet. It is only used for the port unreachable response elicited by the U1 test. That length varies by implementation because they are allowed to choose how much data from the original probe to include, as long as they meet the minimum RFC 1122 requirement. That requirement is to include the original IP header and at least eight bytes of data.

Unused port unreachable field nonzero (UN) An ICMP port unreachable message header is eight bytes long, but only the first four are used. RFC 792 states that the last four bytes must be zero. A few implementations (mostly ethernet switches and some specialized embedded devices) set it anyway. The value of those last four bytes is recorded in this field.

Returned probe IP total length value (RIPL) ICMP port unreachable messages (as are sent in response to the U1 probe) are required to include the IP header which generated them. This header should be returned just as they received it, but some implementations send back a corrupted version due to changes they made during IP processing. This test simply records the returned IP total length value. If the correct value of 0x148 (328) is returned, the value G (for good) is stored instead of the actual value.

Returned probe IP ID value (RID) The U1 probe has a static IP ID value of 0x1042. If that value is returned in the port unreachable message, the value G is stored for this test. Otherwise the exact value returned is stored. Some systems, such as Solaris, manipulate IP ID values for raw IP packets that Nmap sends. In such cases, this test is skipped. We have found that some systems, particularly HP and Xerox printers, flip the bytes and return 0x4210 instead.

Integrity of returned probe IP checksum value (RIPCK) The IP checksum is one value that we don’t expect to remain the same when returned in a port unreachable message. After all, each network hop during transit changes the checksum as the TTL is decremented. However, the checksum we receive should match the enclosing IP packet. If it does, the value G (good) is stored for this test. If the returned value is zero, then Z is stored. Otherwise the result is I (invalid).

Integrity of returned probe UDP checksum (RUCK) The UDP header checksum value should be returned exactly as it was sent. If it is, G is recorded for this test. Otherwise the value actually returned is recorded.

Integrity of returned UDP data (RUD) This test checks the integrity of the (possibly truncated) returned UDP payload. If all the payload bytes are the expected ‘C’ (0x43), or if the payload was truncated to zero length, G is recorded; otherwise, I (invalid) is recorded.

ICMP response code (CD) The code value of an ICMP echo reply (type zero) packet is supposed to be zero. But some implementations wrongly send other values, particularly if the echo request has a nonzero code (as one of the IE tests does). The response code values for the two probes are combined into a CD value as described in Table 8.7.

Table 8.7. CD test values

Value Description Z Both code values are zero. S Both code values are the same as in the corresponding probe. When they both use the same non-zero number, it is shown here. O Any other combination.

Usage and Examples The inner workings of OS detection are quite complex, but it is one of the easiest features to use. Simply add -O to your scan options. You may want to also increase the verbosity with -v for even more OS-related details. This is shown in Example 8.1.

Example 8.1. OS detection with verbosity (-O -v)

# nmap -O -v scanme.nmap.org

Starting Nmap ( https://nmap.org ) Nmap scan report for scanme.nmap.org (74.207.244.221) Not shown: 994 closed ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 646/tcp filtered ldp 1720/tcp filtered H.323/Q.931 9929/tcp open nping-echo 31337/tcp open Elite Device type: general purpose Running: Linux 2.6.X OS CPE: cpe:/o:linux:linux_kernel:2.6.39 OS details: Linux 2.6.39 Uptime guess: 1.674 days (since Fri Sep 9 12:03:04 2011) Network Distance: 10 hops TCP Sequence Prediction: Difficulty=205 (Good luck!) IP ID Sequence Generation: All zeros

Read data files from: /usr/local/bin/../share/nmap Nmap done: 1 IP address (1 host up) scanned in 5.58 seconds Raw packets sent: 1063 (47.432KB) | Rcvd: 1031 (41.664KB)

Including the -O -v options caused Nmap to generate the following extra line items:

Device type All fingerprints are classified with one or more high-level device types, such as router, printer, firewall, or (as in this case) general purpose. These are further described in the section called “Device and OS classification (Class lines)”. Several device types may be shown, in which case they will be separated with the pipe symbol as in “Device Type: router|firewall”.

Running This field is also related to the OS classification scheme described in the section called “Device and OS classification (Class lines)”. It shows the OS Family (Linux in this case) and OS generation (2.6.X) if available. If there are multiple OS families, they are separated by commas. When Nmap can’t narrow down OS generations to one specific choice, options are separated by the pipe symbol (‘|’) Examples include OpenBSD 3.X, NetBSD 3.X|4.X and Linux 2.4.X|2.5.X|2.6.X.

If Nmap finds too many OS families to print concisely, it will omit this line. When there are no perfect matches, Nmap changes the field to Running (JUST GUESSING) and adds an accuracy percentage (100% is a perfect match) in parentheses after each candidate family name. If no fingerprints are close matches, the line is omitted.

OS CPE This shows a Common Platform Enumeration (CPE) representation of the operating system when available. It may also have a CPE representation of the hardware type. OS CPE begins with cpe:/o and hardware CPE begins with cpe:/h. For more about CPE see the section called “Common Platform Enumeration (CPE)”.

OS details This line gives the detailed description for each fingerprint that matches. While the Device type and Running lines are from predefined enumerated lists that are easy to parse by a computer, the OS details line contains free-form data which is useful to a human reading the report. This can include more exact version numbers, device models, and architectures specific to a given fingerprint. In this example, the only matching fingerprint was Linux 2.6.20-1 ( Fedora Core 5). When there are multiple exact matches, they are comma-separated. If there aren’t any perfect matches, but some close guesses, the field is renamed Aggressive OS guesses and fingerprints are shown followed by a percentage in parentheses which specifies how close each match was.

Uptime guess As part of OS detection, Nmap receives several SYN/ACK TCP packets in a row and checks the headers for a timestamp option. Many operating systems use a simple counter for this which starts at zero at boot time then increments at a constant rate such as twice per second. By looking at several responses, Nmap can determine the current values and rate of increase. Simple linear extrapolation determines boot time. The timestamp algorithm is used for OS detection too (see the section called “TCP timestamp option algorithm (TS)”) since the increment rate on different systems varies from 2 Hz to 1,000 Hz.

The uptime guess is labeled a “guess” because various factors can make it completely inaccurate. Some operating systems do not start the timestamp counter at zero, but initialize it with a random value, making extrapolation to zero meaningless. Even on systems using a simple counter starting at zero, the counter eventually overflows and wraps around. With a 1,000 Hz counter increment rate, the counter resets to zero roughly every 50 days. So a host that has been up for 102 days will appear to have been up only two days. Even with these caveats, the uptime guess is accurate much of the time for most operating systems, so it is printed when available, but only in verbose mode. The uptime guess is omitted if the target gives zeros or no timestamp options in its SYN/ACK packets, or if it does not reply at all. The line is also omitted if Nmap cannot discern the timestamp increment rate or it seems suspicious (like a 30-year uptime).

Network Distance A side effect of one of the OS detection tests allows Nmap to compute how many routers are between it and a target host. The distance is zero when you are scanning localhost, and one for a machine on the same network segment. Each additional router on the path adds one to the hop count. The Network Distance line is not printed in this example, since Nmap omits the line when it cannot be computed (no reply to the relevant probe).

TCP Sequence Prediction Systems with poor TCP initial sequence number generation are vulnerable to blind TCP spoofing attacks. In other words, you can make a full connection to those systems and send (but not receive) data while spoofing a different IP address. The target’s logs will show the spoofed IP, and you can take advantage of any trust relationship between them. This attack was all the rage in the mid-nineties when people commonly used rlogin to allow logins to their account without any password from trusted IP addresses. Kevin Mitnick is alleged to have used this attack to break into Tsutomu Shimomura’s computers in December 1994.

The good news is that hardly anyone uses rlogin anymore, and many operating systems have been fixed to use unpredictable initial sequence numbers as proposed by RFC 1948. For these reasons, this line is only printed in verbose mode. Sadly, many vendors still ship vulnerable operating systems and devices. Even the fixed ones often vary in implementation, which leaves them valuable for OS detection purposes. The class describes the ISN generation algorithm used by the target, and difficulty is a rough estimate of how hard the system makes blind IP spoofing (0 is the easiest). The parenthesized comment is based on the difficulty index and ranges from Trivial joke to Easy, Medium, Formidable, Worthy challenge, and finally Good luck! Further details about sequence tests are provided in the section called “TCP ISN greatest common divisor (GCD)”.

While the rlogin family is mostly a relic of the past, clever attackers can still find effective uses for blind TCP spoofing. For example, it allows for spoofed HTTP requests. You don’t see the results, but just the URL (POST or GET request) can have dramatic side effects. The spoofing allows attackers to hide their identity, frame someone else, or exploit IP address restrictions.

IP ID sequence generation Many systems unwittingly give away sensitive information about their traffic levels based on how they generate the lowly 16-bit ID field in IP packets. This can be abused to spoof a port scan against other systems and for other mischievous purposes discussed in the section called “TCP Idle Scan (-sI)”. This field describes the ID generation algorithm that Nmap was able to discern. More information on how it classifies them is available in the section called “IP ID sequence generation algorithm (TI, CI, II)”. Note that many systems use a different IP ID space for each host they communicate with. In that case, they may appear vulnerable (such as showing the Incremental class) while still being secure against attacks such as the idle scan. For this reason, and because the issue is rarely critical, the IP ID sequence generation line is only printed in verbose mode. If Nmap does not receive sufficient responses during OS detection, it will omit the whole line. The best way to test whether a host is vulnerable to being an idle scan zombie is to test it with -sI.

While TCP fingerprinting is a powerful method for OS detection, interrogating open ports for clues is another effective approach. Some applications, such as Microsoft IIS, only run on a single platform (thus giving it away), while many other apps divulge their platform in overly verbose banner messages. Adding the -sV option enables Nmap version detection, which is trained to look for these clues (among others). In Example 8.2, Nmap catches the platform details from an FTP server.

Example 8.2. Using version scan to detect the OS

# nmap -sV -O -v 129.128.X.XX

Starting Nmap ( https://nmap.org ) Nmap scan report for [hostname] (129.128.X.XX) Not shown: 994 closed ports PORT STATE SERVICE VERSION 21/tcp open ftp HP-UX 10.x ftpd 4.1 22/tcp open ssh OpenSSH 3.7.1p1 (protocol 1.99) 111/tcp open rpc 445/tcp filtered microsoft-ds 1526/tcp open oracle-tns Oracle TNS Listener 32775/tcp open rpc No exact OS matches for host TCP Sequence Prediction: Class=truly random Difficulty=9999999 (Good luck!) IP ID Sequence Generation: Incremental Service Info: OS: HP-UX

In this example, the line “No exact OS matches for host” means that TCP/IP fingerprinting failed to find an exact match. Fortunately, the Service Info field a few lines down discloses that the OS is HP-UX. If several operating systems were detected (which can happen with NAT gateway boxes that redirect ports to several different machines), the field would be OSs and the values would be comma separated. The Service Info line can also contain hostnames and device types found during the version scan. The focus of this chapter is on TCP/IP fingerprinting though, since version detection was covered in Chapter 7, Service and Application Version Detection.

With two effective Os Detection Techniques available, which one should you use? The best answer is usually both. In some cases, such as a proxy firewall forwarding to an application on another host, the answers may legitimately differ. TCP/IP fingerprinting will identify the proxy while version scanning will generally detect the server running the proxied application. Even when no proxying or port forwarding is involved, using both techniques is beneficial. If they come out the same, that makes the results more credible. If they come out wildly different, investigate further to determine what is going on before relying on either. Since OS and version detection go together so well, the -A option enables them both.

OS detection is far more effective if at least one open and one closed TCP port are found. Set the –osscan-limit option and Nmap will not even try OS detection against hosts which do not meet this criteria. This can save substantial time, particularly on -Pn scans against many hosts. You still need to enable OS detection with -O (or -A) for the –osscan-limit option to have any effect.

Another OS detection option is –osscan-guess. When Nmap is unable to detect a perfect OS match, it sometimes offers up near-matches as possibilities. The match has to be very close for Nmap to do this by default. If you specify this option (or the equivalent –fuzzy option), Nmap will guess more aggressively. Nmap still tells you when an imperfect match is found and display its confidence level (percentage) for each guess.

When Nmap performs OS detection against a target and fails to find a perfect match, it usually repeats the attempt. By default, Nmap tries five times if conditions are favorable for OS fingerprint submission, and twice when conditions aren’t so good. The –max-os-tries option lets you change this maximum number of OS detection tries. Lowering it (usually to 1) speeds Nmap up, though you miss out on retries which could potentially identify the OS. Alternatively, a high value may be set to allow even more retries when conditions are favorable. This is rarely done, except to generate better fingerprints for submission and integration into the Nmap OS database.

Like just about every other part of Nmap, results ultimately come from the target machine itself. While rare, systems are occasionally configured to confuse or mislead Nmap. Several programs have even been developed specifically to trick Nmap OS detection (see the section called “OS Spoofing”). Your best bet is to use numerous reconnaissance methods to explore a network, and don’t trust any one of them.

TCP/IP fingerprinting requires collecting detailed information about the target’s IP stack. The most commonly useful results, such as TTL information, are printed to Nmap output whenever they are obtained. Slightly less pertinent information, such as IP ID sequence generation and TCP sequence prediction difficulty, is only printed in verbose mode. But if you want all of the IP stack details that Nmap collected, you can find it in a compact form called a subject fingerprint. Nmap sometimes prints this (for user submission purposes) when it doesn’t recognize a host. You can also force Nmap to print it (in normal, interactive, and XML formats) by enabling debugging with (-d). Then read the section called “Understanding an Nmap Fingerprint” to interpret it.

P0f brief overview of how tool does os detection

p0f – https://lcamtuf.coredump.cx/p0f3/

P0f is a tool that utilizes an array of sophisticated, purely passive traffic fingerprinting mechanisms to identify the players behind any incidental TCP/IP communications (often as little as a single normal SYN) without interfering in any way. Version 3 is a complete rewrite of the original codebase, incorporating a significant number of improvements to network-level fingerprinting, and introducing the ability to reason about application-level payloads (e.g., HTTP). Some of p0f’s capabilities include:

Highly scalable and extremely fast identification of the operating system and software on both endpoints of a vanilla TCP connection – especially in settings where NMap probes are blocked, too slow, unreliable, or would simply set off alarms.Measurement of system uptime and network hookup, distance (including topology behind NAT or packet filters), user language preferences, and so on. Automated detection of connection sharing / NAT, load balancing, and application-level proxying setups. Detection of clients and servers that forge declarative statements such as X-Mailer or User-Agent. The tool can be operated in the foreground or as a daemon, and offers a simple real-time API for third-party components that wish to obtain additional information about the actors they are talking to. Common uses for p0f include reconnaissance during penetration tests; routine network monitoring; detection of unauthorized network interconnects in corporate environments; providing signals for abuse-prevention tools; and miscellanous forensics.

p0f is a passive TCP/IP stack fingerprinting tool. p0f can attempt to identify the system running on machines that send network traffic to the box it is running on, or to a machine that shares a medium with the machine it is running on. p0f can also assist in analysing other aspects of the remote system

By inspecting network traffic passively, p0f can attempt to identify the operating systems on remote machines that send TCP packets to the detecting machine’s network interface, or to a physical subnet that the detecting machine can listen on.[1] Since version 3, p0f is also able to deduce aspects of the remote system by inspecting application-level HTTP messages.[1]

p0f can also check for firewall presence. It can estimate the distance to a remote system and calculate its uptime. It also guesses the remote system’s means of connecting to the network ( DSL, OC3, etc.).[1]

Unlike tools like nmap, p0f does not generate traffic.[1] Instead, it determines the operating system of the remote host by analyzing certain fields in the captured packets. This can have benefits in environments where actively creating network traffic would cause unhelpful side effects. In particular, the remote system will not be able to detect the packet capture and inspection.

Usage Signatures used for packet inspection are stored in a simple text file.[2] This allows them to be modified without recompiling p0f. The user is allowed to use a different fingerprinting file by selecting another one at run time.

p0f does not have a graphical user interface. It is instead run from the command line prompt.

============================= p0f v3: passive fingerprinter =============================

https://lcamtuf.coredump.cx/p0f3.shtml

Some of its capabilities include:

– Highly scalable and extremely fast identification of the operating system and software on both endpoints of a vanilla TCP connection – especially in settings where NMap probes are blocked, too slow, unreliable, or would simply set off alarms,

– Measurement of system uptime and network hookup, distance (including topology behind NAT or packet filters), and so on.

– Automated detection of connection sharing / NAT, load balancing, and application-level proxying setups.

– Detection of dishonest clients / servers that forge declarative statements such as X-Mailer or User-Agent.

The tool can be operated in the foreground or as a daemon, and offers a simple real-time API for third-party components that wish to obtain additional information about the actors they are talking to.

Common uses for p0f include reconnaissance during penetration tests; routine network monitoring; detection of unauthorized network interconnects in corporate environments; providing signals for abuse-prevention tools; and miscellanous forensics.

A snippet of typical p0f output may look like this:

A live demonstration can be seen here:

https://lcamtuf.coredump.cx/p0f3/

A vast majority of metrics used by p0f were invented specifically for this tool, and include data extracted from IPv4 and IPv6 headers, TCP headers, the dynamics of the TCP handshake, and the contents of application-level payloads.

For TCP/IP, the tool fingerprints the client-originating SYN packet and the first SYN+ACK response from the server, paying attention to factors such as the ordering of TCP options, the relation between maximum segment size and window size, the progression of TCP timestamps, and the state of about a dozen possible implementation quirks (e.g. non-zero values in “must be zero” fields).

The metrics used for application-level traffic vary from one module to another; where possible, the tool relies on signals such as the ordering or syntax of HTTP headers or SMTP commands, rather than any declarative statements such as User-Agent. Application-level fingerprinting modules currently support HTTP. Before the tool leaves “beta”, I want to add SMTP and FTP. Other protocols, such as FTP, POP3, IMAP, SSH, and SSL, may follow.

The list of all the measured parameters is reviewed in section 5 later on. Some of the analysis also happens on a higher level: inconsistencies in the data collected from various sources, or in the data from the same source obtained over time, may be indicative of address translation, proxying, or just plain trickery. For example, a system where TCP timestamps jump back and forth, or where TTLs and MTUs change subtly, is probably a NAT device.

https://subinsb.com/default-device-ttl-values/ https://securitytrails.com/blog/cybersecurity-fingerprinting https://forensicswiki.org/wiki/OS_fingerprinting https://en.wikipedia.org/wiki/Zero-configuration_networking#Service_discovery https://en.wikipedia.org/wiki/Service_discovery https://en.wikipedia.org/wiki/Server_Message_Block https://en.wikipedia.org/wiki/List_of_products_that_support_SMB https://en.wikipedia.org/wiki/NetBIOS https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol https://en.wikipedia.org/wiki/User_agent

That’s a great job you’ve done

Hi! Thanks for the guide. I wonder if you could help me out. For the assemble command Im getting error:…

Thank! For MakOS Yosemite, the 3rd method is the simplest.

That was awesome thanks! Since my app is greater than 150mb i had to use file extensions, do you know…

This is very informative and well detailed. That is great and very helpful for hacking beginners. Thanks!

Nmap OS Detection: TCP/IP Fingerprinting Methods Explained

TCP/IP Fingerprinting Methods Supported by Nmap – Os Detection Techniques

Leave a Reply Cancel reply