Tools to use for network troubleshooting

I spent 3 days last week working with a customer, Microsoft support and out networking team identifying a number of slow network connections. While I don\’t want to go into any case specifics, I did want to share a couple of useful tools we used for troubleshooting and doing root cause analysis.

This is by no means a complete guide, but hopefully might give you a good idea on tools to use for network troubleshooting. Here are the tools I\’ll show you, and what to look for in those tools (most tools are Windows focused):

• ping: most basic network troubleshooting tool, showing you \”can I reach the destination\” and showing you latency to the endpoint. Please note that not every destination will reply to pings, so if ping says destination unreachable, that doesn\’t mean things don\’t work.
• traceroute or tracert: another basic network troubleshooting tool showing you the traffic path you take to a destination and the latency along the way.
• PSPing: ping leverages the ICMP protocol in stead of TCP. Generally speaking, ICMP traffic is dropped first in case routers/switches on the network get overloaded. PSPing is a ping equivalent that can use a TCP connection, which will not be dropped.
• WinMTR: traceroute on steroids, there is no other way to describe WinMTR. WinMTR shows you the same results as traceroute, but runs continously and shows you packet loss details as well.
• iperf: great tool for testing bandwidth. With iperf you can stresstest your network link.
• WinMTR + iperf combo: a great testing combo is combining WinMTR and iperf to see if any links on the network drop packets when traffic is increased.
• Wireshark: WireShark is a tool that allows you to do packet captures. This means WireShark actually stores a copy of all the packets sent to/from the machine it\’s running. This can give you great insight in network issues, such as missed packets, TCP retransmissions etc.

Let\’s have a look at these tools:

ping

I believe ping is the most basic network troubleshooting out there. It\’s the first thing I resolve to when I lose connectivity. If it seems like I lost internet, the first thing I do is try ping 1.1.1.1.

Under the covers, ping leverages the ICMP protocol. This protocol is used by network gear to communicate about network status. Important to know about the ping protocol is that it is not prioritized by network gear and ICMP packets are the first packets to be dropped by network gear if a piece of gear gets overloaded.

Let\’s walk through some of the data a ping result can show you:

very basic ping

The most basic ping you can do is ping hostname or ping ip-address.

An example below:

As you can see in the example below, you get a number of data points from this ping:

1. The hostname (google.com) is translated into an ip address. This means DNS is functioning.
2. By default on Windows, ping sends 4 packets. For each packet, you get the latency results.
3. In summary, you see that you had 0% packet loss and you get the average latency results.

If you want to force ping to use ipv4, you can force this using the -4 flag. Conversely, if you want to force ipv6, you can use the -6 flag.

Couple advanced ping tests:

There a number advanced flags in ping that are useful for more interesting results:

• The -l flag: This flag controls the data payload of the ping packet. By increasing the size of your pings, you could identify issues in the network by potential fragmentation.
• The -t flag or -i flag: These two flags can cause ping to run for more 4 times. The -t flag runs a continous ping, and the -i flag can be used together with a number to run a ping for X times. By having more than 4 results, you get more interest info, as shown below in the results of a ping I ran for 20 minutes (-i 1200)

What this shows me is that the network is loss-free, but there latency is highly variable. The average is double the minimum, and the maximum is 40x the average. Although this doesn\’t prove anything (remember, ICMP traffic is not prioritized on network gear) it can be indicative of something on the network.

Traceroute or tracert

Traceroute (tracert on the command line) is a tool that literally traces the route to a destination. Under the covers, like ping, traceroute relies on the ICMP protocol. It shows you all the intermediate hops on the network – if those hosts are replying to ICMP messages.

What traceroute can show you:

• What is your network path. Often you see public DNS names, meaning you can also identify which networks you\’re traversing.
• What is the latency to each step in the network.
• Are there major latency jumps in the network path to a host.(*)

(*) I once worked with a customer who noticed that traffic from Europe traveled to Asia by crossing the Atlantic Ocean three times and the Pacific Ocean once. This showed that the traffic was obviously not taking an optimal path.

Let\’s look at an example of a traceroute:

As you can see, you see the IP address of all the hops in the traffic path. We do see the latency to these hops, with three individual results. In this case, they don\’t resolve to DNS names, so we cannot really identify which network we\’re on. I\’ll show an example that does show DNS results on the Microsoft network (msn.net) later. As with ping, you can force traceroute traffic to use ipv4 with the -4 flag:

Interesting as well is that you can do a traceroute to a destination that does not reply to pings – or even to a destination that is not replying to any traffic. An example of this is the Azure portal. The Azure portal does not reply to ping messages, but as you can see from the screenshot below, you can still trace the route. This example also shows DNS names of the intermediate hops, showing you which network you\’re packets are on.

Please note that a traceroute isn\’t a static result. The internet is like the freeway network, and there are multiple routes that connect two endpoints together. Some of those routes have the same ETA/costs, and you\’re connection to a particular destination might be spread across multiple routes.

PSPing

As mentioned in the section about ping, the ping protocol uses the ICMP protocol which has a couple of issues. One is that the protocol is the first that will be dropped by network gear, another one is that a lot of endpoints don\’t reply to ping messages.

PSPing is a tool that you could call ping on steroids. You can use it to do more advanced ICMP based pings, as well as doing ping-like tests to TCP/UDP endpoints. You can download the tool here.

For example, portal.azure.com does not reply to ping messages. We can leverage the psping tool to do a TCP based ping to the endpoint and get the results we\’re looking for:

As with ping, you can tune a couple of parameters in psping as well, like the packet size, the amount of pspings that get executed. More info in the documentation of psping.

WinMTR

WinMTR is traceroute on steroids. It runs a similar process as traceroute continously. It shows you averages of latency and packets sent/received. You can download the tool here. Using WinMTR is pretty easy, you enter the hostname/IP address of the destination you\’re trying to reach, and hit the start button. Below you see a screenshot of my WinMTR results to google.com:

Iperf

Iperf is a tool that is used to test bandwidth to a destination. It\’s a tool that needs to be run on destination (called server) as well as on the client side.