Debugging Network Issues4 things learned the hard way and by asking all my friends a bunch of questions.

Who is this guy?

● Jaime Piña● Software engineer at Apcera● Apcera platform deploys apps to hybrid cloud with policy● Work on Apcera Setup, gateways, and other things

(Hai-meh)

Architecture (simplified)

Microservices sit on the network.

One does not simply

use the network.

Is the server plugged in?

(Is your app running?)

Connection refused!

$ ssh jaime@localhost

ssh: connect to host localhost port 22: Connection refused

$ curl http://localhost

curl: (7) Failed to connect to localhost port 80: Connection refused

Is there a firewall?

Connection refused! (Part 2)

$ curl http://1.2.3.4

curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused

Server side check with ufw

# ufw status

Status: active

To Action From

-- ------ ----

22 ALLOW Anywhere

Connection refused! (Part 2)

$ curl http://1.2.3.4

curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused

Server side check with iptables

A little more complicated.

iptables vocabulary

● INPUT chain○ Incoming

● OUTPUT chain○ Outgoing

● ACCEPT action○ Allow

● DROP action○ Block

iptables --listChain INPUT (policy DROP)target prot opt source destinationufw-user-input all -- anywhere anywhere

Chain ufw-user-input (1 references)target prot opt source destinationACCEPT tcp -- anywhere anywhere tcp dpt:sshACCEPT udp -- anywhere anywhere udp dpt:ssh

Connection refused! (Part 2)

$ curl http://1.2.3.4

curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused

Don’t have server access?

Client side check with nmap

$ nmap scanme.nmap.orgNot shown: 971 closed portsPORT STATE SERVICE22/tcp open ssh5269/tcp filtered xmpp-server6007/tcp filtered X11:7

nmap vocabulary

● open state○ Port accessible, app listening

● closed state○ Port accessible, no app listening

● filtered state○ IDK? ¯\_(ツ)_/¯

Client side check with nmap

$ nmap scanme.nmap.orgNot shown: 971 closed portsPORT STATE SERVICE22/tcp open ssh5269/tcp filtered xmpp-server6007/tcp filtered X11:7

Moar logs!

(And verbose output!)

Triple verbose SSH

$ ssh -vvv jaime@foo.com

We can reach the server

debug2: resolving "foo.com" port 22debug1: Connecting to ejemplo.com [1.2.3.4] port 22.

debug1: Connection established.

Trying to read my public key

debug1: key_load_public: No such file or directorydebug1: identity file /home/jaime/.ssh/id_rsa type -1debug1: key_load_public: No such file or directorydebug1: identity file /home/jaime/.ssh/id_ed25519 type -1

Trying to use key authdebug1: Authentications that can continue: publickey,passworddebug1: Next authentication method: publickey

debug1: Trying private key: /home/jaime/.ssh/id_rsadebug3: no such identity: /home/jaime/.ssh/id_rsa: No such file or directorydebug1: Trying private key: /home/jaime/.ssh/id_ed25519debug3: no such identity: /home/jaime/.ssh/id_ed25519: No such file or directory

Trying to use password auth

debug1: Next authentication method: password

jaime@foo.com's password:

systemd

journalctl --unit ssh.service

Careful with DNS

DNS

The thing that turns English words into numbers.

Input: www.google.com

Output: 172.217.6.36

The thing is

DNS is not required for working internet.

One day...# rkt run --insecure-options=image --interactive docker://ubuntu:14.04root@rkt:/# apt-get update

Err http://archive.ubuntu.com trusty-updates InReleaseErr http://archive.ubuntu.com trusty-security InReleaseErr http://archive.ubuntu.com trusty-updates Release.gpg Could not resolve 'archive.ubuntu.com'Err http://archive.ubuntu.com trusty-security Release.gpg Could not resolve 'archive.ubuntu.com'Reading package lists... DoneW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InReleaseW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease

What’s happening??

Is the internet cable plugged in?

What’s happening??

Closer lookErr http://archive.ubuntu.com trusty-updates Release.gpg Could not resolve 'archive.ubuntu.com'Err http://archive.ubuntu.com trusty-security Release.gpg Could not resolve 'archive.ubuntu.com'Reading package lists... DoneW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InReleaseW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease

Test for success

root@rkt:/# ping -c 1 172.217.6.36PING 172.217.6.36 (172.217.6.36) 56(84) bytes of data.64 bytes from 172.217.6.36: icmp_seq=1 ttl=49 time=26.2 ms--- 172.217.6.36 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 26.297/26.297/26.297/0.000 ms

Test for failure

root@rkt:/# ping -c 1 www.google.comping: unknown host www.google.com

OMG it’s the DNS!

Fix

rkt run --insecure-options=image --dns 8.8.8.8 --interactive docker://ubuntu:14.04

Worked.

How to get DNS info?

drill (or dig)

drill usage

drill name [@nameserver] [record type]

DNS vocabulary

● nameserver○ Server who has DNS info about a domain

● A record○ Contains IP address for a domain

● NS record○ Contains nameservers for a domain

drill example

$ drill www.google.com @8.8.8.8 A;; QUESTION SECTION:;; www.google.com. IN A

;; ANSWER SECTION:www.google.com. 297 IN A 172.217.6.36

Packet inspection

(I do not think you’re sending what you think you’re sending.)

tcpdump syntax

tcpdump [options] 'BPF'

tcpdump basic usage

tcpdump -i lo 'BPF'

More: tcpdump --list-interfaces

Berkeley Packet Filter

● host foo.com○ src/dst host is foo.com

● tcp port 22○ TCP traffic coming/going to port 22

● dst port 53○ Traffic going to port 53

tcpdump example# tcpdump -i wlp58s0 'tcp port 80'192.168.0.109.37370 > 107.170.18.175.http: Flags [S], length 0107.170.18.175.http > 192.168.0.109.37370: Flags [S.], length 0192.168.0.109.37370 > 107.170.18.175.http: Flags [.], length 0

Flags:

S = SYN . = ACK

Me -SYN-> serverMe <-SYN ACK- serverMe -ACK-> server

World Famous

Three-way Handshake

tcpdump flag Flag name Description

S SYN Signal start of connection

. ACK Acknowledge packet

P PSH Sending data

F FIN Signal end of connection

R RST Connection killed

Common packet flags

ngrep syntax

ngrep [options] 'pattern' 'BPF'

ngrep example

# ngrep -q -Wbyline "HTTP" "tcp port 80"T 192.168.0.109:59990 -> 107.170.18.175:80 [AP]POST /portfolio/wp-login.php HTTP/1.1.Accept-Encoding: gzip, deflate..log=HELLO&pwd=WORLD&wp-submit=Log+In&redirect_to=http%3A%2F%2Ffoo.com%2Fportfolio%2Fwp-admin%2F&testcookie=1

Credentials

log=HELLO&pwd=WORLD

How to debug (some) network issues

● Is your app running?● Is there a firewall?● Does the DNS work?● Are you sending and receiving what you think you

are?

Thanks

Jaime Piña

Software engineer at Apcera

@variadico