Debugging Network Issues
-
Upload
apcera -
Category
Technology
-
view
67 -
download
0
Transcript of Debugging Network Issues
Debugging Network Issues4 things learned the hard way and by asking all my friends a bunch of questions.
Who is this guy?
● Jaime Piña● Software engineer at Apcera● Apcera platform deploys apps to hybrid cloud with policy● Work on Apcera Setup, gateways, and other things
(Hai-meh)
Architecture (simplified)
Microservices sit on the network.
One does not simply
use the network.
Is the server plugged in?
(Is your app running?)
Connection refused!
$ ssh jaime@localhost
ssh: connect to host localhost port 22: Connection refused
$ curl http://localhost
curl: (7) Failed to connect to localhost port 80: Connection refused
Is there a firewall?
Connection refused! (Part 2)
$ curl http://1.2.3.4
curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused
Server side check with ufw
# ufw status
Status: active
To Action From
-- ------ ----
22 ALLOW Anywhere
Connection refused! (Part 2)
$ curl http://1.2.3.4
curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused
Server side check with iptables
A little more complicated.
iptables vocabulary
● INPUT chain○ Incoming
● OUTPUT chain○ Outgoing
● ACCEPT action○ Allow
● DROP action○ Block
iptables --listChain INPUT (policy DROP)target prot opt source destinationufw-user-input all -- anywhere anywhere
Chain ufw-user-input (1 references)target prot opt source destinationACCEPT tcp -- anywhere anywhere tcp dpt:sshACCEPT udp -- anywhere anywhere udp dpt:ssh
Connection refused! (Part 2)
$ curl http://1.2.3.4
curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused
Don’t have server access?
Client side check with nmap
$ nmap scanme.nmap.orgNot shown: 971 closed portsPORT STATE SERVICE22/tcp open ssh5269/tcp filtered xmpp-server6007/tcp filtered X11:7
nmap vocabulary
● open state○ Port accessible, app listening
● closed state○ Port accessible, no app listening
● filtered state○ IDK? ¯\_(ツ)_/¯
Client side check with nmap
$ nmap scanme.nmap.orgNot shown: 971 closed portsPORT STATE SERVICE22/tcp open ssh5269/tcp filtered xmpp-server6007/tcp filtered X11:7
Moar logs!
(And verbose output!)
Triple verbose SSH
$ ssh -vvv [email protected]
We can reach the server
debug2: resolving "foo.com" port 22debug1: Connecting to ejemplo.com [1.2.3.4] port 22.
debug1: Connection established.
Trying to read my public key
debug1: key_load_public: No such file or directorydebug1: identity file /home/jaime/.ssh/id_rsa type -1debug1: key_load_public: No such file or directorydebug1: identity file /home/jaime/.ssh/id_ed25519 type -1
Trying to use key authdebug1: Authentications that can continue: publickey,passworddebug1: Next authentication method: publickey
debug1: Trying private key: /home/jaime/.ssh/id_rsadebug3: no such identity: /home/jaime/.ssh/id_rsa: No such file or directorydebug1: Trying private key: /home/jaime/.ssh/id_ed25519debug3: no such identity: /home/jaime/.ssh/id_ed25519: No such file or directory
Trying to use password auth
debug1: Next authentication method: password
[email protected]'s password:
systemd
journalctl --unit ssh.service
Careful with DNS
DNS
The thing that turns English words into numbers.
Input: www.google.com
Output: 172.217.6.36
The thing is
DNS is not required for working internet.
One day...# rkt run --insecure-options=image --interactive docker://ubuntu:14.04root@rkt:/# apt-get update
Err http://archive.ubuntu.com trusty-updates InReleaseErr http://archive.ubuntu.com trusty-security InReleaseErr http://archive.ubuntu.com trusty-updates Release.gpg Could not resolve 'archive.ubuntu.com'Err http://archive.ubuntu.com trusty-security Release.gpg Could not resolve 'archive.ubuntu.com'Reading package lists... DoneW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InReleaseW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease
What’s happening??
Is the internet cable plugged in?
What’s happening??
Closer lookErr http://archive.ubuntu.com trusty-updates Release.gpg Could not resolve 'archive.ubuntu.com'Err http://archive.ubuntu.com trusty-security Release.gpg Could not resolve 'archive.ubuntu.com'Reading package lists... DoneW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InReleaseW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease
Test for success
root@rkt:/# ping -c 1 172.217.6.36PING 172.217.6.36 (172.217.6.36) 56(84) bytes of data.64 bytes from 172.217.6.36: icmp_seq=1 ttl=49 time=26.2 ms--- 172.217.6.36 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 26.297/26.297/26.297/0.000 ms
Test for failure
root@rkt:/# ping -c 1 www.google.comping: unknown host www.google.com
OMG it’s the DNS!
Fix
rkt run --insecure-options=image --dns 8.8.8.8 --interactive docker://ubuntu:14.04
Worked.
How to get DNS info?
drill (or dig)
drill usage
drill name [@nameserver] [record type]
DNS vocabulary
● nameserver○ Server who has DNS info about a domain
● A record○ Contains IP address for a domain
● NS record○ Contains nameservers for a domain
drill example
$ drill www.google.com @8.8.8.8 A;; QUESTION SECTION:;; www.google.com. IN A
;; ANSWER SECTION:www.google.com. 297 IN A 172.217.6.36
Packet inspection
(I do not think you’re sending what you think you’re sending.)
tcpdump syntax
tcpdump [options] 'BPF'
tcpdump basic usage
tcpdump -i lo 'BPF'
More: tcpdump --list-interfaces
Berkeley Packet Filter
● host foo.com○ src/dst host is foo.com
● tcp port 22○ TCP traffic coming/going to port 22
● dst port 53○ Traffic going to port 53
tcpdump example# tcpdump -i wlp58s0 'tcp port 80'192.168.0.109.37370 > 107.170.18.175.http: Flags [S], length 0107.170.18.175.http > 192.168.0.109.37370: Flags [S.], length 0192.168.0.109.37370 > 107.170.18.175.http: Flags [.], length 0
Flags:
S = SYN . = ACK
Me -SYN-> serverMe <-SYN ACK- serverMe -ACK-> server
World Famous
Three-way Handshake
tcpdump flag Flag name Description
S SYN Signal start of connection
. ACK Acknowledge packet
P PSH Sending data
F FIN Signal end of connection
R RST Connection killed
Common packet flags
ngrep syntax
ngrep [options] 'pattern' 'BPF'
ngrep example
# ngrep -q -Wbyline "HTTP" "tcp port 80"T 192.168.0.109:59990 -> 107.170.18.175:80 [AP]POST /portfolio/wp-login.php HTTP/1.1.Accept-Encoding: gzip, deflate..log=HELLO&pwd=WORLD&wp-submit=Log+In&redirect_to=http%3A%2F%2Ffoo.com%2Fportfolio%2Fwp-admin%2F&testcookie=1
Credentials
log=HELLO&pwd=WORLD
How to debug (some) network issues
● Is your app running?● Is there a firewall?● Does the DNS work?● Are you sending and receiving what you think you
are?
Thanks
Jaime Piña
Software engineer at Apcera
@variadico