Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak...
Transcript of Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak...
![Page 1: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/1.jpg)
Network
Troubleshooting The Inter-hospital
EchoPACS Case
By Tak Fan
![Page 2: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/2.jpg)
About Me
Tak Fan
UoT Graduate, Centennial Graduate
Approaching 5 years professional
experience
UHN/SHS Biomedical Engineering dept.
Work with ultrasound and x-ray modalities
So I'm here to share my personal and
professional experience
Network Troubleshooting
by Tak Fan
![Page 3: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/3.jpg)
Network Basics
What is a Network?
A system of interconnected things [devices]
Network Troubleshooting
by Tak Fan
![Page 4: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/4.jpg)
Troubleshooting Step by Step
Resolution
Investigate
Interpret
Hypothesize
Test
(Review)
Investigate
Interpret
Problem
Test
N
Y
Network Troubleshooting
by Tak Fan
Good
Hypothesis
?
Resolved?
Y
N
![Page 5: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/5.jpg)
Troubleshooting Methodology
No perfect way to troubleshoot
Decide case-by-case
Network Troubleshooting
by Tak Fan
Screenshot of Monument Valley, © Ustwo Games
![Page 6: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/6.jpg)
Troubleshooting Some methods include
Experience Essentially knowing the answer
Replicating/Functional Test See where the problem goes
Theory-crafting If x then y because z
Isolating Divide and conquer
“Shotgunning” Quick semi-random checks
Be creative! (but also cautious)
Isolating
“Shotgunning”
Replicating/
Functional Test
Theory-Crafting
Experience
Network Troubleshooting
by Tak Fan
![Page 7: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/7.jpg)
Methodology Replication
Very often you start with Replication
Make sure you have a problem
It’s fast and usually very low effort
It gives you a platform to stand on and
base your ideas around
You might see something the users did not
Network Troubleshooting
by Tak Fan
![Page 8: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/8.jpg)
Methodology Isolation
Divide and conquer
In hospitals, networking systems are
divided among different departments
Thus very likely to try to Isolate
Network Troubleshooting
by Tak Fan
Clinical
Dept.
Biomedical Engineering
Information Technology Facilities
PACS
![Page 9: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/9.jpg)
Methodology Isolating – Client or Network?
Simplify the network
Is the issue local to the device?
Is there another similar device and does it work?
Do you have a network connection?
Network Troubleshooting
by Tak Fan
Server (eg. PACS)
Clinical Device
Workstation
Network Node Network Node
Different Device
![Page 10: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/10.jpg)
Isolating Client Verification – Connected?
Hardware Based Network cable attached / Wi-fi working
Network jack is in a good condition Network Tester (+cable verification)
Software Based Ipconfig for network address
Ping = on the network
Network Troubleshooting
by Tak Fan
![Page 11: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/11.jpg)
Isolating Network Verification
Is it an intermittent issue?
Could be static IP conflict (Ping your IP)
Try DHCP if possible
Is the network down or congested? Connect a laptop
Contact IT
Is the server down or rejecting?
Dicom ping
Downtime/backup Server works?
Contact PACS team
Network Troubleshooting
by Tak Fan
![Page 12: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/12.jpg)
Methodology Theory-crafting and “Shotgunning”
A bit of theory is necessary
Some people like to sit and think on problems
Saves the trouble of physical labour
Ask a coworker and talk it over with a coffee
Or Vice-versa you can “shotgun”
Often quick for simple problems
Luck is a skill too (if you don’t have any other
ideas)
Network Troubleshooting
by Tak Fan
![Page 13: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/13.jpg)
Theory-Crafting OSI Model/Theory
Clients follow OSI Model
Network Troubleshooting
by Tak Fan
Application (Layer 7)
Presentation (Layer 6)
Session (Layer 5)
Transport (Layer 4)
Network (Layer 3)
Data Link (Layer 2)
Physical (Layer 1)
DICOM Ping
Ping
Connection
Server
Connection= IP/Networked
Ping = Connected with destination
DICOM Ping = Authorized for services
![Page 14: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/14.jpg)
Methodology ”Shotgunning”
Restart the system, who knows when it
comes to software!
Sometimes you just look at the network
cable in the back and it’s broken
I get these hunches with some users…
Network Troubleshooting
by Tak Fan
![Page 15: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/15.jpg)
Case Study The Inter-hospital EchoPACS Case
EchoPACS is a GE software application that manages Ultrasound Echocardiogram studies
Stores on ImageVault
These are loaded onto hospital computers
Studies are often sent
between hospitals
(eg. Surgery referrals)
Network Troubleshooting
by Tak Fan
A B
![Page 16: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/16.jpg)
Case Study The Problem
EchoPACS workstation could not
send an echocardiogram to
another hospital (PACS)
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y A B
![Page 17: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/17.jpg)
Case Study Investigate
Replicated and confirmed issue
Manager confirmed destination
PACS says they don’t have issues
Studies were sent just a week ago
Can the problem be isolated?
Only one PC set up to send
Communicates with ImageVault
No problems internal to hospital
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
![Page 18: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/18.jpg)
Case Study Investigate
In theory, everything should be
working
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
A B
EchoPACS PC
In-house IT
ImageVault
Server
Destination PC
Destination IT
Destination PACS
![Page 19: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/19.jpg)
Case Study Some Hypotheses
“The destination PACS is not working. There's a disconnect or rejection.”
“The hospital IT must be not routed or blocking outgoing traffic.”
“The system doesn't work. Must be a software issue.”
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
X
![Page 20: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/20.jpg)
Test Is it the Destination PACS?
Ping; OK
DICOM PING; OK
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
A B
![Page 21: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/21.jpg)
Test Is it the Destination PACS?
In fact they were able to help set
up another EchoPAC workstation
so we could test it further
And that workstation could send
images (at least we’re OK for
now)
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
A B
A2
X
![Page 22: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/22.jpg)
Test Is it the In-house IT?
In-house IT had been making a lot
of network changes recently
“No, it’s your end”, “OK.. thx ”
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
A B
X
![Page 23: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/23.jpg)
Test Is it the system’s fault?
Everything is working except
sending outside the hospital
Maybe it’s a network
configuration issue
“Dataflows” (network
configuration profiles)
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
![Page 24: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/24.jpg)
Test Is it the system’s fault?
I comb over all the network configurations, they seem correct
Load a correct configuration
Software was reloaded 4 weeks ago
Could not send from local or remote
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
A B
X
![Page 25: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/25.jpg)
Investigate What now?
Continue Investigating
Life line – Call a “friend”
I contacted GE who forwarded me
to remote help.
They were able to track the studies
sent
There was a flagged error in their logs
“Not enough memory in spooler”???
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
![Page 26: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/26.jpg)
Interpret Spooler Out of Memory
Maybe… there is not memory in
the spooler to queue up studies
thus they cannot process
But…
The hard drive wasn’t full
The system sends internally on the
hospital network without problems
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
![Page 27: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/27.jpg)
Hypothesis Clear the spooler?
If there is not enough memory to
process the studies, then clearing
up space might allow them to
send
Is it good (logical)? No, but I’m
desperate
(Remote support wasn’t 100%
certain either)
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
![Page 28: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/28.jpg)
Test Clear the Spooler!
Shotgun away!
We deleted the 50 gb of old
studies from the last 4-5 years
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
![Page 29: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/29.jpg)
Resolved? Sent a Study
Sent! (Finally…)
It turns out, that folder is a DICOM
only spooler that would only be
utilized for DICOM Servers (like the
destination PACS)
And there is a memory cap
around 50GB
That's just how it's programmed
Network Troubleshooting
by Tak Fan
Resolution
Investigate
Interpret
Problem
Test
Y
Good
Hypothesis
?
Resolved?
Y
![Page 30: Network Troubleshooting - Clinical Engineering Society of Ontario · 2019-03-06 · About Me Tak Fan UoT Graduate, Centennial Graduate Approaching 5 years professional experience](https://reader034.fdocuments.us/reader034/viewer/2022042304/5ecf97e3e806de00210544b9/html5/thumbnails/30.jpg)
Review What did I learn?
Be resourceful
Leave no stone unturned
You don’t know, until you know…
(Always keep learning!)
Network Troubleshooting
by Tak Fan