Internet Security Protocols

Protecting IP Traffic

UYLESS BLACK

Prentice Hall PTR Upper Saddle River, New Jersey 07458

www. phptr. com

Contents

PREFACE XVII

CHAPTER 1 Introduction

Security Problems 1 How Pervasive Are Security Attacks? 2 Types of Security Services 4 Introduction to the Firewall 5 The Security Policy 6

Trusted and Untrusted Networks 6 Security and Risk Management 7 Virtual Private Networks (VPNs) 7

The Modern VPN 8 VPNs and SLAs 11 The Debate of Privacy vs. Law Enforcement 12

CHAPTER 2 Types of Security Violations 15

Types of Security Problems 15 Denial of Service: Attacks and Counter-attacks 15

Virus 16

vii

Worm 17 Clogging or Flooding 18 Trojan Horse 18 Bomb 18 Trap Door 19 Salami 20 Replay Violations 20 Cookies 20 Applets and Sandboxes 22 Other Problems 24 Summary 24

CHAPTER 3 Basic Security Concepts 25

How Secure is Secure? 26 Definitions 26

Encryption and Decryption 26 Basic Encryption and Decryption Methods 27

The German Enigma Machine 29 Substitution and Transposition 30 One-way Functions and Modular Arithmetic 30

Example of a One-way Function 32 The Diffie-Hellman Idea Using

Modular Arithmetic 32 The Hash Function 33 Use of a One-way Hash Function 35 Randomness of Keys 35

Randomness or Lack Thereof Equals the Demise of a Crypto System 36

Key Problem: Exchanging Keys 37 Awkwardness of Key Distribution 38

The Asymmetric Key 39 Use of the Asymmetric Keys in Reverse Order 39 Asymmetric Keys for Privacy 40 Asymmetric Keys for Authentication:

The Digital Signature 41 The Next Step: RSA 42

The RSA Key Pairs 43 Key Transport and Key Generation 45 Message Authentication Code (MAC) and Key Hashing 45 Putting Together the Security Functions 46

CONTENTS ix

Paul Zimmerman and Pretty Good Privacy (PGP) 49 PGP's Use of Key Certificates 50 Example of a PGP Public Key 50 OpenPGP 51

Perfect Forward Secrecy (PFS) 52 Man-in-the-Middle Attack 52 Certification 54 The Certification Procedure 55 Anti-Replay Measures 55 Security in a Mobile Network 57

Authentication 57 Privacy Operations 58

Summary 59

CHAPTER 4 Firewalls 60

What is a Firewall? 60 Protection from Untrusted Networks 61

Permitting and Denying Services 62 What Firewalls Can Do and Cannot Do 62 Packet Filtering 64 Proxy or Application Firewalls 65 NCSA Guidance 66 Managed Firewall Services (MFWS) 67

Evaluating a Firewall Service Provider 69 Firewalls with Internet Security Protocols (IPSec) 70 SOCKS 71 Summary 71

CHAPTER 5 Prominent Internet Security Procedures 72

Diffie-Hellman 73 Diffie-Hellman and RFC 2631 74

Rivest, Shamir, and Adleman (RSA) 76 RSA in RFC 2437 76

MD5 78 MD5 Vulnerabilities? 81

RFC 2537: RSA, MD5, and DNS 81 RSA Public KEY Resource Records 81 RSA/MD5 SIG Resource Records 82 Performance Considerations 82

The Secure Hash Standard (SHA-1) and The Secure Hash Algorithm (SHA) 83

RIPEMD-160 83 Comparisons of MD5, SHA-1, RIPEMD-160,

and MD5-HMAC 83 HMAC 84 Performance and Security of HMAC 86 HMAC with IPSec 87 The OAKLEY Key Determination Protocol 87

Beyond Diffie-Hellman and STS 88 OAKLEY Key Exchange Processing 89 The Essential Key Exchange Message Fields 89

Summary 90

CHAPTER 6 PPP, ECP, TLS, EAP, DESE-bis, and 3DESE 91

PPP and HDLC 91 LCP 93 General Example of PPP Operations 94 PPP Phase Diagram 94

Link Dead (Physical Layer Not Ready) 95 Link Establishment Phase 96 Authentication Phase 96 Network Layer Protocol Phase 96 Link Termination Phase 96

LCP Packets 97 Configure-Request 98 Configure-Ack 98 Configure-Nak 98 Configure-Reject 99 Terminate-Request and Terminate-Ack 99 Code-Reject 99 Protocol-Reject 99 Echo-Request and Echo-Reply 99 Discard-Request 99

Other Supporting Cast Members for PPP Security Services 100

Transport Layer Security Protocol (TLS) 100 Goals of TLS 100

PPP Encryption Control Protocol (ECP) 101 PPP Extensible Authentication Protocol (EAP) 102

CONTENTS XI

PPP DES Encryption Protocol, Version 2 (DESE-bis) 103 Configuration Option for ECP 104 Packet Format for DESE 104

PPP Triple-DES Encryption Protocol (3DESE) 105 The Algorithm 105 Keys 106 3DESE Configuration Option for ECP 106 Packet Format for 3DESE 107

Summary 108

CHAPTER 7 Dial-in Operations with PAP, CHAP, RADIUS and DIAMETER 109

PAP and CHAP 109 PAP 109 Key Aspects of PAP 110

CHAP 111 CHAP Messages 111

RADIUS 113 RADIUS Configuration 114

Example of a RADIUS Message Exchange 115 UseofUDP 116 RADIUS Message Format 117 RADIUS Attributes 118 Examples of RADIUS Operations 120 Problems with RADIUS 121

DIAMETER 122 DIAMETER Message Formats 122

Message Header 122 Message Body for the AVP 123 DIAMETER-Command AVP 124 Message-Reject-Ind Command 125 Approach to the Remainder of Message

Descriptions 125 Basic Operations 129 DIAMETER Support of Dial-Ins To/From SS7 131

Session Setup Messages Signaling Gateway/ NAS Controller Interaction 131

Message Exchanges Examples 133 Summary 136

xii

CHAPTER 8 IPSec Architecture 137

Basics of IPSec 137 IPSec Services 138 IPSec Traffic Security Protocols 138 Security Association (SA) Databases 138 The IPSec Tunnel 139

The Security Association (SA) 139 Cases of Security Associations: A General View 141

Types of SAs: Transport Mode and Tunnel Mode 142 Combining Security Associations:

A More Detailed View 145 Placements of IPSec 147 The IPSec Databases 148 Selectors and SAD/SPD Operations 148

Destination IP Address 149 Source IP Address 149 Name 149 Transport Layer Protocol 149 Source and Destination Ports 150 Selectors and SAD/SPD Entries 150 Looking Up the SA in the SAD 150

Examples of IPSec Sending and Receiving Operations 151 Selecting and Using an SA or SA Bundle 155 Summary 156

CHAPTER 9 The IPSec AH and ESP Protocols 157

Services of the IPSec Protocols 157 Integrity Check Value (ICV) 158

Relationships of AH, ESP, and the Transport and Tunnel Modes 159

Handling Mutable Fields 159 Protection Coverage of the AH and ESP Packets 161

AH Protection 161 Services and Operations of AH 163

RFC 1826 163 RFC 2402 163 Integrity Check Value (ICV) for Outbound Packets 164 Integrity Check Value (ICV) for Inbound Packets 165

CONTENTS

Services and Operations of ESP 165 ESP Protection 165 RFC 1827 167 RFC 2406 168 Outbound Packet Processing 169 Inbound Packet Processing 170

AH and ESP and the "Cases" 171 IP Addressing in the Headers 173 Construction of the ESP Packet 175 Header Construction for Tunnel Mode 177 HMAC Applied to AH and ESP 179

MD5-HMAC-96 Within ESP and AH 180 MHAC-SHA-1 -96 Within ESP and AH 180

IPSecandNAT 181 Summary 182

CHAPTER 10 The Internet Key Distribution, Certification, and Management 183

What is Public Key Infrastructure (PKI)? 183 Certificates and Certification Authorities (CAs) 184 Support for Non-repudiation 185 Key Backup and Recovery 185 Using Two Key Pairs 186 Key Update and Management of Key Histories 186 Certificate Repositories and Certificate Distribution 187 Cross-certification 187

ISAKMP, ISAKMP DOI, and IKE 187 ISAKMP 188 The "Protection Suite" 188

Other Thoughts on Key Exchange 189 ISAKMP Negotiation Phases 190

Messages 191 The Generic Header 193 Data Attributes 194 The Payloads 194 OAKLEY and ISAKMP 205

Examples of ISAKMP Negotiations 205 The Base Exchange 206 The Identity Protection Exchange 207

xiii

XIV CONTENTS

Authentication Only Exchange 208 The Aggressive Exchange 208

ISAKMP Domain of Interpretation (DOI) 209 IPSec/ISAKMP Payloads 210 Summary 214

CHAPTER 11 Internet Key Exchange (IKE) 215

IKE Basics 215 Definitions 216 Perfect Forward Secrecy 218 Aspects of IKE and ISAKMP 218 Modes to Establish Authenticated Key Exchange 218

Main Mode 219 Aggressive Mode 219 Quick Mode and New Group Mode 219 Four Methods Used with Main or Aggressive

Mode 219 Examples of IKE Message Exchanges 220

Phase One: Authenticated with Signatures 221 Phase One: Authenticated with

Public Key Encryption 222 Phase One: Authenticated with a

Revised Mode of Public Key Encryption 223 Phase One: Authenticated With a Pre-Shared Key 225 Phase Two: Quick Mode 225 New Group Mode 225 ISAKMP Informational Exchanges 226

Oakley Groups 226 Messages for a Complete IKE Exchange 226

Phase Two Using Quick Mode 227 IPSec, NAT, and IKE 227 Examples of PKI Vendors 231 Summary 232

CHAPTER 12 Security Operations in a Mobile Network 233

The IS-41-C Specification 233 The IS-41-C Model 234 The Five Security/Privacy Operations 235

CONTENTS xv

Authentication Parameters 236 Authentication of Mobile Station

Registration Procedures 237 The Parameters 237 At the Air Interface 237 On the Network Side 239

Unique Challenge-Response Procedures 240 The Parameters 240 At the Air Interface 240 On the Network Side 242

Authentication of Mobile Station Originating a Call 243 The Parameters 243 At the Air Interface 243 On the Network Side 243

Authentication of Call to a Terminating Mobile Station 245 The Parameters 245 At the Air Interface 247 On the Network Side 247

Updating the Shared Secret Data (SSD) 248 The Parameters 248 At the Air Interface and on the Network Side 249

Summary 250

CHAPTER 13 Follow-ups to This Book 251

APPENDIX A Coding for Prominant Security Functions 255

APPENDIX B Network Address Translation (NAT) 275

ABBREVIATIONS 279

INDEX 281