IBM QRadar: Architecture and Deployment Guide · Introduction to QRadar deployments. The IBM QRadar...

IBM QRadarVersion 7.3.3

Architecture and Deployment Guide

IBM

Note

Before you use this information and the product that it supports, read the information in “Notices” onpage 45.

Product information

This document applies to IBM® QRadar® Security Intelligence Platform V7.3.3 and subsequent releases unlesssuperseded by an updated version of this document.© Copyright International Business Machines Corporation 2016, 2019.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract withIBM Corp.

Contents

Introduction to QRadar deployments......................................................................v

Chapter 1. QRadar architecture overview............................................................... 1QRadar components.................................................................................................................................... 4QRadar events and flows............................................................................................................................. 5

Chapter 2. QRadar deployment overview..............................................................11All-in-One deployment.............................................................................................................................. 12Expanding deployments to add more capacity.........................................................................................13

Adding remote collectors to a deployment......................................................................................... 14Adding processing capacity to an All-in-One deployment..................................................................15

Geographically distributed deployments..................................................................................................17QRadar Vulnerability Manager deployments............................................................................................ 18

QRadar Risk Manager and QRadar Vulnerability Manager ................................................................. 23Forensics and full packet collection..........................................................................................................25

Forwarding packets to QRadar Packet Capture.................................................................................. 31

Chapter 3. Data Nodes and data storage...............................................................35

Chapter 4. App Host.............................................................................................39

Chapter 5. HA deployment overview.................................................................... 41

Chapter 6. Backup strategies............................................................................... 43QRadar data backups................................................................................................................................ 43Retention settings......................................................................................................................................43Backup location......................................................................................................................................... 43

Notices................................................................................................................45Trademarks................................................................................................................................................ 46Terms and conditions for product documentation................................................................................... 46IBM Online Privacy Statement.................................................................................................................. 47General Data Protection Regulation..........................................................................................................47Privacy policy considerations ................................................................................................................... 48

iii

Introduction to QRadar deployments

The IBM QRadar Deployment Guide helps you plan your QRadar installation.

Intended audience

This information is intended for use by security administrators who are responsible for investigating andmanaging network security. To use this guide you must have a knowledge of your corporate networkinfrastructure and networking technologies.

Technical documentation

For information about how to access more technical documentation, technical notes, and release notes,see Accessing IBM Security Documentation Technical Note (http://www.ibm.com/support/docview.wss?rs=0&uid=swg21612861).

Contacting customer support

For information about contacting customer support, see the Support and Download Technical Note(http://www.ibm.com/support/docview.wss?rs=0&uid=swg21612861).

Statement of good security practices

IT system security involves protecting systems and information through prevention, detection andresponse to improper access from within and outside your enterprise. Improper access can result ininformation being altered, destroyed, misappropriated or misused or can result in damage to or misuse ofyour systems, including for use in attacks on others. No IT system or product should be consideredcompletely secure and no single product, service or security measure can be completely effective inpreventing improper use or access. IBM systems, products and services are designed to be part of alawful comprehensive security approach, which will necessarily involve additional operationalprocedures, and may require other systems, products or services to be most effective. IBM DOES NOTWARRANT THAT ANY SYSTEMS, PRODUCTS OR SERVICES ARE IMMUNE FROM, OR WILL MAKE YOURENTERPRISE IMMUNE FROM, THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.

Please Note:

Use of this Program may implicate various laws or regulations, including those related to privacy, dataprotection, employment, and electronic communications and storage. IBM QRadar may be used only forlawful purposes and in a lawful manner. Customer agrees to use this Program pursuant to, and assumesall responsibility for complying with, applicable laws, regulations and policies. Licensee represents that itwill obtain or has obtained any consents, permissions, or licenses required to enable its lawful use of IBMQRadar.

© Copyright IBM Corp. 2016, 2019 v

http://www.ibm.com/support/docview.wss?rs=0&uid=swg21614644

http://www.ibm.com/support/docview.wss?rs=0&uid=swg21612861

vi IBM QRadar: Architecture and Deployment Guide

Chapter 1. QRadar architecture overviewWhen you plan or create your IBM QRadar deployment, it's helpful to have a good awareness of QRadararchitecture to assess how QRadar components might function in your network, and then to plan andcreate your QRadar deployment.

IBM QRadar collects, processes, aggregates, and stores network data in real time. QRadar uses that datato manage network security by providing real-time information and monitoring, alerts and offenses, andresponses to network threats.

IBM QRadar SIEM (Security Information and Event Management) is a modular architecture that providesreal-time visibility of your IT infrastructure, which you can use for threat detection and prioritization. Youcan scale QRadar to meet your log and flow collection, and analysis needs. You can add integratedmodules to your QRadar platform, such as QRadar Risk Manager, QRadar Vulnerability Manager, andQRadar Incident Forensics.

The operation of the QRadar security intelligence platform consists of three layers, and applies to anyQRadar deployment structure, regardless of its size and complexity. The following diagram shows thelayers that make up the QRadar architecture.

© Copyright IBM Corp. 2016, 2019 1

QRadar Console

Router

SwitchUnix serversSyslog

Firewall

1

2

3

Data processing Data storage

Data collection

Data searches

ReportsAlerts and offenses

Windows servers

Graphs

NormalizationParsing

3

2

1

User interface

Flow data

Scan data

Event data

Packet capture for ForensicsConfiguration data for QRadar Risk Manager

Custom rules

QRadar Vulnerability ManagerNessus

Rapid7

Log source

Log source

Other data

Log source

Event Collector

Event Processor Flow Processor

Flow Collector

Proxy serverLog source

Figure 1. QRadar architecture

2 IBM QRadar: Architecture and Deployment Guide

The QRadar architecture functions the same way regardless of the size or number of components in adeployment. The following three layers that are represented in the diagram represent the corefunctionality of any QRadar system.

Data collection

Data collection is the first layer, where data such as events or flows is collected from your network. TheAll-in-One appliance can be used to collect the data directly from your network or you can use collectorssuch as QRadar Event Collectors or QRadar QFlow Collectors to collect event or flow data. The data isparsed and normalized before it passed to the processing layer. When the raw data is parsed, it isnormalized to present it in a structured and usable format.

The core functionality of QRadar SIEM is focused on event data collection, and flow collection.

Event data represents events that occur at a point in time in the user's environment such as user logins,email, VPN connections, firewall denys, proxy connections, and any other events that you might want tolog in your device logs.

Flow data is network activity information or session information between two hosts on a network, whichQRadar translates in to flow records. QRadar translates or normalizes raw data in to IP addresses, ports,byte and packet counts, and other information into flow records, which effectively represents a sessionbetween two hosts. In addition to collecting flow information with a Flow Collector, full packet capture isavailable with the QRadar Incident Forensics component.

Data processing

After data collection, the second layer or data processing layer is where event data and flow data are runthrough the Custom Rules Engine (CRE), which generates offenses and alerts, and then the data is writtento storage.

Event data, and flow data can be processed by an All-in-One appliance without the need for adding EventProcessors or Flow Processors. If the processing capacity of the All-in-One appliance is exceeded, thenyou might need to add Event Processors, Flow Processors or any other processing appliance to handle theadditional requirements. You might also need more storage capacity, which can be handled by addingData Nodes.

Other features such as QRadar Risk Manager (QRM), QRadar Vulnerability Manager (QVM), or QRadarIncident Forensics collect different types of data and provide more functions.

QRadar Risk Manager collects network infrastructure configuration, and provides a map of your networktopology. You can use the data to manage risk by simulating various network scenarios through alteringconfigurations and implementing rules in your network.

Use QRadar Vulnerability Manager to scan your network and process the vulnerability data or manage thevulnerability data that is collected from other scanners such as Nessus, and Rapid7. The vulnerabilitydata that is collected is used to identify various security risks in your network.

Use QRadar Incident Forensics to perform in-depth forensic investigations, and replay full networksessions.

Data searches

In the third or top layer, data that is collected and processed by QRadar is available to users for searches,analysis, reporting, and alerts or offense investigation. Users can search, and manage the security admintasks for their network from the user interface on the QRadar Console.

In an All-in-One system, all data is collected, processed, and stored on the All-in-One appliance.

In distributed environments, the QRadar Console does not perform event and flow processing, or storage.Instead, the QRadar Console is used primarily as the user interface where users can use it for searches,reports, alerts, and investigations.

Chapter 1. QRadar architecture overview 3

QRadar componentsUse IBM QRadar components to scale a QRadar deployment, and to manage data collection andprocessing in distributed networks.

Important: Software versions for all IBM QRadar appliances in a deployment must be same version andfix pack level. Deployments that use different versions of software are not supported becauseenvironments that use mixed versions can cause rules not to fire, offenses not to be created or updated,and errors in search results.

QRadar deployments can include the following components:

QRadar Console

The QRadar Console provides the QRadar user interface, and real-time event and flow views, reports,offenses, asset information, and administrative functions.

In distributed QRadar deployments, use the QRadar Console to manage hosts that include othercomponents.

QRadar Event Collector

The Event Collector collects events from local and remote log sources, and normalizes raw log sourceevents to format them for use by QRadar. The Event Collector bundles or coalesces identical events toconserve system usage and sends the data to the Event Processor.

• Use the QRadar Event Collector 1501 in remote locations with slow WAN links. The Event Collectorappliances do not store events locally. Instead, the appliances collect and parse events before theysend events to an Event Processor appliance for storage.

• The Event Collector can use bandwidth limiters and schedules to send events to the EventProcessor to overcome WAN limitations such as intermittent connectivity.

• The Event Collector is assigned to an EPS license that matches the Event Processor that it isconnected to.

QRadar Event Processor

The Event Processor processes events that are collected from one or more Event Collectorcomponents. The Event Processor processes events by using the Custom Rules Engine (CRE). Ifevents are matched to the CRE custom rules that are predefined on the Console, the Event Processorexecutes the action that is defined for the rule response.

Each Event Processor has local storage, and event data is stored on the processor, or it can be storedon a Data Node.

The processing rate for events is determined by your events per second (EPS) license. If you exceedthe EPS rate, events are buffered and remain in the Event Collector source queues until the ratedrops. However, if you continue to exceed the EPS license rate, and the queue fills up, your systemdrops events, and QRadar issues a warning about exceeding your licensed EPS rate.

When you add an Event Processor to an All-in-One appliance, the event processing function is movedfrom the All-in-One to the Event Processor.

QRadar QFlow Collector

The Flow Collector collects flows by connecting to a SPAN port, or a network TAP. The IBM QRadarQFlow Collector also supports the collection of external flow-based data sources, such as NetFlowfrom routers.

QRadar QFlow Collectors are not designed to be full packet capture systems. For full packet capture,review the QRadar Incident Forensics option. The QRadar QFlow Collector 1310 appliancespecifically, can forward packets to a QRadar Packet Capture appliance, which allows for flowcollection and packet collection from a single packet source.


You can install a QRadar QFlow Collector on your own hardware or use one of the QRadar QFlowCollector appliances.

Restriction: The QRadar Log Manager does not support flow collection or Flow Collectors, which issupported only in QRadar SIEM deployments.

QRadar Flow Processor

The Flow Processor processes flows from one or more QRadar QFlow Collector appliances. The FlowProcessor appliance can also collect external network flows such as NetFlow, J-Flow, and sFlowdirectly from routers in your network. You can use the Flow Processor appliance to scale your QRadardeployment to manage higher flows per minute (FPM) rates. Flow Processors include an on-boardFlow Processor, and internal storage for flow data. When you add a Flow Processor to an All-in-Oneappliance, the processing function is moved from the All-in-One appliance to the Flow Processor.

QRadar Data Node

Data Nodes enable new and existing QRadar deployments to add storage and processing capacity ondemand as required. Data Nodes help to increase the search speed in your deployment by providingmore hardware resources to run search queries on.

QRadar App Host

An App Host is a managed host that is dedicated to running apps. App Hosts provide extra storage,memory, and CPU resources for your apps without impacting the processing capacity of your QRadarConsole. Apps such as User Behavior Analytics with Machine Learning Analytics require moreresources than are currently available on the Console.

For more information about managing QRadar components, see the IBM QRadar Administration Guide.

For more information about QRadar appliance specifications, see the IBM QRadar Hardware Guide.

QRadar events and flowsThe core functions of IBM QRadar SIEM are managing network security by monitoring flows and events.

A significant difference between event and flow data is that an event, which typically is a log of a specificaction such as a user login, or a VPN connection, occurs at a specific time and the event is logged at thattime. A flow is a record of network activity that can last for seconds, minutes, hours, or days, dependingon the activity within the session. For example, a web request might download multiple files such asimages, ads, video, and last for 5 to 10 seconds, or a user who watches a Netflix movie might be in anetwork session that lasts up to a few hours. The flow is a record of network activity between two hosts.

Events

QRadar accepts event logs from log sources that are on your network. A log source is a data source suchas a firewall or intrusion protection system (IPS) that creates an event log.

QRadar accepts events from log sources by using protocols such as syslog, syslog-tcp, and SNMP. QRadarcan also set up outbound connections to retrieve events by using protocols such as SCP, SFTP, FTP, JDBC,Check Point OPSEC, and SMB/CIFS.

Event pipeline

Before you can view and use the event data on the QRadar Console, events are collected from log sourcesand then processed by the Event Processor. A QRadar All-in-One appliance functions as the EventCollector and Event Processor, in addition to fulfilling the role of the QRadar Console.

QRadar can collect events by using a dedicated Event Collector appliance, or by using an All-in-Oneappliance where the event collection service and event processing service runs on the All-in-Oneappliance.

The following diagram shows the layers of the event pipeline.


QRadar Console

Switch

Router

Switch

Unix serversSyslog

Firewalls

1

2

3

Event Processor Streaming

Event Collector

Console

Reports

Offense rules

Windows servers

Graphs

LicensingProtocol

3

2

1

User interface

Event data

Custom Rules Engine

Proxy servers

CoalescingParsing, traffic analysis

Event forwarding

Data storage

Offense managementOffense storage

Magistrate

Log sources

Event Collector component

Event Processor component

Magistrate component

Figure 2. Event pipeline

Event collection

The Event Collector component completes the following functions:Protocol

Collects data from log source protocols such as Syslog, JDBC, OPSEC, Log File, and SNMP.License throttling

Monitors the number of incoming events to the system to manage input queues and EPS licensing.Parsing

Takes the raw events from the source device and parses the fields into a QRadar usable format.Log source traffic analysis and auto discover

Applies the parsed and normalized event data to the possible DSMs that support automaticdiscovery.

CoalescingEvents are parsed and then coalesced based on common attributes across events.

Event forwardingApplies routing rules for the system to forward data to offsite targets, external Syslog systems,JSON systems, and other SIEMs.

When the Event Collector receives the events from log sources such as firewalls, the events areplaced into input queues for processing.

The queue sizes vary based on the protocol or method that is used, and from these queues, the eventsare parsed and normalized. The normalization process involves turning raw data into a format that hasfields such as IP address that QRadar can use.


QRadar recognizes known log sources by the source IP address or host name that is contained in theheader.

QRadar parses and coalesces events from known log sources into records. Events from new orunknown log sources that were not detected in the past are redirected to the traffic analysis (autodetection) engine.

When new log sources are discovered, a configuration request message to add the log source is sentto the QRadar Console. If auto detection is disabled, or you exceed your log source licensed limit, thenew log sources are not added.

Event processingThe Event Processor component completes the following functions:Custom Rules Engine (CRE)

The Custom Rules Engine (CRE) is responsible for processing events that are received by QRadarand comparing them against defined rules, keeping track of systems involved in incidents overtime, generating notifications to users. When events match a rule, a notification is sent from theEvent Processor to the Magistrate on the QRadar Console that a specific event triggered a rule.The Magistrate component on the QRadar Console creates and manages offenses. When rules aretriggered, responses or actions such as notifications, syslog, SNMP, email messages, new events,and offenses are generated.

StreamingSends real-time event data to the QRadar Console when a user is viewing events from the LogActivity tab with Real time (streaming). Streamed events are not provided from the database.

Event storage (Ariel)A time-series database for events where data is stored on a minute by minute basis. Data is storedwhere the event is processed.

The Event Collector sends normalized event data to the Event Processor where the events areprocessed by Custom Rules Engine (CRE). If events are matched to the CRE custom rules that arepredefined on the QRadar Console, the Event Processor executes the action that is defined for the ruleresponse.

Magistrate on the QRadar ConsoleThe Magistrate component completes the following functions:Offense rules

Monitors and acts on offenses, such as generating email notifications.Offense management

Updates active offenses, changes statuses of offenses, and provides user access to offenseinformation from the Offenses tab.

Offense storageWrites offense data to a Postgres database.

The Magistrate Processing Core (MPC) is responsible for correlating offenses with event notificationsfrom multiple Event Processor components. Only the QRadar Console or All-in-One appliance has aMagistrate component.

Flows

QRadar flows represent network activity by normalizing IP addresses, ports, byte and packet counts, andother data, into flow records, which effectively are records of network sessions between two hosts. Thecomponent in QRadar that collects and creates flow information is known as QFlow.

QRadar Flow collection is not full packet capture. For network sessions that span multiple time intervals(minutes), the flow pipeline reports a record at the end of each minute with the current data for metricssuch as bytes, and packets. You might see multiple records (per minute) in QRadar with the same "FirstPacket Time" but the "Last Packet Time" values increment through time.


A flow starts when the Flow Collector detects the first packet that has a unique source IP address,destination IP address, source port, destination port, and other specific protocol options, including802.1q VLAN fields.

Each new packet is evaluated. Counts of bytes and packets are added to the statistical counters in theflow record. At the end of an interval, a status record of the flow is sent to a Flow Processor and statisticalcounters for the flow are reset. A flow ends when no activity for the flow is detected within the configuredtime.

QFlow can process flows from the following internal or external sources:

• External sources are flow sources such as netflow, sflow, jflow. External sources can be sent to adedicated Flow Collector or to a Flow Processor such as the QRadar Flow Processor 1705 appliance.External sources do not require as much CPU processing because every packet is not processed to buildflows. In this configuration, you might have a dedicated Flow Collector and a Flow Processor that bothreceive and create flow data. In smaller environments (less than 50 Mbps), an All-in-One appliancemight handle all the data processing.

• The Flow Collector collects internal flows by connecting to a SPAN port, or a network TAP. The QRadarQFlow Collector 1310 can forward full packets from it's capture card to a packet capture appliance butit does not capture full packets itself.

The following diagram shows the options for collecting flows in a network.


QRadar Console

Switch

Router

FirewallsUnix servers

Flow Collector 1202

Flow Processor

Span port TAP

External flows

Internal flows

ipfixjflowNetflow

Internal flows

Router

Figure 3. QRadar flows


Flow pipeline

The Flow Collector generates flow data from raw packets that are collected from monitor ports such asSPANs, TAPs and monitor sessions, or from external flow sources such as netflow, sflow, jflow. This datais then converted to QRadar flow format and sent down the pipeline for processing.

The Flow Processor runs the following functions:Flow deduplication

Flow deduplication is a process that removes duplicate flows when multiple Flow Collectors providedata to Flow Processors appliances.

Asymmetric recombinationResponsible for combining two sides of each flow when data is provided asymmetrically. This processcan recognize flows from each side and combine them in to one record. However, sometimes only oneside of the flow exists.

License throttlingMonitors the number of incoming flows to the system to manage input queues and licensing.

ForwardingApplies routing rules for the system, such as sending flow data to offsite targets, external Syslogsystems, JSON systems, and other SIEMs.

Flow data passes through the Custom Rules Engine (CRE), and it is correlated against the rules that areconfigured, and an offense can be generated based on this correlation. You view offenses on the Offensestab.


Chapter 2. QRadar deployment overview

IBM QRadar architecture supports deployments of varying sizes and topologies, from a single hostdeployment, where all the software components run on a single system, to multiple hosts, whereappliances such as Event Collectors, and Flow Collectors, Data Nodes, an App Host, Event Processors,and Flow Processors, have specific roles.

The primary focus of the first deployment example is to describe a single All-in-One appliancedeployment for a medium-size company. Later examples describe the deployment options as thecompany expands. The examples describe when to add QRadar components, such as Flow Processors,Event Collectors, and Data Nodes, and when you might need to co-locate specific components.

The requirements for your QRadar deployment depend on the capacity of your chosen deployment toboth process and store all the data that you want to analyze in your network.

Before you plan your deployment, consider the following questions:

• How does your company use the Internet? Do you upload as much as you download? Increased usagecan increase your exposure to potential security issues.

• How many events per second (EPS) and flows per minute (FPM) do you need to monitor?

EPS and FPM license capacity requirements increase as a deployment grows.• How much information do you need to store, and for how long?

The following diagram shows the QRadar components that you can use to collect, process, and storeevent and flow data in your QRadar deployment. An All-in-One appliance includes the data collection,processing, storage, monitoring, searching, reporting, and offense management capabilities.

The Event Collector collects event data from log sources in your network, and then sends the event datato the Event Processor. The Flow Collector collects flow data from network devices such as a switch SPANport, and then sends the data to the Flow Processor. Both processors process the data from the collectorsand provide data to the QRadar Console. The processor appliances can store data but they can also usethe Data Nodes to store data. The QRadar Console appliance is used for monitoring, data searches,reporting, offense management, and administration of your QRadar deployment.


Windows servers

QRadar Console

Switch

Router

Universal Collection Module

Firewalls

Unix serversSyslog

Event Collector Flow Collector

Flow ProcessorEvent Processor

Data Nodes

Proxy servers

Log source

Log source

Log source

Log source

Figure 4. QRadar event and flow components

All-in-One deploymentIn a single host QRadar deployment, you have an All-in-One QRadar appliance that is a single serverwhich collects data, such as syslog event data logs, and Windows events, and also flow data, from yournetwork.

An All-in-One appliance is suitable for a medium-sized company that has low exposure to the Internet, orfor testing and evaluation purposes. Single server deployments are suitable for companies that monitornetwork activity and events such as authentication services and firewall activity.

An All-in-One appliance provides you with the capabilities that you need, up to a specific capacity that isdetermined by your license and the hardware specifications of the system. For example, a QRadar 3105(All-in-One) typically processes up to 5000 EPS (events per second), and 200,000 FPM (flows perminute), whereas a QRadar 3128 (All-in-One) typically processes up to 15,000 EPS and 300,000 FPM.

Manufacturing company deploys a single QRadar server

You are a medium-sized manufacturing company with less than 1000 employees. You deploy a QRadar3105 All-in-One appliance to collect, process, and monitor event and flow data. With that deployment,you can collect up to 5,000 events per second (EPS), and 200,000 flows per minute (FPM).

The following diagram shows an All-in-One appliance, which collects data from event and flow sources,processes the data, and provides a web application where you can search, monitor, and respond tosecurity threats.


Windows servers

QRadar All-in-One

Switch

Router

Universal Collection Module

Firewalls

Switch

Unix serversSyslog

Data searches

Log source

Log source

Log source

Log source

Proxy server

Figure 5. All-in-One deployment

The QRadar All-in-One appliance performs the following tasks:

• Collects event and network flow data, and then normalizes the data in to a data format that QRadar canuse.

• Analyzes and stores the data, and identifies security threats to the company.• Provides access to the QRadar web application.

As your data sources grow, or your processing or storage needs increase, you can add appliances toexpand your deployment.

Expanding deployments to add more capacityYour business might create or expand a deployment beyond an IBM QRadar All-in-One appliance becauseof the lack of processing or data storage capacity, or when you have specific data collection requirements.

The topology and composition of your QRadar deployment are influenced by the capability and capacity ofthat deployment to collect, process, and store all the data that you want to analyze in your network.

To get rough estimates of the events per second (EPS) or flows per minute (FPM) that you need to processin your deployment, use the size of your logs that are collected from firewalls, proxy servers, andWindows boxes.

Chapter 2. QRadar deployment overview 13

Reasons to add event or flow collectors to an All-in-One deployment

You might need to add flow or event collectors to your deployment under these conditions:

• Your data collection requirements exceed the collection capability of the All-in-One appliance.• You must collect events and flows in a different location than where your All-in-One appliance is

installed.• You are monitoring larger, or higher-rate packet-based flow sources that are faster than the 50 Mbps

connection on the All-in-One.

A 3128 All-in-One appliance can collect up to 15,000 events per second (EPS) and 300,000 flows perminute (FPM). If your collection requirements are greater, you might want to add Event Collectors andFlow Collectors to your deployment. For example, you can add a QRadar QFlow Collector 1202, whichcollects up to 3 Gbps.

An All-in-One appliance processes the events and flows that are collected. By adding Event Collectorsand Flow Collectors, you can use the processing that the All-in-One appliance usually does for searchesand other security tasks.

Packet-based flow sources require a Flow Collector that is connected either to a Flow Processor, or to anAll-in-One appliance in deployments where there is no Flow Processor appliance. You can collect externalflow sources, such as NetFlow, or IPFIX, directly on a Flow Processor or All-in-One appliance.

Adding remote collectors to a deploymentAdd QRadar Event Collectors or QRadar Flow Collectors to expand a deployment when you need to collectmore events locally and collect events and flows from a remote location.

For example, you are a manufacturing company that has a QRadar All-in-One deployment and you add e-commerce and a remote sales office. You now must monitor for security threats and are also now subjectto PCI audits.

You hire more employees and the Internet usage changes from mostly downloading to two-way trafficbetween your employees and the Internet. Here are details about your company.

• The current events per second (EPS) license is 1000 EPS.• You want to collect events and flows at the sales office and events from the e-commerce platform.• Event collection from the e-commerce platform requires up to 2000 events-per-second (EPS).• Event collection from the remote sales office requires up to 2000 events-per-second (EPS).• The flows per minute (FPM) license is sufficient to collect flows at the remote office.

You take the following actions:

1. You add the e-commerce platform at your head office, and then you open a remote sales office.2. You install an Event Collector and a Flow Collector at the remote sales office that sends data over the

Internet to the All-in-One appliance at your head office.3. You upgrade your EPS license from 1000 EPS to 5000 EPS to meet the requirements for the extra

events that are collected at the remote office.

The following diagram shows an example deployment of when an Event Collector and a Flow Collector areadded at a remote office.


Local network

QRadar All-in-One

1

2332

1

Flow CollectorEvent Collector

Remote sales office

Unix serverssyslog

Router

TAP

Event data

Log source

Internal flows

External flows

Flow data

Figure 6. Collectors in remote office

In this deployment, the following processes occur:

• At your remote office, the Event Collector collects data from log sources and the Flow Collector collectsdata from routers and switches. The collectors coalesce and normalize the data.

• The collectors compress and send data to the All-in-One appliance over the wide area network.• The All-in-One appliance processes, and stores the data.• Your company monitors network activity by using the QRadar web application for searches, analysis,

reporting, and for managing alerts and offenses.• The All-in-one collects and processes events from the local network.

Adding processing capacity to an All-in-One deploymentAdd Event Processors and Flow Processors to your QRadar deployment to increase processing capacityand increase storage. Adding processors frees up resources on your QRadar Console by moving theprocessing and storage load to dedicated servers.

When you add Event Processors or Flow Processors to an All-in-One appliance the All-in-One acts as aQRadar Console. The processing power on the All-in-One appliance is dedicated to managing andsearching the data that is sent by the processors, and data is now stored on the Event Processors andother storage devices, rather than on the Console.

You typically add Event Processors and Flow Processors to your QRadar deployment for the followingreasons:

• As your deployment grows, the workload exceeds the processing capacity of the All-in-One appliance.• Your security operations center employs more analysts who do more concurrent searches.


• The types of monitored data, and the retention period for that data increases, which increasesprocessing and storage requirements.

• As your security analyst team grows, you require better search performance.

Running multiple concurrent QRadar searches and adding more types of log sources that you monitor,affects the processing performance of your All-in-One appliance. As you increase the number of searchesand the amount of monitored data, add Event Processors and Flow Processors to improve theperformance of your QRadar deployment.

When you scale your QRadar deployment beyond the 15,000 EPS and 300,000 FPM on the most powerfulAll-in-One appliance, you must add processor appliances to process that data.

Example: Adding a QRadar Event Processor to your deployment

You can add a QRadar Event Processor 1628, which collects and processes up to 40,000 EPS. Youincrease your capacity by another 40,000 EPS every time you add a QRadar Event Processor 1628 to yourdeployment. Add a QRadar Flow Processor 1728, which collects and processes up to 1,200,000 FPM.

The QRadar Event Processor 1628 is a collector and a processor. If you have a distributed network, it’s agood practice to add Event Collectors to distribute the load and to free system resources on the EventProcessor.

In the following diagram, processing capacity is added when an Event Processor and a Flow Processor areadded to an QRadar 3128 (All-in-One), and the following changes take place:

• Event and flow processing is moved off the All-in-One appliance to the event and flow processors.• Event processing capacity increases to 40,000 EPS, which includes the 15,000 EPS that was on the All-

in-One.• Flow processing capacity increases to 1,200,000 FPM, which includes the 300,000 FPM that was on the

All-in-One.• Data that is sent by the event and flow collectors is processed and stored on the event and flow

processors.

Figure 7. Adding processing capacity


Search performance is faster when you install Event Processors and Flow Processors on the samenetwork as your QRadar Console.

Adding processors and collectors expands the processing capacity of your QRadar deployment. You canalso increase the storage capacity of your deployment. Your company's data retention needs can increasedue to more traffic or to changes to retention policies. Adding Data Nodes to your deployment expandsyour data storage capacity, and improves search performance.

When to add Collectors to Processors

Add Event Collectors and Flow Collectors to Event Processors and Flow Processors for the same reasonsthat you add collectors to an All-in-One appliance:

• Your data collection requirements exceed the collection capability of your processor.• You must collect events and flows at a different location than where your processor is installed.• You are monitoring packet-based flow sources.

Note: Event Collectors can buffer events, but Flow Collectors can't buffer flows.

Because search performance is improved when processors are installed on the same network as theconsole, adding collectors in remote locations, and then sending that data to the processor, speeds upyour QRadar searches.

Geographically distributed deploymentsIn geographically distributed deployments your IBM QRadar deployment might be impacted byintermittent or poor connectivity to remote data centers. You might also be impacted by local regulations,such as complying with specific state or country regulations to keep data in the place of origin. Both ofthese situations require that you keep collectors on site. If you must keep data in the place of origin, thenyou must keep the processor on site.

For example, your company is growing and this growth not only increases activity in the network, but italso requires that you expand your QRadar deployment to other countries. Data retention laws vary fromcountry to country, so the QRadar deployment must be planned with these regulations in mind.

You note these following conditions:

• Your company must collect event data from one of the office locations that has intermittentconnectivity.

• Your company must comply with data retention regulations in the countries where data is collected. Forexample, if Germany requires that data remains in-country, then that data must not be stored outsidethat country.


QRadar Console

Switch

Router

Firewalls

3

German data center

Windows servers

3



French data center

Event Collector

Flow Collector

Database server



Remote data centerStore and Forward Event Collector

Poor connectivity

Company Headquarters

Log source

Remote search

Log source

Log source

Event and flow data

Event and flow data

Event data

Event and flow data

Figure 8. Geographically Distributed Deployment

In the geographically distributed deployment, the following processes occur:

• Your company installs collectors and processors in the German data center to comply with local datalaws.

• In the French data center, your company installs collectors, so that data is sent by the collectors to thehead office, which is processed and stored at the head office. Search speeds are increased by havingthe processor appliances on the same high-speed network segment as the QRadar Console.

• Your company adds a store-and-forward Event Collector that has scheduled and rate-limited forwardingconnections in the remote data center. The scheduled and rate-limited connections compensate forintermittent network connectivity and ensures that the requirement for more bandwidth is avoidedduring regular business hours.

If you’re constantly searching for data on a remote processor, it’s better to have that processor on thesame high-speed network segment as the QRadar Console. If the bandwidth between the QRadarConsole and remote processor is not good, you might experience latency with your searches, especiallywhen you're doing multiple concurrent searches.

QRadar Vulnerability Manager deploymentsLocate and manage the vulnerabilities in your network by deploying IBM QRadar Vulnerability Manager.Enhance your network security by integrating add-on features such as IBM BigFix® and IBM SecuritySiteProtector.

IBM QRadar Vulnerability Manager discovers vulnerabilities on your network devices, applications, andsoftware adds context to the vulnerabilities, prioritizes asset risk in your network, and supports theremediation of discovered vulnerabilities.


You can integrate QRadar Risk Manager for added protection, which provides network topology, activeattack paths and high-risk assets risk-score adjustment on assets based on policy compliance. QRadarVulnerability Manager and QRadar Risk Manager are combined into one offering and both are enabledthrough a single base license.

Depending on the product that you install, and whether you upgrade IBM QRadar or install a new system,the Vulnerabilities tab might not be displayed. Access IBM QRadar Vulnerability Manager by using theVulnerabilities tab. If you install IBM QRadar SIEM, the Vulnerabilities tab is enabled by default with atemporary license key. If you install QRadar Log Manager, the Vulnerabilities tab is not enabled. You canuse the Try it Out option to try out QRadar Vulnerability Manager for 30 days. You can purchase thelicense for QRadar Vulnerability Manager separately and enable it by using a license key. For moreinformation about upgrading, see the IBM QRadar Upgrade Guide.

QRadar Vulnerability Manager components

The following information describes the QRadar Vulnerability Manager Processor.

• The scan processor is responsible for the scheduling and managing scans, and delegating work to thescanners that might be distributed throughout your network.

• You can have only one scan processor in a QRadar deployment.• When you install and license QRadar Vulnerability Manager on an All-in-One system, a vulnerability

processor is automatically deployed on your QRadar Console and includes a scanning component.• The vulnerability processor provides a scanning component by default. If required, you can move the

vulnerability processor to a different managed host in your deployment.• If you add a 600 managed host appliance, and QRadar Vulnerability Manager is used for the first time,

then the scan processor is assigned to the 600 managed host appliance.• The scanning processor is governed by the processing license, which determines the maximum number

of assets that can be processed by QRadar Vulnerability Manager.• The scan processor can run on the QRadar Console or a managed host.

The following information describes the QRadar Vulnerability Manager scanner.

• You can deploy a scanner on a virtual machine or as software only.• You can deploy a QRadar Vulnerability Manager scanner dedicated scanner appliance, which is a 610

appliance.• You can deploy a scanner on a QRadar Console or on the following managed hosts: Flow Collector, Flow

Processor Event Collector, Event Processor, or Data Node.• The number of assets that you can scan with a scanner is determined by the scanner capacity and is not

impacted by licensing.

Components and scan process

Scan jobs are completed by a processor and a scanner component. The following diagram shows the scancomponents and the processes that run.


QRadar Console

1

2

3Asset database

exporter

PostgresSQL

Result

Processor

1

3User interface

Scan tools

Result writer

SchedulerdispatcherScan tool

dispatcherResult

2

1

3

4 5

678

9

Processor Scanner

QRadar Vulnerability Manager

Figure 9. Scan components and process

The following list describes the steps in the scan process:

1. You create a scan job by specifying parameters such as IP addresses of assets, type of scan, andrequired credentials for authenticated scans.

2. The scan job is accepted by the processor, logged, and added to the database along with schedulinginformation to determine when the job runs.

3. The scheduler component manages the scheduling of scans. When the scheduler initiates a scan, itdetermines the list of tools that are required and queues them for invocation, and then the tools areassigned to the relevant scanner.

4. Scanners poll the scan processor continuously for scan tools that it must run by sending a uniquescanner ID. When the scheduler has queued tools that are relevant to the specific scanner the toolsare sent to the scanner for invocation.

QRadar Vulnerability Manager uses an attack tree methodology to manage scans and to determinewhich tools are launched. The phases are: asset discovery, port/service discovery, service scan, andpatch scan.

5. The dispatcher runs and manages each scan tool in the list. For each tool that is run, the dispatchersends a message to the processor that indicates when a scan tool starts and finishes.

6. The output from the scan tool is read by the result writer, which then passes these results back to theprocessor.

7. The result dispatcher processes the raw results from the scan tools and records them into thePostgres database.

8. The result exporter finds completed scans in the processor database and exports the results to theQRadar Console.

9. The exported results are added to the QRadar database where users can view and manage the scanresults.


All-in-one deployment

You can run QRadar Vulnerability Manager from an All-in-one system, where the scanning and processingfunctions are on the Console. The following information describes what you can do with a basic setup:

• Scan up to 255 assets.• Unlimited discovery scans.• Use hosted scanner for DMZ scanning.• Manage scan data from third-party scanners that are integrated with QRadar.• Deploy a scanner on any managed host.• Deploy unlimited stand-alone software or virtual scanners.

Expanding a deployment

As your deployment grows, you might need to move the processing function off the QRadar Console tofree up resources, and you might want to deploy scanners closer to your assets.

The following list describes reasons to add scanners to your deployment:

• To scan assets in a different geographic region than the QRadar Vulnerability Manager processor.• If you want to scan many assets concurrently within a short time frame.• You might want to add a scanner to avoid scanning through a firewall that is a log source. You might also

consider adding the scanner directly to the network by adding an interface on the scanner host that by-passes the firewall.

The following diagram shows a scanning deployment with external scanning and scanners deployed onmanaged hosts.


1

3

13

Processor

Scanner

Hosted scanner

Switch

ExternalDMZ servers

Internet gateway

Internal DMZ Firewall

Event Collectorand scanner

Flow Collectorand scanner

10.1.0.0/16

10.2.0.0/16

10.3.0.0/16

QRadar Console

HTTPS

HTTPSHTTPS

DMZ scan

HTTPS

Figure 10. Scanning deployment


DMZ hosted scanner

A hosted scanner scans your DMZ from the internet by using your public IP address. If you want to scanthe assets in the DMZ for vulnerabilities, you do not need to deploy a scanner in your DMZ. You mustconfigure QRadar Vulnerability Manager with a hosted IBM scanner that is located outside your network.For more information, see the IBM QRadar Vulnerability Manager User Guide.

QRadar Vulnerability Manager integrations

IBM QRadar Vulnerability Manager integrates with IBM BigFix to help you filter and prioritize thevulnerabilities that can be fixed. BigFix provides shared visibility and control between IT operations andsecurity. BigFix applies Fixlets to high priority vulnerabilities that are identified and sent by QRadarVulnerability Manager to BigFix. Fixlets are packages that you deploy to your assets or endpoints toremediate specific vulnerabilities.

QRadar Vulnerability Manager integrates with IBM Security SiteProtector to help direct intrusionprevention system (IPS) policy. When you configure IBM Security SiteProtector, the vulnerabilities thatare detected by scans are automatically forwarded to IBM Security SiteProtector. IBM SecuritySiteProtector receives vulnerability data from QRadar Vulnerability Manager scans that are run only afterthe integration is configured. Connecting to IBM Security SiteProtector.

Third-party scanners

QRadar Vulnerability Manager delivers an effective vulnerability management platform, regardless of thesource of the scan data. QRadar Vulnerability Manager integrates seamlessly with third-party scannerssuch as Nessus, nCircle, and Rapid 7.

You require QRadar Vulnerability Manager scanning to get the following options:

• Event driven and on-demand scanning• Asset database and watchlist based scanning• Scanning from existing QRadar appliances and managed hosts• Detection of newly published vulnerabilities that are not present in any scan results

You require QRadar Risk Manager to get the following options:

• Asset, vulnerability, and traffic-based vulnerability management• Adjusted vulnerability scores and context aware risk scoring.

QRadar Risk Manager and QRadar Vulnerability ManagerEnhance your network security by integrating IBM QRadar Risk Manager with IBM QRadar VulnerabilityManager. Data sources, such as scan data, enable QRadar Risk Manager to identify security, policy, andcompliance risks in your network and calculate the probability of risk exploitation.

QRadar Vulnerability Manager and QRadar Risk Manager are combined into one offering and both areenabled through a single base license.

Add a QRadar Risk Manager 700 appliance to get the following capabilities:

• Compliance assessment• Risk policies that are based on vulnerability data and risk scores that help you quickly identify high-risk

vulnerabilities.• Visibility into potential exploit paths from potential threats and untrusted networks through the network

topology view.• Risk policy-based filtering.• Topology visualization• False positives reduction in vulnerability assessments.• Visibility into what vulnerabilities are blocked by firewalls and Intrusion Prevention Systems (IPS).


QRadar Risk Manager applianceInstall QRadar Risk Manager separately on a QRadar Risk Manager 700 appliance.

You must install IBM QRadar Console before you set up and configure the QRadar Risk Managerappliance. It is a good practice to install QRadar and QRadar Risk Manager on the same network switch.

You require only one QRadar Risk Manager appliance per deployment.

The following diagram shows a deployment that has a scanner and QRadar Risk Manager.

1

31

3

Scanner

QRadar Console

Risk Manager

Scan

Device configurationsRisk

Network topology

Figure 11. Scanning deployment with Risk Manager

Use Risk Manager to complete the following tasks:

• Centralized risk management.• View and filter your network topology• Import and compare device configurations• View connections between network devices.• Search firewall rules.• View existing rules and the event count for triggered rules.• Search devices and paths• Query network connections• Simulate the possible outcomes of updating device configurations.• Monitor and audit your network to ensure compliance.• Simulate threats or attacks against a virtual model.• Search for vulnerabilities.


Forensics and full packet collectionUse IBM QRadar Incident Forensics in your deployment to retrace the step-by-step actions of a potentialattacker, and conduct an in-depth forensics investigation of suspected malicious network securityincidents.

QRadar Incident Forensics reconstructs raw network data that is related to a security incident back intoits original form.

QRadar Incident Forensics integrates with the IBM QRadar Security Intelligence Platform and iscompatible with many third-party packet capture offerings.

QRadar Incident Forensics offers an optional QRadar Packet Capture appliance to store and manage datathat is used by QRadar Incident Forensics if no other network packet capture (PCAP) device is deployed.Any number of these appliances can be installed as a tap on a network or sub-network to collect the rawpacket data.

QRadar Packet Capture components

The following components can be included in a QRadar deployment:

QRadar ConsoleProvides the QRadar product user interface. In distributed deployments, use the QRadar Console tomanage multiple QRadar Incident Forensics Processor hosts.

QRadar Incident Forensics Processor

Provides the QRadar Incident Forensics product interface. The interface delivers tools to retrace thestep-by-step actions of cyber criminals, reconstruct raw network data that is related to a securityincident, search across available unstructured data, and visually reconstruct sessions and events.

You must add QRadar Incident Forensics Processor as a managed host before you can use thesecurity intelligence forensics capability.

QRadar Incident Forensics StandaloneProvides the QRadar Incident Forensics product user interface. Installing QRadar Incident ForensicsStandalone provides the tools that you need to do forensics investigations. Only forensicsinvestigative and the related administrative functions are available.

QRadar Packet Capture

You can install an optional QRadar Packet Capture appliance. If no other network packet capture(PCAP) device is deployed, you can use this appliance to store data that is used by QRadar IncidentForensics. You can install any number of these appliances as a network tap or subnetwork to collectthe raw packet data.

If no packet capture device is attached, you can manually upload the packet capture files in the userinterface or by using FTP.

Depending on your network and packet capture requirements, you can connect up to five packetcapture devices to a QRadar Incident Forensics appliance.

QRadar Packet Capture Data Node appliances

For extra storage capacity, you can connect up to two QRadar Packet Capture Data Node appliancesto each QRadar Packet Capture master system.

All-in-One deployment

In standalone or all-in-one deployments, you install the IBM QRadar Incident Forensics Standalonesoftware. These single appliance deployments are similar to installing the QRadar Console and QRadarIncident Forensics managed host on one appliance, but without log management, network activitymonitoring, or other security intelligence features. For a stand-alone network forensics solution, installthe QRadar Incident Forensics Standalone in small to midsize deployments.


The following diagram shows a basic QRadar Incident Forensics All-in-One deployment.

QRadar Console

Switch

Router1

33

22

TAP

Packet Capture

Figure 12. All-in-one deployment


Distributed deployment

In a distributed deployment, you can have the following three appliances:

• QRadar Console• QRadar Packet Capture managed host (QRadar Packet Capture processor)• QRadar Packet Capture (optional)

Software versions for all IBM QRadar appliances in a deployment must be the same version and fix level.Deployments that use different versions of software are not supported.

The following diagram shows a QRadar Incident Forensics distributed deployment.


QRadar Console

Router

33

Data Nodes

TAP

Forensics Processor


Forensics Processor

QRadar NetworkPacket Capture


TAP

Figure 13. Distributed deployment


The following diagram shows packet forwarding from a IBM QRadar QFlow Collector 1310 with a 10GNapatech network card to a QRadar Packet Capture appliance.

The QRadar QFlow Collector uses a dedicated Napatech monitoring card to copy incoming packets fromone port on the card to a second port that connects to a IBM QRadar Packet Capture appliance.


QRadar Console

Switch

1

33

22

TAP

QFlow Collector 1310 appliance

Napatech 10G

Packet Capture

Network card

Forensics Processor

Figure 14. Packet forwarding


Forwarding packets to QRadar Packet CaptureYou can monitor network traffic by sending raw data packets to a IBM QRadar QFlow Collector 1310appliance. The QRadar QFlow Collector uses a dedicated Napatech monitoring card to copy incomingpackets from one port on the card to a second port that connects to a IBM QRadar Packet Captureappliance.

If you already have a QRadar QFlow Collector 1310 with a 10G Napatech network card, you can mirrorthe traffic to QRadar Packet Capture.

As shown in the following diagram, if you already have a QRadar QFlow Collector 1310 with a 10GNapatech network card, you can mirror the traffic to QRadar Packet Capture.


QRadar Console

Switch

1

33

22

TAP

QFlow Collector 1310 appliance

Napatech 10G

Packet Capture

Network card

Forensics Processor

Figure 15. Packet data forwarding from a QRadar QFlow Collector to QRadar Packet Capture by using theNapatech card


Before you begin

Ensure that the following hardware is set up in your environment:

• You attached the cable to port 1 of the Napatech card on the QRadar QFlow Collector 1310 appliance.• You attached the cable that is connected to port 2 of the Napatech card, which is the forwarding port, to

the QRadar Packet Capture appliance.• Verify layer 2 connectivity by checking for link lights on both appliances.

Procedure

1. Using SSH from your IBM QRadar Console, log in to QRadar QFlow Collector as the root user. On theQRadar QFlow Collector appliance, edit the following file.

/opt/qradar/init/apply_tunings

a) Locate the following line, which is around line 137.

apply_multithread_qflow_changes(){ APPLIANCEID=`$NVABIN/myver -a` if [ "$APPLIANCEID" == "1310" ]; then MODELNUM=$(/opt/napatech/bin/AdapterInfo 2>&1 | grep "Active FPGA Image" | cut -d'-' -f2) if [ "$MODELNUM" == "9220" ]; then..

b) In the AppendToConf lines that follow the code in the preceding step, add these lines:

AppendToConf SV_NAPATECH_FORWARD YES AppendToConf SV_NAPATECH_FORWARD_INTERFACE_SRCDST "0:1"

These statements enable packet forwarding, and forward packets from port 0 to port 1.c) Ensure that multithreading is enabled, by verifying that the following line is in the /opt/qradar/conf/nva.conffile.

MULTI_THREAD_ON=YES2. Run the apply_tunings script to update the configuration files on the QRadar QFlow Collector, by

typing the following command:

./apply_tunings restart3. Restart IBM QRadar services by typing the following command:

systemctl restart hostcontext4. Optional: Verify that your Napatech card is receiving and transmitting data.

a) To verify that the Napatech card is receiving data, type the following command:

/opt/napatech/bin/Statistics -dec -interactive

The "RX" packet and byte statistics increment if the card is receiving data.b) To verify that the Napatech card is transmitting data, type the following command:

/opt/napatech/bin/Statistics -dec -interactive

The "TX" statistics increment if the card is transmitting data.5. Optional: Verify that your QRadar Packet Capture is receiving packets from your QRadar QFlow

Collector appliance.a) Using SSH from your QRadar Console, log in to your QRadar Packet Capture appliance as root on

port 4477.b) Verify that the QRadar Packet Capture appliance is receiving packets by typing the following

command:

watch -d cat /var/www/html/statisdata/int0.txt


The int0.txt file updates as data flows into your QRadar Packet Capture appliance.

For more information about packet capture, see the IBM QRadar Packet Capture Quick ReferenceGuide.


Chapter 3. Data Nodes and data storageIBM QRadar processor appliances and All-in-One appliances can store data but many companies requirethe stand-alone storage and processing capabilities of the Data Node to handle specific storagerequirements and to help with implementing data retention policies. Many companies are impacted byregulations and laws that mandate keeping data records for specific periods.

Data Node information

The following list describes information about Data Nodes:

• Data Nodes add storage and processing capacity.• Data Nodes are plug-n-play and can be added to a deployment at any time.• Data Nodes integrate seamlessly with existing deployments.• Use Data Nodes to reduce the processing load on processor appliances by removing the data storage

processing load from the processor.• Users can scale storage and processing power independently of data collection.• As of QRadar V.7.2.7, a new data format with native data compression is used. Data is compressed in

memory and is written out to disk in a proprietary binary compressed format. The new data formatenables a better search performance and a more efficient use of system resources than the previousdata format. The previous data format did not have a native built-in compression in older versions ofQRadar.

The following diagram shows an example of some uses for Data Nodes in a deployment.

QRadar Console

2Data processing

Data collection

Data searches

Storage and retention

Data Nodes

Data retention periods2 years 6 months 90 daysSearch

UK USFrance

Performance

Data centers

New York

Data category

Events Flows Events EventsFlows Flows

EventsEventsEvents Flows FlowsFlows

Figure 16. Using Data Node appliances to manage your data storage

The following list describes the different elements that you need to consider when you deploy DataNodes.

Data clustering

Data Nodes add storage capacity to a deployment, and also improve performance by distributing datathat is collected across multiple storage volumes. When the data is searched, multiple hosts, or acluster does the search. The cluster can improve search performance, but doesn't require you to addmultiple event processors. Data Nodes multiply the storage for each processor.


Note: You can connect a Data Node to only one processor at a time, but a processor can supportmultiple Data Nodes.

Deployment considerations

Keep the following information in mind as you set up Data Nodes in a deployment.

• Data Nodes are available with QRadar V7.2.2 and later.• Data Nodes perform similar search and analytic functions as event and flow processors in a QRadar

deployment.

The operational speed on a cluster is affected by the slowest member of a cluster. Data Nodesystem performance improves if Data Nodes are sized similarly to the Event Processors and FlowProcessors in a deployment. To facilitate similar sizing between Data Nodes and event and flowprocessors, Data Nodes are available on XX05, XX28, and XX29 core appliances.

• Data Nodes are available in three formats: software (on your own hardware), physical, andappliances. You can mix the formats in a single cluster.

Bandwidth and latency

Ensure that you have a 1 Gbps link and less than 10 ms latency between hosts in the cluster.Searches that yield many results require more bandwidth.

Appliance compatibilityData Nodes are compatible with all existing QRadar appliances that have an Event Processor or FlowProcessor component, including All-In-One appliances. Data Nodes are not compatible with QRadarIncident Forensics PCAP appliances.

Data Nodes support high availability (HA).

Installation of Data Nodes

Data Nodes use standard TCP/IP networking, and do not require proprietary or specializedinterconnect hardware.

Install each Data Node that you want to add to your deployment the same as you would install anyother QRadar appliance. Associate Data Nodes with either an event or flow processor. For moreinformation, see Configuring a managed host in the IBM QRadar Administration Guide.

You can attach multiple Data Nodes to a single Event Processor or Flow Processor in a many-to-oneconfiguration.

When you deploy high availability (HA) pairs with Data Node appliances, install, deploy, and rebalancedata with the HA appliances before you synchronize the HA pair. The combined effect of the datarebalancing and the replication process that is utilized for HA results in significant performancedegradation. If HA is set up on appliances to which Data Nodes are being introduced, then disconnectHA on the appliances and then reconnect it when the rebalance of the cluster is complete.

Decommissioning Data Nodes

Use the System and License Management window to remove Data Nodes from your deployment, aswith any other QRadar appliance. Decommissioning does not erase data on the host, nor does it movethe data to your other appliances. If you need to retain access to the data that was on the Data Nodes,you must identify a location to move that data to.

Data Rebalancing

Adding a Data Node to a cluster distributes data to each Data Node. If it is possible, data rebalancingtries to maintain the same percentage of available space on each Data Node. New Data Nodes addedto a cluster initiate more rebalancing from cluster event and flow processors to achieve efficient diskusage on the newly added Data Node appliances.

Starting with QRadar V7.2.3, data rebalancing is automatic and concurrent with other cluster activity,such as queries and data collection. No downtime is experienced during data rebalancing.


Data Nodes offer no performance improvement in the cluster until data rebalancing is complete.Rebalancing can cause minor performance degradation during search operations, but data collectionand processing continue unaffected.

Note: Encrypted data transmission between Data Nodes and Event Processors is not supported. Thefollowing firewall ports must be opened for Data Node communication with the Event Processor:

• Port 32006 between Data Nodes and the Event Processor appliance.• Port 32011 between Data Nodes and the Console's Event Processor.

Management and Operations

Data Nodes are self-managed and require no regular user intervention to maintain normal operation.QRadar manages activities, such as data backups, high availability, and retention policies, for allhosts, including Data Node appliances.

Data Node failure

If a Data Node fails, the remaining members of the cluster continue to process data.

When the failed Data Node returns to service, data rebalancing can occur to maintain proper datadistribution in the cluster, and then normal processing resumes. During the downtime, data on thefailed Data Node is unavailable, and I/O errors that occur appear in search results from the log andnetwork activity viewers in the QRadar user interface.

For catastrophic failures that require appliance replacement or the reinstallation of QRadar,decommission Data Nodes from the deployment and replace them using standard installation steps.Copy any data that is not lost in the failure to the new Data Node before you deploy. The rebalancingalgorithm accounts for data that exists on a Data Node, and shuffles only data that was collectedduring the failure.

For Data Nodes deployed with an HA pair, a hardware failure causes a failover, and operationscontinue to function normally.

SAN Overview

To increase the amount of storage space on your appliance, you can move a portion of your data to anoffboard storage device. You can move your /store, /store/ariel, or /store/backup file systems.

Multiple methods are available for adding external storage, including iSCSI, Fiber Channel, and NFS(Network File System). You must use iSCSI or Fiber Channel to store data that is accessible andsearchable in the UI, such as the /store/ariel directory, and reserve the use of NFS for data backupsonly.

Moving the /store file system to an external device might affect QRadar performance.

After migration, all data I/O to the /store file system is no longer done on the local disk. Before youmove your QRadar data to an external storage device, you must consider the following information:

• Searches that are marked as saved are also in the /transient directory. If you experience a local diskfailure, these searches are not saved.

• A transient partition that exists before you move your data is likely to remain in existence after themove, and it can be mounted on an iSCSI, or Fiber Channel storage mount.

For more information about offboard storage, see the IBM QRadar Security Intelligence Offboard StorageGuide.

Chapter 3. Data Nodes and data storage 37

Chapter 4. App HostAn App Host is a managed host that is dedicated to running apps. App Hosts provide extra storage,memory, and CPU resources for your apps without impacting the processing capacity of your QRadarConsole. Apps such as User Behavior Analytics with Machine Learning Analytics require more resourcesthan are currently available on the Console.

App Host information

The following list describes information about App Hosts:

• The App Host was added in QRadar V7.3.2 to replace the deprecated App Node.• You cannot upgrade to V7.3.2 or later with an App Node in your deployment.• You can only have one App Host per deployment.• You can run all of your apps on an App Host, or on the Console. It's not possible to run some apps on

the App Host and some others on the Console.• Port 5000 must be open on your Console.

App Host specifications

The following table shows the minimum requirements and suggested specifications for an App Host.

Note: *The suggested specifications for medium and large sized deployments haven't been tested. If youare using some of the larger apps, such as the Pulse Dashboard or User Behavior Analytics with MachineLearning, the minimum requirements are probably insufficient. Consider upgrading your deploymentenvironment.

Table 1. App Host specifications

CPU cores RAM Disk Space Description

Small 4 12 GB 256 GB Minimum requirements for an App Host. You canrun most apps with the minimum requirements,but not larger apps such as QRadar DNS Analyzerand User Behavior Analytics with MachineLearning.

Medium 12 or more 64 GB ormore

500 GB ormore

*You can run all apps that exist today, but thisspecification does not give you room for futureapps.

Large 24 or more 128 GB ormore

1 TB ormore

*You can run all apps that exist today and youwould have room for future apps.


Chapter 5. HA deployment overviewImplement high availability (HA) in your IBM QRadar deployment to keep QRadar functions running, ifthere is a hardware or software failure in your deployment.

By using high availability, you can continue to collect, store, and process event and flow data, if anyfailures occur.

To enable HA, QRadar connects a primary HA host with a secondary HA host to create an HA cluster.

The following diagram shows a basic HA setup.

QRadar Console

1

33

22

Event Processor

Flow Collector

Events and Flows

Event CollectorFlow CollectorEvent Collector

QRadar ConsolePrimary Secondary

Primary Primary

PrimaryPrimary

Secondary

SecondarySecondary

Secondary

Event Processor Flow ProcessorFlow Processor

Network

HA overview

In an HA deployment, you install and configure a second appliance that takes over the role of the device,if the primary appliance fails in one of the following scenarios:

• A power supply failure• A network failure that is detected by network connectivity tests• An operating system malfunction that delays or stops the heartbeat ping tests• A complete RAID failure on the primary HA host• A manual failover• A management interface failure on the primary HA host

For best performance in large deployments it is strongly recommended to use a 10 Gbps interface foryour HA Crossover. Using a 10 Gbps interface reduces the time needed for system synchronization andensures optimal performance of the pair. If you do not have a 10 Gbps interface available considerbonding multiple 1 Gbps interfaces for crossover.


For more information about HA, see the IBM Security QRadar SIEM High Availability Guide.


Chapter 6. Backup strategiesBack up your business critical information to safeguard against loss of that data. Different types of datarequire different backup strategies.

QRadar data backupsData classification is an important consideration for backup strategies for the following reasons:

• Data such as personal identity information (PII) needs to be stored securely, and might need to be keptseparate from bulk data backups, and retained for longer periods for compliance reasons.

• Keep QRadar system configuration data separate from your security data such as events and flows. It issafer to keep the system configuration separate and easier to restore this data if it stored separately.

• Store data such as PCI data in a separate location so that you can easily access this data when auditorswant to see it.

• Think about types of data and retention periods when you develop your backup strategies.• You can back up some types of data more frequently than others and you can use offsite storage for

some data to insure against data loss.

Retention settingsThe default setting for QRadar backup retention is 7 days. You can also do an on-demand backup afteryou make major configuration changes. You can give this on-demand backup a descriptive name to easilyfind your changes if you need to return to this configuration.

Scheduled backups overwrite older scheduled backups. On-demand backups are kept indefinitely. Afterthe QRadar backup volume reaches 75% of its capacity, scheduled backups no longer run.

Backup locationThe backup location is also a significant consideration when you deploy QRadar. If your backups remainon a host, and that host fails, then all backup data is lost.

You can either create your backups on an external system, or copy backups to an external system.

Store copies of important data locally and remotely for added data security.


Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries.Consult your local IBM representative for information on the products and services currently available inyour area. Any reference to an IBM product, program, or service is not intended to state or imply that onlythat IBM product, program, or service may be used. Any functionally equivalent product, program, orservice that does not infringe any IBM intellectual property right may be used instead. However, it is theuser's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in thisdocument. The furnishing of this document does not grant you any license to these patents. You can sendlicense inquiries, in writing, to:

IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A.

For license inquiries regarding double-byte character set (DBCS) information, contact the IBM IntellectualProperty Department in your country or send inquiries, in writing, to:

Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd. 19-21, Nihonbashi-Hakozakicho, Chuo-kuTokyo 103-8510, Japan

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS"WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR APARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties incertain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodicallymade to the information herein; these changes will be incorporated in new editions of the publication.IBM may make improvements and/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not inany manner serve as an endorsement of those websites. The materials at those websites are not part ofthe materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way it believes appropriate withoutincurring any obligation to you.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) theexchange of information between independently created programs and other programs (including thisone) and (ii) the mutual use of the information which has been exchanged, should contact:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

Such information may be available, subject to appropriate terms and conditions, including in some cases,payment of a fee.


The licensed program described in this document and all licensed material available for it are provided byIBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or anyequivalent agreement between us.

The performance data and client examples cited are presented for illustrative purposes only. Actualperformance results may vary depending on specific configurations and operating conditions..

Information concerning non-IBM products was obtained from the suppliers of those products, theirpublished announcements or other publicly available sources. IBM has not tested those products andcannot confirm the accuracy of performance, compatibility or any other claims related to non-IBMproducts. Questions on the capabilities of non-IBM products should be addressed to the suppliers ofthose products.

Statements regarding IBM's future direction or intent are subject to change or withdrawal without notice,and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subject to change withoutnotice. Dealer prices may vary.

This information contains examples of data and reports used in daily business operations. To illustratethem as completely as possible, the examples include the names of individuals, companies, brands, andproducts. All of these names are fictitious and any similarity to actual people or business enterprises isentirely coincidental.

TrademarksIBM, the IBM logo, and ibm.com® are trademarks or registered trademarks of International BusinessMachines Corp., registered in many jurisdictions worldwide. Other product and service names might betrademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at"Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Java™ and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/orits affiliates.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in theUnited States, other countries, or both.

Terms and conditions for product documentationPermissions for the use of these publications are granted subject to the following terms and conditions.

Applicability

These terms and conditions are in addition to any terms of use for the IBM website.

Personal use

You may reproduce these publications for your personal, noncommercial use provided that all proprietarynotices are preserved. You may not distribute, display or make derivative work of these publications, orany portion thereof, without the express consent of IBM.

46 Notices

http://www.ibm.com/legal/copytrade.shtml

Commercial use

You may reproduce, distribute and display these publications solely within your enterprise provided thatall proprietary notices are preserved. You may not make derivative works of these publications, orreproduce, distribute or display these publications or any portion thereof outside your enterprise, withoutthe express consent of IBM.

Rights

Except as expressly granted in this permission, no other permissions, licenses or rights are granted, eitherexpress or implied, to the publications or any information, data, software or other intellectual propertycontained therein.

IBM reserves the right to withdraw the permissions granted herein whenever, in its discretion, the use ofthe publications is detrimental to its interest or, as determined by IBM, the above instructions are notbeing properly followed.

You may not download, export or re-export this information except in full compliance with all applicablelaws and regulations, including all United States export laws and regulations.

IBM MAKES NO GUARANTEE ABOUT THE CONTENT OF THESE PUBLICATIONS. THE PUBLICATIONS AREPROVIDED "AS-IS" AND WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED,INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE.

IBM Online Privacy StatementIBM Software products, including software as a service solutions, (“Software Offerings”) may use cookiesor other technologies to collect product usage information, to help improve the end user experience, totailor interactions with the end user or for other purposes. In many cases no personally identifiableinformation is collected by the Software Offerings. Some of our Software Offerings can help enable you tocollect personally identifiable information. If this Software Offering uses cookies to collect personallyidentifiable information, specific information about this offering’s use of cookies is set forth below.

Depending upon the configurations deployed, this Software Offering may use session cookies that collecteach user’s session id for purposes of session management and authentication. These cookies can bedisabled, but disabling them will also eliminate the functionality they enable.

If the configurations deployed for this Software Offering provide you as customer the ability to collectpersonally identifiable information from end users via cookies and other technologies, you should seekyour own legal advice about any laws applicable to such data collection, including any requirements fornotice and consent.

For more information about the use of various technologies, including cookies, for these purposes, SeeIBM’s Privacy Policy at http://www.ibm.com/privacy and IBM’s Online Privacy Statement at http://www.ibm.com/privacy/details the section entitled “Cookies, Web Beacons and Other Technologies” andthe “IBM Software Products and Software-as-a-Service Privacy Statement” at http://www.ibm.com/software/info/product-privacy.

General Data Protection RegulationClients are responsible for ensuring their own compliance with various laws and regulations, including theEuropean Union General Data Protection Regulation. Clients are solely responsible for obtaining advice ofcompetent legal counsel as to the identification and interpretation of any relevant laws and regulationsthat may affect the clients’ business and any actions the clients may need to take to comply with suchlaws and regulations. The products, services, and other capabilities described herein are not suitable forall client situations and may have restricted availability. IBM does not provide legal, accounting orauditing advice or represent or warrant that its services or products will ensure that clients are incompliance with any law or regulation.

Notices 47

http://www.ibm.com/privacy

http://www.ibm.com/privacy/details/us/en/


http://www.ibm.com/software/info/product-privacy


Learn more about the IBM GDPR readiness journey and our GDPR capabilities and Offerings here: https://ibm.com/gdpr

Privacy policy considerationsIBM Software products, including software as a service solutions, (“Software Offerings”) may use cookiesor other technologies to collect product usage information, to help improve the end user experience, totailor interactions with the end user or for other purposes. In many cases no personally identifiableinformation is collected by the Software Offerings. Some of our Software Offerings can help enable you tocollect personally identifiable information. If this Software Offering uses cookies to collect personallyidentifiable information, specific information about this offering’s use of cookies is set forth below.

Depending upon the configurations deployed, this Software Offering may use session cookies that collecteach user’s session id for purposes of session management and authentication. These cookies can bedisabled, but disabling them will also eliminate the functionality they enable.

If the configurations deployed for this Software Offering provide you as customer the ability to collectpersonally identifiable information from end users via cookies and other technologies, you should seekyour own legal advice about any laws applicable to such data collection, including any requirements fornotice and consent.

For more information about the use of various technologies, including cookies, for these purposes, SeeIBM’s Privacy Policy at http://www.ibm.com/privacy and IBM’s Online Privacy Statement at http://www.ibm.com/privacy/details the section entitled “Cookies, Web Beacons and Other Technologies” andthe “IBM Software Products and Software-as-a-Service Privacy Statement” at http://www.ibm.com/software/info/product-privacy.


https://ibm.com/gdpr

https://ibm.com/gdpr

http://www.ibm.com/privacy





IBM QRadar: Architecture and Deployment Guide · Introduction to QRadar deployments. The IBM QRadar...

Documents

Transcript of IBM QRadar: Architecture and Deployment Guide · Introduction to QRadar deployments. The IBM QRadar...