File Audit Logging -...
Transcript of File Audit Logging -...
![Page 1: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/1.jpg)
File Audit LoggingMay 16th, 2018
Boston User Group Event
By
Subashini Balachandran1
![Page 2: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/2.jpg)
Motivation and Description
• Capturefileoperationsonagivenfilesystemandlogthemforauditingpurposes
• Displaythestoredevents• Capturemostcommontypesoffileoperationactivityonthefilesystem{create,open,close,destroy,rename,ACLchanges,XATTRchanges,rmdir,unlink}
• Protocolagnostic– SupportNativeGPFS,NFS,SMB• EventsareloggedinaJSONformattedstring• Configurable options for log output include the device where it is
mounted, name, retention period.• Integrated into the system health infrastructure for easy
monitoring of audit logging message queues and components
2©2018 IBM Corporation
![Page 3: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/3.jpg)
Kafka Publish-Subscribe model
• Eachauditedfilesystemwillhaveanuniquetopic assignedtoitintheMsgQueue
• ProducersliveinsidetheGPFSdaemonpublisheventstotherelevanttopic
• Consumerssubscribetoonetopic• Reliablearchitecture
• Brokersareclustered• Consumergroups• Eventsreplicationacross
Brokers
3©2018 IBM Corporation
![Page 4: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/4.jpg)
Architecture Overview
*Zookeeperresidesonthequorumnodes**KafkaBrokerscanresideonanynode(notconfinedtoprotocolnodesasdepictedinthisfigure)***UsingthestandardizedJSONformat,clientfacingAPIcanbederived.
GPFSDaemon
KafkaProducer
Quorumnode
Zookeeper
Kafkabroker
Partitions
GPFSDaemon
KafkaProducer
Protocolnode
…
GPFSDaemon
KafkaProducer
Protocolnode
GPFSDaemon
KafkaProducer
Zookeeper
ProtocolnodeQuorumnode
LWE+
librdkafka
GPFSDaemon
KafkaProducer
Quorumnode
Zookeeper
GPFSDaemon
KafkaProducer
Quorumnode
Zookeeper
KafkaClusterKafkabroker Kafkabroker Kafkabroker
ConsumerGroup(Topicfs0)
Consumer Consumer Consumer
JSONmessages
4©2018 IBM Corporation
![Page 5: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/5.jpg)
Flow of an event
SeqNbr Description
1 Clientperformsafileoperation(read/write/remove, ..)onafileinanauditedfilesystem
2 Externalclientnodesendstheclientrequesttotherelevant gpfs-node
3 Gpfs daemonusing internalLWE(lightweight events)machinerysendstheeventstothemsgQueue
4 Event messagesarereliablydeliveredtomsgQueue listeningonthistopic.
GPFSDaemon
KafkaProducer
ExternalClientnode
LWE+
librdkafkaKafkabroker Consumerlibrdkafka
JSONmessages
1 2 3 4 56 7
SeqNbr Description
5,6 Consumersbelonging toaconsumerGroup listeningonthiseventtopic,willperiodicallypulleventsfromthemsgQueue
7 ConsumerswillwritetheconsumedeventsfromtheMsgQueue intotheauditedfilesystem’s“.audit_log”fileset.
5©2018 IBM Corporation
![Page 6: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/6.jpg)
ConfigurationandSetup
• OnlyLinuxnodes(RHELandUbuntu)• LinuxKernelversionabove>3.10• Minimumof3Linuxquorumnodes• Minimumof3nodesmustbedesignatedasBrokernodes• Supportedhardwareplatforms(x86andPPCLE)
• RHELissupportedonx86andPPCLE• Ubuntuisonlysupportedonx86
• Recommendthattheports9092,9093(notusedcurrently,butwillinfuture),2181and2888-3888areopenedforTCPonly.
• AdvancedLicenseeditionortheDataManagementedition
6©2018 IBM Corporation
![Page 7: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/7.jpg)
• DuringInstallation,mostconfigurationisautomaticallydoneandstoredin/opt/kafka folder
• Freespacerequirements• min5GBlocaldiskspaceperfilesystembeingaudited• suggested10GBlocaldiskspaceperfilesystembeing
auditedonallbrokernodes• 2newrpmsaddedtothepackage5.0.0release
• gpfs.kafka-* • gpfs.librdkafka-*
• JavarpmsinstalledontheBrokerandZookeepernodes• gpfs.java-*
7©2018 IBM Corporation
![Page 8: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/8.jpg)
InstallGPFSpackages
./spectrumscalefileauditlogging
enable./spectrumscaleinstall–precheck
./spectrumscaleinstall–postcheck
Installation- LinuxNodesOnly
# ./spectrumscale fileauditlogging enable[ INFO ] Enabling file audit logging in the cluster configuration file.[ INFO ] Tip :If all node designations and any required file audit logging configurations are complete, proceed to assign filesystem to enable file audit logging configuration: ./spectrumscale filesystem modify --fileauditloggingenable <filesystem name>.
# ./spectrumscale node list..[ INFO ] File Audit logging : Enabled
# ./spectrumscale install –precheck..[ INFO ] Performing FILE AUDIT LOGGING checks.[ INFO ] Running environment checks for file Audit logging[ INFO ] File audit logging precheck OK≈
Afterinstallcompletes,verifythatinstallinstalledthenecessaryGPFSrpms# rpm -qa | egrep 'gpfs.java|kafka'gpfs.java*gpfs.kafka*gpfs.librdkafka*
# ./spectrumscale install –postcheck8
©2018 IBM Corporation
![Page 9: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/9.jpg)
Installation– Duringdeploy
# ./spectrumscale node add my_protocol_node1 -p[ INFO ] Setting my_protocol_node1.xxx.com as a protocol node.[ INFO ] Configuration updated.[ INFO ] Tip : If all node designations are complete, configure the protocol environment as needed: ./spectrumscaleconfig protocols -f cesSharedRoot -m /ibm/cesSharedRoot# ./spectrumscale node add my_protocol_node2 -p[ INFO ] Setting my_protocol_node2.xxx.com as a protocol node.[ INFO ] Configuration updated.[ INFO ] Tip : If all node designations are complete, configure the protocol environment as needed: ./spectrumscaleconfig protocols -f cesSharedRoot -m /ibm/cesSharedRoot
1. SpecifyprotocolnodeswhereKafkaBrokerswillreside.Note:Shownbeloware2nodesforbrevity,defaultconfigurationneeds3protocolnodes.
./spectrumscale nodeadd<Node1>-p
./spectrumscale nodeadd<Node2>-p
./spectrumscale filesystemmodify<Device>--fileauditloggingenable --logfileset .audit_log --retention365
./spectrumscale deploy--precheck -f
2.EnableNFSandSMBduringdeploy# ./spectrumscale enable nfs[ INFO ] Enabling NFS on all protocol nodes.[ INFO ] Tip :If all node designations and any required protocol configurations are complete, proceed to check the installation configuration:./spectrumscale deploy –precheck
# ./spectrumscale enable smb[ INFO ] Enabling SMB on all protocol nodes.[ INFO ] Tip :If all node designations and any required protocol configurations are complete, proceed to check the installation configuration:./spectrumscale deploy --precheck
9©2018 IBM Corporation
![Page 10: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/10.jpg)
# ./spectrumscale filesystem modify fs0 --fileauditloggingenable --logfileset .audit_log --retention 2[ INFO ] The filesystem fs0 will be configured with file audit logging.[ INFO ] Tip : Now that you have modified this filesystem to use file audit logging, you need to enable it using the'./spectrumscale fileauditlogging enable' command. please ignore if you have already enabled file audit logging.[ INFO ] The filesystem fs0 will be configured file audit logging with .audit_log log fileset.[ INFO ] The filesystem fs0 will be configured file audit logging with 2 retention days.
3.Duringdeployconfiguration,modifyfilesystem(s)forauditlogging
4.Deployprecheck willdisplayprecheck statusoffileauditlogging# ./spectrumscale deploy --precheck -f
.
.[ INFO ] Performing FILE AUDIT LOGGING checks.[ INFO ] Running environment checks for file Audit logging[ INFO ] File audit logging precheck OK
5.Afterrunningdeploy,validateusingmm-CLIcommandstoensurefileauditloggingisenabled# mmaudit all listAudit Cluster Fileset Fileset RetentionDevice ID Device Name (Days)-----------------------------------------------------------------------------------------fs0 4842233323150338002 fs0 .audit_log 2# mmlsfs fs0 --file-audit-logflag value description------------------- ------------------------ -------------------------------------file-audit-log Yes File Audit Logging enabled? 10
©2018 IBM Corporation
![Page 11: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/11.jpg)
Enablement- mmmsgqueuecommand
• CustomenablementofMsgQueue,toaccommodatenon-protocolnodesasBrokernodes
11©2018 IBM Corporation
![Page 12: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/12.jpg)
Enablement- mmauditcommand
• PostInstallationanddeployment,Fileauditloggingcanbeenabledusing“mmaudit”
12©2018 IBM Corporation
![Page 13: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/13.jpg)
Loggingdetails- Whereisitlogged
• Eachfilesystemenabledforfileauditlogging,hasadedicatedfilesetwheretheauditlogswillgo.Defaultoptionis.audit_log
• .audit_log fileset iscreatedasIAMmodenoncompliant.• Filescannotbedeletedifretentiontimeisnotexpired.• Butretentiontimescanberesetandfilescanbedeletedbutnotchanged,byroot
useronly.• AuditLog filesarenestedwithin/FS/.audit_log/topic/year/month/date/*• Easytosearchandconsume
13©2018 IBM Corporation
![Page 14: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/14.jpg)
• LiveeventscanbemonitoredbytailingthecurrentauditLogFile<…>• Logfileiswrittentoanappendonlymode• Rotationtoanewlogfile,uponreachingathreshold(500,000events),iscompressedandmarkedimmutablefortheretentionperiod.
• Defaultretentionperiodis365days
14©2018 IBM Corporation
![Page 15: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/15.jpg)
Loggingdetails-Whatislogged (JSON)
{"LWE_JSON":"0.0.1","path": "/newfs/1Kfile2.restore","oldPath": null, "clusterName": "pardie.cluster", "nodeName":
"c6f2bc3n10","nfsClientIp": "", "fsName": "newfs", "event":"OPEN","inode": "26626", "openFlags": "32962", "poolName":
"sp1","fileSize": "0", "ownerUserId": "0", "ownerGroupId": "0", "atime": "2017-10-25_12:36:22-0400", "ctime": "2017-10-25_12:36:22-0400", "eventTime": "2017-10-25_12:36:22-0400", "clientUserId": "0", "clientGroupId": "0", "processId": "10437", "permissions": "200100644", "acls": "u::rwc, g::r, o::r, ", "xattrs": null }
AttributeName Description
LWE_JSON Versionoftherecord
Path Pathnameofthefileinvolvedintheevent
oldPath PreviouspathnameofthefileduringRENAMEevent.Forallothereventsindicatedasnull.
clusterName Nameoftheclusterwheretheeventtookplace
nodeName Nameofthenodewheretheeventtookplace
nfsClientIp IPaddressoftheremoteclientinvolvedintheevent
fsName nameofthefilesysteminvolvedintheevent
event eventtype.Oneofthefollowingevents{OPEN,CREATE,CLOSE,RENAME,XATTRCHANGE,ACLCHANGE,UNLINK,DESTROY,RMDIR}
inode inode numberofthefileinvolvedintheevent 15©2018 IBM Corporation
![Page 16: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/16.jpg)
AttributeName Description
openFlags openflagsspecifiedduringtheevent(O_RDONLY,O_WRONLY,O_RDWR,O_CREAT,...)asdefinedinfcntl.h
poolName poolnamewherethefileresides
fileSize currentsizeofthefileinbytes
ownerUserId owneridofthefileinvolvedintheevent
ownerGroupId groupidofthefileinvolvedintheevent
atime ThetimeinUTCformatofthelastaccessofthefileinvolvedintheevent
ctime ThetimeinUTCformatofthelaststatuschangeofthefileinvolvedintheevent
eventTime ThetimeinUTCformatoftheevent
clientUserId useridofprocessinvolvedintheevent
clientGroupId groupidoftheprocessinvolvedintheevent
processId processidinvolvedintheevent
permissions permissionsonthefileinvolvedintheevent
acls theaccesscontrollistsinvolvedintheevent(Onlyincaseofaclchangeevent)
xattrs theextendedattributesinvolvedintheevent(OnlyincaseofanXattr changeevent)
16©2018 IBM Corporation
![Page 17: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/17.jpg)
17
Authentication
• Protectionfornon-GPFSproducer/consumersfromconnectingtotheMsgQueue
• Brokers(MsgQueue)isstartedwithauth mode• SASL_PLAINTEXT(msgQ-gen=0)– forrelease5.0.0• SASL_SCRAM(SHA-512)-- starting5.0.1release
• SASL_SCRAM thedefaultauthenticationmodegoingforward.• usernameandpasswordarestoredintheCCR• ProducerandConsumerswillfetch{username:password}fromCCRatFAL-enable/mountofthefilesystem
• WheneverMsgQueue isdisabledandre-enabled,MsgQueuegenerationnumberisincrementedgeneratingnew{username:password}
• AdditionallevelofvalidationwithProducerandConsumersregisteringwiththeCCRusingtheMsgQueue-genNbr whenfetching{username:password} ©2018 IBM Corporation
![Page 18: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/18.jpg)
18
Upgradefrom5.0.0to5.0.1
• ChangeinauthenticationmodefromPLAINTEXTtoSCRAM• Onetimere-configurationoftheMsgQueue withSCRAM
configuration• Additionalopenssl andlibssl-devLinuxlibrariesneededfor
thenewauthenticationmode• ForRHEL,openssl-devel andcyrus-sasl-devel packages• ForUbuntu,libssl-devandlibsasl2-devpackages
Install5.0.1packagesUpgradeclustermmchconfig
release=LATEST
mmauditalllistmmauditalldisable
mmmsgqueuestatusmmmsgqueueconfig --removemmsgqueueenable-N<listof
brokers>
mmauditallenable
©2018 IBM Corporation
![Page 19: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/19.jpg)
root@windwalker-vm1:~#mmchconfig release=LATEST
Verifyingthatallnodesintheclusterareup-to-date...mmchconfig:Commandsuccessfullycompletedmmchconfig:Propagatingtheclusterconfigurationdatatoallaffectednodes. Thisisanasynchronousprocess.
root@windwalker-vm1:~#mmauditalllist
Audit Cluster Fileset Fileset RetentionDevice ID Device Name (Days)------------------------------------------------------------------------------------------------------------------fs0 6391413883505451835 fs0 .audit_log_wind_fs0 25fs1 6391413883505451835 fs1 .audit_log_wind_fs1 365
1.Upgradeclustertolatestrelease(5.0.0to5.0.1)
2.Listtheexistingfilesystemsthatarefileauditloggingenabled
3.Disablingallthefileauditloggingenabledfilesystems,inthisexampleroot@windwalker-vm1:~#mmauditfs0disable
[I]SuccessfullydeletedFileAuditLoggingpolicypartition(s)fordevice:fs0[I]SuccessfullydisabledFileAuditLoggingconsumergroupfordevice:fs0[I]SuccessfullydisabledACLaccesstotheFileAuditLoggingtopicoftheMsgQueue fordevice:fs0[I]SuccessfullydeletedFileAuditLoggingtopicfromtheMsgQueue fordevice:fs0[I]SuccessfullyupdatedFileAuditLoggingconfigurationfordevice:fs0[I]SuccessfullydisabledFileAuditLoggingfordevice:fs0
ManuallyupgradingFALfrom5.0.0to5.0.1
19©2018 IBM Corporation
![Page 20: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/20.jpg)
root@windwalker-vm1:~#mmauditfs1disable
[I]SuccessfullydeletedFileAuditLoggingpolicypartition(s)fordevice:fs1[I]SuccessfullydisabledFileAuditLoggingconsumergroupfordevice:fs1[I]SuccessfullydisabledACLaccesstotheFileAuditLoggingtopicoftheMsgQueue fordevice:fs1[I]SuccessfullydeletedFileAuditLoggingtopicfromtheMsgQueue fordevice:fs1[I]SuccessfullyupdatedFileAuditLoggingconfigurationfordevice:fs1[I]SuccessfullyremovedFileAuditLoggingconsumercallbacks[I]SuccessfullyremovedFileAuditLoggingconsumernodeclasskafkaAuditConsumerServers[I]SuccessfullydisabledFileAuditLoggingfordevice:fs1
4.Checkingthemessagequeuestatus,recordingwhichnodesarebrokernodes,andremovingthemessagequeueroot@windwalker-vm1:~#mmmsgqueuestatus
Node Contains Broker Contains ZookeeperName Broker Status Zookeeper Statuswindwalker-vm1.tuc.stglabs.ibm.com yes good yes goodwindwalker-vm2.tuc.stglabs.ibm.com yes good yes goodwindwalker-vm3.tuc.stglabs.ibm.com yes good yes goodwindwalker-vm4.tuc.stglabs.ibm.com yes good nowindwalker-vm5.tuc.stglabs.ibm.com no yes goodwindwalker-vm6.tuc.stglabs.ibm.com no yes goodroot@windwalker-vm1:~#mmmsgqueueconfig --remove
[I]AttemptingtodisabletheMsgQueue. Thismaytakesometime.[I]DisablingMsgQueue daemons.[I]RemovingcallbacksthatcontrolstartingandstoppingtheMsgQueue daemons.[I]MsgQueue successfullydisabled.[I]RemovingMsgQueue callbacks,nodeclassesandconfigurationinformationifpresent.[I]MsgQueue successfullydisabledandconfigurationremoved.
20©2018 IBM Corporation
![Page 21: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/21.jpg)
5.Re-enablingthemessagequeueusingthesamebrokernodesfrombeforeroot@windwalker-vm1:~#mmmsgqueueenable-Nwindwalker-vm1.tuc.stglabs.ibm.com,windwalker-vm2.tuc.stglabs.ibm.com,
windwalker-vm3.tuc.stglabs.ibm.com,windwalker-vm4.tuc.stglabs.ibm.com
[I]ThekafkaZookeeperServers nodeclasswassuccessfullycreatedwith5membernodes.[I]ThekafkaBrokerServers nodeclasswassuccessfullycreatedwith4membernodes.[I]SuccessfullycreatedKafkabrokerconfigurationfileandaddedtoCCR.[I]SuccessfullycreatedKafkaZookeeperconfigurationfileandaddedtoCCR.[I]EnablingMsgQueue daemons.[I]CreatingcallbackstocontrolstartingandstoppingtheMsgQueue daemons.[I]Pushingproducerauthenticationinformationtoeligibleclusternodes.Dependingonclustersize,thismaytakesometime.
[I]MsgQueue successfullyenabled.
6.EnableFALforfs0andfs1root@windwalker-vm1:~#mmauditfs0enable
[I]SuccessfullycreatedFileAuditLoggingconsumernodeclasskafkaAuditConsumerServers[I]VerifyingMsgQueue nodesmeetminimumlocalspacerequirementsforFileAuditLoggingtobeenabledfordevice:fs0.Dependingonclustersize,thismaytakesometime.
[I]SuccessfullyverifiedallconfiguredMsgQueue nodesmeetminimumlocalspacerequirementsforFileAuditLoggingtobeenabledfordevice:fs0[I]SuccessfullyupdatedFileAuditLoggingconfigurationfordevice:fs0[I]SuccessfullycreatedFileAuditLoggingtopicontheMsgQueue fordevice:fs0[I]SuccessfullyenabledACLaccesstothetopicforproducersandconsumersfordevice:fs0[I]Successfullycreated/linkedFileAuditLoggingauditfileset .audit_log withlinkpoint/fs0/.audit_log[I]SuccessfullyenabledFileAuditLoggingconsumergrouptoauditdevice:fs0[I]SuccessfullycreatedFileAuditLoggingpolicypartition(s)toauditdevice:fs0[I]SuccessfullycreatedFileAuditLoggingconsumercallbacks[I]SuccessfullyenabledFileAuditLoggingfordevice:fs0
21©2018 IBM Corporation
![Page 22: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/22.jpg)
22
6.Finally,viewthenewfileauditloggingconfiguration
root@windwalker-vm1:~#mmauditalllist
Audit Cluster Fileset Fileset RetentionDevice ID Device Name (Days)-------------------------------------------------------------------------------------------------------------------------fs0 6391413883505451835 fs0 .audit_log 365fs1 6391413883505451835 fs1 .audit_log_SCRAM_fs1 10
root@windwalker-vm1 [root@fin21p~]#mmauditfs1enable--log-fileset .audit_log_SCRAM_fs1--retention10
[I]SuccessfullycreatedFileAuditLoggingconsumernodeclasskafkaAuditConsumerServers[I]VerifyingMsgQueue nodesmeetminimumlocalspacerequirementsforFileAuditLoggingtobeenabledfordevice:fs1.Dependingonclustersize,thismaytakesometime.
[I]SuccessfullyverifiedallconfiguredMsgQueue nodesmeetminimumlocalspacerequirementsforFileAuditLoggingtobeenabledfordevice:fs1[I]SuccessfullyupdatedFileAuditLoggingconfigurationfordevice:fs1[I]SuccessfullycreatedFileAuditLoggingtopicontheMsgQueue fordevice:fs1[I]SuccessfullyenabledACLaccesstothetopicforproducersandconsumersfordevice:fs1[I]Successfullycreated/linkedFileAuditLoggingauditfileset .audit_log_SCRAM_lroc_fs withlinkpoint/fs1/.audit_log_SCRAM_fs1[I]SuccessfullyenabledFileAuditLoggingconsumergrouptoauditdevice:fs1[I]SuccessfullycreatedFileAuditLoggingpolicypartition(s)toauditdevice:fs1[I]SuccessfullycreatedFileAuditLoggingconsumercallbacks[I]SuccessfullyenabledFileAuditLoggingfordevice:fs1
![Page 23: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/23.jpg)
HealthmonitoringforFAL
• MonitoringusingCLIcommands• mmaudit• mmmsgqueue• mmpmon
• Monitoringusingmmhealth• Clusterwide• Nodeview
• MonitoringofFILEAUDITLOGcomponent• auditc_xxx eventsraisedforvariouserrorandwarningscenarios
• MonitoringofMSGQUEUEcomponent• Kafka_xxx |zookeeper_xxx eventsraisedforvariousmsgQueue error
andwarningscenarios• MonitoringusingGUI
• ViatheServiceandEventspanel23
©2018 IBM Corporation
![Page 24: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/24.jpg)
FALmonitoringusingCLI-cmds
• mmauditallconsumerStatus–N…
• mmmsgqueuestatus
24©2018 IBM Corporation
![Page 25: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/25.jpg)
FALmonitoringusingCLI-cmds
• mmpmon lkp_s
25©2018 IBM Corporation
![Page 26: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/26.jpg)
Clusterwide:mmhealthclustershow
• Periodicpollingandeventcallbackregistrationmechanismisused.• Possiblelagindeterminingthehealthduetopollingconstraints.
26©2018 IBM Corporation
![Page 27: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/27.jpg)
Nodeview:mmhealthnodeshow
27
Twoseparatecomponentsmonitored• FILEAUDITLOG• MSGQUEUE
©2018 IBM Corporation
![Page 28: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/28.jpg)
Eventsview:mmhealtheventlog show
28©2018 IBM Corporation
![Page 29: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/29.jpg)
29
FALmonitoringfromtheGUI
Homescreen• Ontheright-handyoucanseetheoverallFileAuditingandMessageQueuestatus
©2018 IBM Corporation
![Page 30: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/30.jpg)
30
• WhichfilesystemsareenabledforFAL.• RequestthisbyusingtheActionspull-downthatisshownandthencustomizethecolumnstoviewthefileauditedfilesystems.
GUI– FileSystemsPanel
©2018 IBM Corporation
![Page 31: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/31.jpg)
GUI– Servicesè FileAuditingPanel
31
• View the overall File Auditing status for each node.
• This is a healthy system, so there is nothing in the Events section.
©2018 IBM Corporation
![Page 32: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/32.jpg)
32
GUI– Servicesè FileAuditingPanel
• View the Auditing status at the File System level.
©2018 IBM Corporation
![Page 33: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/33.jpg)
33
GUI– ServicesèMessageQueuePanel
• view the members of the message queue.
• aligns with the "mmmsgqueue status" CLI command.
• This is a healthy system, so there is nothing in the Events section.
©2018 IBM Corporation
![Page 34: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/34.jpg)
34
GUI– Accessè CommandAuditLogPanel
• Every time a command related to FAL is ran (mmaudit, mmmsgqueue, mmcrnodeclass, etc.), it is logged in this panel.
©2018 IBM Corporation
![Page 35: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/35.jpg)
35
Performance
• Runperfteststoevaluatetheaboveconcerns• Setup
• Kafkacluster:4Brokernodes,3zookeepernodes,4consumernodes• Gpfs Cluster:4protocol nodes,2NSDservernodes (Linux3.10.0-229.el7.x86_64)• Network:10GE• Storage:IBMDCS3700
• Testsrun• Metadataintensiveworkloadbenchmark
• WithandwithoutFAL• mdtest
• WithFALenabled• FilecreatewithMPI-count
• Concerns• DoesenablingFALimpactIO-performanceonmyfilesystem?• HowperformantisFAL?
©2018 IBM Corporation
![Page 36: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/36.jpg)
• IBM’sstatementsregardingitsplans,directions,andintentaresubjecttochangeorwithdrawalwithoutnoticeatIBM’ssolediscretion.Informationregardingpotentialfutureproductsisintendedtooutlineourgeneralproductdirectionanditshouldnotbereliedoninmakingapurchasingdecision.Theinformationmentionedregardingpotentialfutureproductsisnotacommitment,promise,orlegalobligationtodeliveranymaterial,codeorfunctionality.Informationaboutpotentialfutureproductsmaynot beincorporatedintoanycontract.Thedevelopment,release,andtimingofanyfuturefeaturesorfunctionalitydescribedforourproductsremainsatoursolediscretion.
• Performanceisbasedonmeasurementsandprojectionsusingstandardbenchmarksinacontrolledenvironment.TheactualthroughputorperformancethatanyuserwillexperiencewillvarydependinguponmanyfactorssuchastheI/Oconfiguration,thestorageconfiguration,andtheworkloadcharacteristics.Therefore,noassurancecanbegiventhatanindividualuserwillachieveresultssimilartothosestatedhere.
Disclaimer
35IBM Confidential ©2018 IBM Corporation
![Page 37: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/37.jpg)
37IBM Confidential ©2018 IBM Corporation
![Page 38: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/38.jpg)
3837IBM Confidential ©2018 IBM Corporation
![Page 39: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/39.jpg)
Troubleshooting
• /var/adm/ras/mmmsgqueue.log
• Containsinformationregardingthesetupandconfigurationoperationsthattakeplacethataffectthemessagequeue
• Validonanynodecontainingabrokerand/orzookeeper• /var/adm/ras/mmaudit.log
• ContainsinformationregardingthesetupandconfigurationoperationsthattakeplacethataffecttheFileAuditLogging
• ValidonanynoderunningtheFileAuditLoggingcommandorlocationwherethesubcommandmayberun(suchasaconsumer)
• /var/adm/ras/mmfs.log.latest
• Daemonlog,andcontainsentrieswhenmajormessagequeueorFileAuditLoggingactivityoccurs.
• /var/log/messages(Redhat)• /var/log/syslog(Ubuntu)
• ContainsmessagesfromKafkacomponentsaswellastheproducerandconsumersthatarerunningonanode.
• Logscollectedviagpfs.snap39
©2018 IBM Corporation
![Page 40: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/40.jpg)
References
• https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1ins_quickrefadlg.htm
40©2018 IBM Corporation
![Page 41: File Audit Logging - files.gpfsug.orgfiles.gpfsug.org/presentations/2018/USA/FileAuditLogging_SSUG_Boston.pdf · Disabling all the file audit logging enabled file systems, in this](https://reader030.fdocuments.us/reader030/viewer/2022040610/5ecff79647a72a76b67bebd1/html5/thumbnails/41.jpg)
MerciGrazie
Gracias
Obrigado
Danke
Japanese
Hebrew
Thank YouThank YouEnglish
French
Russian
German
Italian
Spanish
Brazilian Portuguese
Hindi
Tamil
Korean
Thai
Simplified Chinese
Arabic
Traditional Chinese