Table of ContentsDatacenter to datacenter Replication on Kubernetes Highlights Scalability...
Transcript of Table of ContentsDatacenter to datacenter Replication on Kubernetes Highlights Scalability...
1.1
1.2
1.2.1
1.2.1.1
1.2.1.2
1.2.1.3
1.2.1.4
1.2.2
1.2.3
1.2.4
1.3
1.3.1
1.3.2
1.4
1.5
1.5.1
1.5.2
1.5.3
1.6
1.6.1
1.6.2
1.6.2.1
1.6.2.2
1.6.3
1.6.3.1
1.6.3.2
1.6.4
1.6.4.1
1.6.4.2
1.6.4.3
1.6.5
1.6.6
1.6.6.1
1.6.6.2
1.6.6.3
1.6.6.4
1.7
1.7.1
1.7.2
1.7.3
1.7.4
TableofContentsIntroduction
GettingStarted
Installing
Linux
MacOSX
Windows
Compiling
Authentication
AccessingtheWebInterface
ComingfromSQL
Tutorials
Kubernetes
DatacentertodatacenterReplicationonKubernetes
Highlights
Scalability
Architecture
Datamodels
Limitations
Datamodels&modeling
Concepts
Databases
WorkingwithDatabases
NotesaboutDatabases
Collections
CollectionMethods
DatabaseMethods
Documents
BasicsandTerminology
CollectionMethods
DatabaseMethods
Graphs,Vertices&Edges
NamingConventions
DatabaseNames
CollectionNames
DocumentKeys
AttributeNames
Indexing
IndexBasics
Whichindextousewhen
IndexUtilization
WorkingwithIndexes
1
1.7.4.1
1.7.4.2
1.7.4.3
1.7.4.4
1.7.4.5
1.7.4.6
1.8
1.8.1
1.8.1.1
1.8.1.2
1.8.2
1.8.2.1
1.8.3
1.8.3.1
1.8.3.2
1.8.4
1.8.5
1.9
1.9.1
1.9.2
1.9.3
1.9.4
1.9.5
1.9.6
1.9.7
1.9.7.1
1.9.7.2
1.9.7.3
1.9.7.4
1.9.8
1.9.9
1.9.9.1
1.9.9.1.1
1.9.9.1.2
1.9.9.2
1.9.9.2.1
1.9.9.2.2
1.9.10
1.9.11
1.9.12
1.9.13
1.9.14
1.9.14.1
1.9.14.2
HashIndexes
Skiplists
Persistent
FulltextIndexes
GeoIndexes
VertexCentricIndexes
Graphs
GeneralGraphs
GraphManagement
GraphFunctions
SmartGraphs
SmartGraphManagement
Traversals
UsingTraversalObjects
ExampleData
WorkingwithEdges
Pregel
FoxxMicroservices
Ataglance
Gettingstarted
Servicemanifest
Servicecontext
Configuration
Dependencies
Routers
Endpoints
Middleware
Request
Response
UsingGraphQL
Sessionsmiddleware
Sessionstorages
Collectionstorage
JWTstorage
Sessiontransports
Cookietransport
Headertransport
Servingfiles
Writingtests
CrossOrigin
Scriptsandqueuedjobs
Migrating2.xservices
Migratingfrompre-2.8
manifest.json
2
1.9.14.3
1.9.14.4
1.9.14.5
1.9.14.5.1
1.9.14.5.2
1.9.14.5.3
1.9.14.5.4
1.9.14.5.5
1.9.14.5.6
1.9.14.6
1.9.14.7
1.9.14.8
1.9.15
1.9.16
1.9.17
1.9.17.1
1.9.17.2
1.9.17.3
1.10
1.10.1
1.10.2
1.10.3
1.10.4
1.10.5
1.11
1.11.1
1.11.2
1.11.2.1
1.11.2.2
1.11.2.3
1.11.2.3.1
1.11.2.3.2
1.11.2.3.3
1.11.2.3.4
1.11.3
1.11.4
1.11.4.1
1.11.4.2
1.11.4.3
1.11.4.4
1.11.4.5
1.11.4.6
1.11.4.7
1.11.4.8
applicationContext
RepositoriesandModels
Controllers
Requestcontext
Errorhandling
Before/After/Around
Requestobject
Responseobject
DependencyInjection
Sessions
AuthandOAuth2
FoxxQueries
Legacycompatibilitymode
Usermanagement
Relatedmodules
Authentication
OAuth1.0a
OAuth2.0
Transactions
Transactioninvocation
Passingparameters
Lockingandisolation
Durability
Limitations
Deployment
Singleinstance
Cluster
Mesos,DC/OS
Generic&Docker
AdvancedTopics
StandaloneAgency
Localtestsetups
Processes
Docker
MultipleDatacenters
Kubernetes
UsingtheOperator
DeploymentResourceReference
DriverConfiguration
Authentication
Scaling
Upgrading
ArangoDBConfiguration&Secrets
Metrics
3
1.11.4.9
1.11.4.10
1.11.4.11
1.11.4.12
1.11.4.13
1.11.4.14
1.12
1.12.1
1.12.1.1
1.12.1.2
1.12.1.3
1.12.1.4
1.12.1.5
1.12.1.6
1.12.1.7
1.12.1.8
1.12.1.9
1.12.2
1.12.2.1
1.12.2.2
1.12.2.3
1.12.3
1.12.4
1.12.5
1.12.6
1.12.7
1.12.7.1
1.12.8
1.12.8.1
1.12.8.2
1.12.8.3
1.12.8.4
1.12.8.5
1.12.8.6
1.12.8.7
1.12.8.8
1.12.8.9
1.12.8.10
1.12.8.11
1.12.8.12
1.12.9
1.12.10
1.12.11
1.12.11.1
Services&Loadbalancer
DeploymentReplicationResourceReference
Storage
StorageResource
TLS
Troubleshooting
Administration
WebInterface
Dashboard
Cluster
Collections
Document
Queries
Graphs
Services
Users
Logs
ArangoDBShell
ShellOutput
Configuration
Details
Arangoimp
Arangodump
Arangorestore
Arangoexport
ManagingUsers
InArangosh
ServerConfiguration
OperatingSystemConfiguration
ManagingEndpoints
SSLConfiguration
LDAPOptions
LoggingOptions
GeneralOptions
Write-AheadLogOptions
CompactionOptions
ClusterOptions
RocksDBEngineOptions
HashCacheOptions
AsynchronousTasks
Durability
Encryption
Auditing
Configuration
4
1.12.11.2
1.12.12
1.12.12.1
1.12.12.1.1
1.12.12.1.2
1.12.12.1.3
1.12.12.1.4
1.12.12.1.5
1.12.12.2
1.12.12.2.1
1.12.12.2.2
1.12.12.3
1.12.13
1.12.14
1.12.14.1
1.12.14.2
1.12.14.3
1.12.14.4
1.12.14.5
1.12.14.6
1.12.14.7
1.12.14.8
1.12.14.9
1.12.14.10
1.13
1.13.1
1.13.2
1.13.3
1.13.4
1.14
1.14.1
1.14.2
1.15
1.15.1
1.15.2
1.15.3
1.15.4
1.15.5
1.15.6
1.15.7
1.15.8
1.15.9
1.15.10
1.15.11
Events
Replication
AsynchronousReplication
Components
Per-DatabaseSetup
Server-LevelSetup
SyncingCollections
ReplicationLimitations
SynchronousReplication
Implementation
Configuration
SatelliteCollections
Sharding
Upgrading
Upgradingto3.3
Upgradingto3.2
Upgradingto3.1
Upgradingto3.0
Upgradingto2.8
Upgradingto2.6
Upgradingto2.5
Upgradingto2.4
Upgradingto2.3
Upgradingto2.2
Troubleshooting
arangod
EmergencyConsole
DatafileDebugger
Arangobench
Architecture
Write-aheadlog
StorageEngines
Releasenotes
WhatsNewin3.3
Incompatiblechangesin3.3
WhatsNewin3.2
KnownIssuesin3.2
Incompatiblechangesin3.2
WhatsNewin3.1
Incompatiblechangesin3.1
WhatsNewin3.0
Incompatiblechangesin3.0
WhatsNewin2.8
Incompatiblechangesin2.8
5
1.15.12
1.15.13
1.15.14
1.15.15
1.15.16
1.15.17
1.15.18
1.15.19
1.15.20
1.15.21
1.15.22
1.15.23
1.16
1.16.1
1.16.1.1
1.16.1.2
1.16.2
1.16.2.1
1.16.2.2
1.16.2.3
1.16.2.4
1.16.2.5
1.16.2.6
1.16.2.7
1.16.2.8
1.16.2.9
1.16.3
1.16.3.1
1.16.3.1.1
1.16.3.1.2
1.16.3.1.3
1.16.3.1.4
1.16.3.1.5
1.16.3.2
1.16.3.2.1
1.16.3.2.2
1.16.3.2.3
1.16.4
1.16.5
WhatsNewin2.7
Incompatiblechangesin2.7
WhatsNewin2.6
Incompatiblechangesin2.6
WhatsNewin2.5
Incompatiblechangesin2.5
WhatsNewin2.4
Incompatiblechangesin2.4
WhatsNewin2.3
Incompatiblechangesin2.3
WhatsNewin2.2
WhatsNewin2.1
Appendix
References
db
collection
JavaScriptModules
@arangodb
console
crypto
fs
request
actions
queries
Write-aheadlog
TaskManagement
Deprecated
SimpleQueries
SequentialAccess
Pagination
ModificationQueries
GeoQueries
FulltextQueries
Actions
DeliveringHTMLPages
JsonObjects
Modifying
Errorcodesandmeanings
Glossary
6
ArangoDBv3.3.10DocumentationWelcometotheArangoDBdocumentation!
NewandeagertotryoutArangoDB?Startrightawaywithourbeginner'sguide:GettingStarted
Thedocumentationisorganizedinfourhandbooks:
ThismanualdescribesArangoDBanditsfeaturesindetailforyouasauser,developerandadministrator.TheAQLhandbookexplainsArangoDB'squerylanguageAQL.TheHTTPhandbookdescribestheinternalAPIofArangoDBthatisusedtocommunicatewithclients.Ingeneral,theHTTPhandbookwillbeofinteresttodriverdevelopers.Ifyouuseanyoftheexistingdriversforthelanguageofyourchoice,youcanskipthishandbook.Ourcookbookwithrecipesforspecificproblemsandsolutions.
Featuresareillustratedwithinteractiveusageexamples;youcancut'n'pastethemintoarangoshtotrythemout.TheHTTPREST-APIfordriverdevelopersisdemonstratedwithcut'n'pasterecipesintendedtobeusedwiththecURL.Driversmayprovidetheirownexamplesbasedonthese.jsbasedexamplestoimproveunderstandabilityfortheirrespectiveusers,i.e.forthejavadriversomeofthesamplesarere-implemented.
Overview
ArangoDBisanativemulti-model,open-sourcedatabasewithflexibledatamodelsfordocuments,graphs,andkey-values.BuildhighperformanceapplicationsusingaconvenientSQL-likequerylanguageorJavaScriptextensions.UseACIDtransactionsifyourequirethem.Scalehorizontallyandverticallywithafewmouseclicks.
Keyfeaturesinclude:
installingArangoDBonaclusterisaseasyasinstallinganapponyourmobileFlexibledatamodeling:modelyourdataascombinationofkey-valuepairs,documentsorgraphs-perfectforsocialrelationsPowerfulquerylanguage(AQL)toretrieveandmodifydataUseArangoDBasanapplicationserverandfuseyourapplicationanddatabasetogetherformaximalthroughputTransactions:runqueriesonmultipledocumentsorcollectionswithoptionaltransactionalconsistencyandisolationReplicationandSharding:setupthedatabaseinamaster-slaveconfigurationorspreadbiggerdatasetsacrossmultipleserversConfigurabledurability:lettheapplicationdecideifitneedsmoredurabilityormoreperformanceNo-nonsensestorage:ArangoDBusesallofthepowerofmodernstoragehardware,likeSSDandlargecachesJavaScriptforall:nolanguagezoo,youcanuseonelanguagefromyourbrowsertoyourback-endArangoDBcanbeeasilydeployedasafault-tolerantdistributedstatemachine,whichcanserveastheanimalbrainofdistributedappliancesItisopensource(ApacheLicense2.0)
Community
IfyouhavequestionsregardingArangoDB,Foxx,drivers,orthisdocumentationdon'thesitatetocontactuson:
GitHubforissuesandmisbehaviororpullrequestsGoogleGroupsfordiscussionsaboutArangoDBingeneralortoannounceyournewFoxxAppStackOverflowforquestionsaboutAQL,usagescenariosetc.Slack,ourcommunitychat
Whenreportingissues,pleasedescribe:
theenvironmentyourunArangoDBintheArangoDBversionyouusewhetheryou'reusingFoxxtheclientyou'reusing
Introduction
7
whichpartsofthedocumentationyou'reworkingwith(link)whatyouexpecttohappenwhatisactuallyhappening
Wewillrespondassoonaspossible.
Introduction
8
Gettingstarted
Overview
Thisbeginner'sguidewillmakeyoufamiliarwithArangoDB.Wewillcoverhowto
installandrunalocalArangoDBserverusethewebinterfacetointeractwithitstoreexampledatainthedatabasequerythedatabasetoretrievethedataagaineditandremoveexistingdata
Installation
Headtoarangodb.com/download,selectyouroperatingsystemanddownloadArangoDB.Youmayalsofollowtheinstructionsonhowtoinstallwithapackagemanager,ifavailable.
IfyouinstalledabinarypackageunderLinux,theserverisautomaticallystarted.
IfyouinstalledArangoDBusinghomebrewunderMacOSX,starttheserverbyrunning/usr/local/sbin/arangod.
IfyouinstalledArangoDBunderWindowsasaservice,theserverisautomaticallystarted.Otherwise,runthearangod.exelocatedintheinstallationfolder'sbindirectory.YoumayhavetorunitasadministratortograntitwritepermissionstoC:\ProgramFiles.
Formorein-depthinformationonhowtoinstallArangoDB,aswellasavailablestartupparameters,installationinaclusterandsoon,seeInstalling.
ArangoDBofferstwostorageengines:MMFilesandRocksDB.Choosetheonewhichsuitsyourneedsbestintheinstallationprocessoronfirststartup.
Securingtheinstallation
Thedefaultinstallationcontainsonedatabase_systemandausernamedroot.
DebianbasedpackagesandtheWindowsinstallerwillaskforapasswordduringtheinstallationprocess.Red-Hatbasedpackageswillsetarandompassword.Forallotherinstallationpackagesyouneedtoexecute
shell>arango-secure-installation
Thiswillaskforarootpasswordandsetsthispassword.
Webinterface
Theserveritself(arangod)speaksHTTP/REST,butyoucanusethegraphicalwebinterfacetokeepitsimple.There'salsoarangosh,asynchronousshellforinteractionwiththeserver.Ifyou'readeveloper,youmightprefertheshellovertheGUI.Itdoesnotprovidefeatureslikesyntaxhighlightinghowever.
WhenyoustartusingArangoDBinyourproject,youwilllikelyuseanofficialorcommunity-madedriverwritteninthesamelanguageasyourproject.Driversimplementaprogramminginterfacethatshouldfeelnaturalforthatprogramminglanguage,anddoallthetalkingtotheserver.Therefore,youcanmostcertainlyignoretheHTTPAPIunlessyouwanttowriteadriveryourselforexplicitlywanttousetherawinterface.
Togetfamiliarwiththedatabasesystemyoucanevenputdriversasideandusethewebinterface(codenameAardvark)forbasicinteraction.Thewebinterfacewillbecomeavailableshortlyafteryoustartedarangod.Youcanaccessitinyourbrowserathttp://localhost:8529-ifnot,pleaseseeTroubleshooting.
GettingStarted
9
Bydefault,authenticationisenabled.Thedefaultuserisroot.Dependingontheinstallationmethodused,theinstallationprocesseitherpromptedfortherootpasswordorthedefaultrootpasswordisempty(seeabove).
Nextyouwillbeaskedwhichdatabasetouse.Everyserverinstancecomeswitha_systemdatabase.Selectthisdatabasetocontinue.
GettingStarted
10
Youshouldthenbepresentedthedashboardwithserverstatisticslikethis:
Foramoredetaileddescriptionoftheinterface,seeWebInterface.
Databases,collectionsanddocuments
Databasesaresetsofcollections.Collectionsstorerecords,whicharereferredtoasdocuments.CollectionsaretheequivalentoftablesinRDBMS,anddocumentscanbethoughtofasrowsinatable.Thedifferenceisthatyoudon'tdefinewhatcolumns(orratherattributes)therewillbeinadvance.Everydocumentinanycollectioncanhavearbitraryattributekeysandvalues.Documentsinasinglecollectionwilllikelyhaveasimilarstructureinpracticehowever,butthedatabasesystemitselfdoesnotimposeitandwilloperatestableandfastnomatterhowyourdatalookslike.
Readmoreinthedata-modelconceptschapter.
Fornow,youcanstickwiththedefault_systemdatabaseandusethewebinterfacetocreatecollectionsanddocuments.StartbyclickingtheCOLLECTIONSmenuentry,thentheAddCollectiontile.Giveitaname,e.g.users,leavetheothersettingsunchanged(wewantittobeadocumentcollection)andSaveit.Anewtilelabeledusersshouldshowup,whichyoucanclicktoopen.
TherewillbeNodocumentsyet.Clickthegreencirclewiththewhiteplusontheright-handsidetocreateafirstdocumentinthiscollection.Adialogwillaskyoufora_key.YoucanleavethefieldblankandclickCreatetoletthedatabasesystemassignanautomaticallygenerated(unique)key.Notethatthe_keypropertyisimmutable,whichmeansyoucannotchangeitoncethedocumentiscreated.Whatyoucanuseasdocumentkeyisdescribedinthenamingconventions.
Anautomaticallygeneratedkeycouldbe"9883"(_keyisalwaysastring!),andthedocument_idwouldbe"users/9883"inthatcase.Asidefromafewsystemattributes,thereisnothinginthisdocumentyet.Let'saddacustomattributebyclickingtheicontotheleftof(emptyobject),thenAppend.Twoinputfieldswillbecomeavailable,FIELD(attributekey)andVALUE(attributevalue).Typenameaskeyandyournameasvalue.Appendanotherattribute,nameitageandsetittoyourage.ClickSavetopersistthechanges.IfyouclickonCollection:usersatthetopontheright-handsideoftheArangoDBlogo,thedocumentbrowserwillshowthedocumentsintheuserscollectionandyouwillseethedocumentyoujustcreatedinthelist.
Queryingthedatabase
GettingStarted
11
TimetoretrieveourdocumentusingAQL,ArangoDB'squerylanguage.Wecandirectlylookupthedocumentwecreatedviathe_id,buttherearealsootheroptions.ClicktheQUERIESmenuentrytobringupthequeryeditorandtypethefollowing(adjustthedocumentIDtomatchyourdocument):
RETURNDOCUMENT("users/9883")
ThenclickExecutetorunthequery.Theresultappearsbelowthequeryeditor:
[
{
"_key":"9883",
"_id":"users/9883",
"_rev":"9883",
"age":32,
"name":"JohnSmith"
}
]
Asyoucansee,theentiredocumentincludingthesystemattributesisreturned.DOCUMENT()isafunctiontoretrieveasingledocumentoralistofdocumentsofwhichyouknowthe_keysor_ids.Wereturntheresultofthefunctioncallasourqueryresult,whichisourdocumentinsideoftheresultarray(wecouldhavereturnedmorethanoneresultwithadifferentquery,butevenforasingledocumentasresult,westillgetanarrayatthetoplevel).
Thistypeofqueryiscalleddataaccessquery.Nodataiscreated,changedordeleted.Thereisanothertypeofquerycalleddatamodificationquery.Let'sinsertaseconddocumentusingamodificationquery:
INSERT{name:"KatieFoster",age:27}INTOusers
Thequeryisprettyself-explanatory:theINSERTkeywordtellsArangoDBthatwewanttoinsertsomething.Whattoinsert,adocumentwithtwoattributesinthiscase,followsnext.Thecurlybraces{}signifydocuments,orobjects.Whentalkingaboutrecordsinacollection,wecallthemdocuments.EncodedasJSON,wecallthemobjects.Objectscanalsobenested.Here'sanexample:
{
"name":{
"first":"Katie",
"last":"Foster"
}
}
INTOisamandatorypartofeveryINSERToperationandisfollowedbythecollectionnamethatwewanttostorethedocumentin.Notethattherearenoquotemarksaroundthecollectionname.
Ifyourunabovequery,therewillbeanemptyarrayasresultbecausewedidnotspecifywhattoreturnusingaRETURNkeyword.Itisoptionalinmodificationqueries,butmandatoryindataaccessqueries.EvenwithRETURN,thereturnvaluecanstillbeanemptyarray,e.g.ifthespecifieddocumentwasnotfound.Despitetheemptyresult,theabovequerystillcreatedanewuserdocument.Youcanverifythiswiththedocumentbrowser.
Let'saddanotheruser,butreturnthenewlycreateddocumentthistime:
INSERT{name:"JamesHendrix",age:69}INTOusers
RETURNNEW
NEWisapseudo-variable,whichreferstothedocumentcreatedbyINSERT.Theresultofthequerywilllooklikethis:
[
{
"_key":"10074",
"_id":"users/10074",
"_rev":"10074",
"age":69,
"name":"JamesHendrix"
}
]
GettingStarted
12
Nowthatwehave3usersinourcollection,howtoretrievethemallwithasinglequery?Thefollowingdoesnotwork:
RETURNDOCUMENT("users/9883")
RETURNDOCUMENT("users/9915")
RETURNDOCUMENT("users/10074")
TherecanonlybeasingleRETURNstatementhereandasyntaxerrorisraisedifyoutrytoexecuteit.TheDOCUMENT()functionoffersasecondarysignaturetospecifymultipledocumenthandles,sowecoulddo:
RETURNDOCUMENT(["users/9883","users/9915","users/10074"])
Anarraywiththe_idsofall3documentsispassedtothefunction.Arraysaredenotedbysquarebrackets[]andtheirelementsareseparatedbycommas.
Butwhatifweaddmoreusers?Wewouldhavetochangethequerytoretrievethenewlyaddedusersaswell.Allwewanttosaywithourqueryis:"Foreveryuserinthecollectionusers,returntheuserdocument".WecanformulatethiswithaFORloop:
FORuserINusers
RETURNuser
Itexpressestoiterateovereverydocumentinusersandtouseuserasvariablename,whichwecanusetorefertothecurrentuserdocument.Itcouldalsobecalleddoc,uorahuacatlguacamole,thisisuptoyou.Itisadvisabletouseashortandself-descriptivenamehowever.
Theloopbodytellsthesystemtoreturnthevalueofthevariableuser,whichisasingleuserdocument.Alluserdocumentsarereturnedthisway:
[
{
"_key":"9915",
"_id":"users/9915",
"_rev":"9915",
"age":27,
"name":"KatieFoster"
},
{
"_key":"9883",
"_id":"users/9883",
"_rev":"9883",
"age":32,
"name":"JohnSmith"
},
{
"_key":"10074",
"_id":"users/10074",
"_rev":"10074",
"age":69,
"name":"JamesHendrix"
}
]
Youmayhavenoticedthattheorderofthereturneddocumentsisnotnecessarilythesameastheywereinserted.Thereisnoorderguaranteedunlessyouexplicitlysortthem.WecanaddaSORToperationveryeasily:
FORuserINusers
SORTuser._key
RETURNuser
Thisdoesstillnotreturnthedesiredresult:James(10074)isreturnedbeforeJohn(9883)andKatie(9915).Thereasonisthatthe_keyattributeisastringinArangoDB,andnotanumber.Theindividualcharactersofthestringsarecompared.1islowerthan9andtheresultistherefore"correct".Ifwewantedtousethenumericalvalueofthe_keyattributesinstead,wecouldconvertthestringtoanumberanduseitforsorting.Therearesomeimplicationshowever.Wearebetteroffsortingsomethingelse.Howabouttheage,indescendingorder?
GettingStarted
13
FORuserINusers
SORTuser.ageDESC
RETURNuser
Theuserswillbereturnedinthefollowingorder:James(69),John(32),Katie(27).InsteadofDESCfordescendingorder,ASCcanbeusedforascendingorder.ASCisthedefaultthoughandcanbeomitted.
Wemightwanttolimittheresultsettoasubsetofusers,basedontheageattributeforexample.Let'sreturnusersolderthan30only:
FORuserINusers
FILTERuser.age>30
SORTuser.age
RETURNuser
ThiswillreturnJohnandJames(inthisorder).Katie'sageattributedoesnotfulfillthecriterion(greaterthan30),sheisonly27andthereforenotpartoftheresultset.Wecanmakeheragetoreturnheruserdocumentagain,usingamodificationquery:
UPDATE"9915"WITH{age:40}INusers
RETURNNEW
UPDATEallowstopartiallyeditanexistingdocument.ThereisalsoREPLACE,whichwouldremoveallattributes(exceptfor_keyand_id,whichremainthesame)andonlyaddthespecifiedones.UPDATEontheotherhandonlyreplacesthespecifiedattributesandkeepseverythingelseas-is.
TheUPDATEkeywordisfollowedbythedocumentkey(oradocument/objectwitha_keyattribute)toidentifywhattomodify.TheattributestoupdatearewrittenasobjectaftertheWITHkeyword.INdenotesinwhichcollectiontoperformthisoperationin,justlikeINTO(bothkeywordsareactuallyinterchangablehere).ThefulldocumentwiththechangesappliedisreturnedifweusetheNEWpseudo-variable:
[
{
"_key":"9915",
"_id":"users/9915",
"_rev":"12864",
"age":40,
"name":"KatieFoster"
}
IfweusedREPLACEinstead,thenameattributewouldbegone.WithUPDATE,theattributeiskept(thesamewouldapplytoadditionalattributesifwehadthem).
LetusrunourFILTERqueryagain,butonlyreturntheusernamesthistime:
FORuserINusers
FILTERuser.age>30
SORTuser.age
RETURNuser.name
Thiswillreturnthenamesofall3users:
[
"JohnSmith",
"KatieFoster",
"JamesHendrix"
]
Itiscalledaprojectionifonlyasubsetofattributesisreturned.Anotherkindofprojectionistochangethestructureoftheresults:
FORuserINusers
RETURN{userName:user.name,age:user.age}
GettingStarted
14
Thequerydefinestheoutputformatforeveryuserdocument.TheusernameisreturnedasuserNameinsteadofname,theagekeepstheattributekeyinthisexample:
[
{
"userName":"JamesHendrix",
"age":69
},
{
"userName":"JohnSmith",
"age":32
},
{
"userName":"KatieFoster",
"age":40
}
]
Itisalsopossibletocomputenewvalues:
FORuserINusers
RETURNCONCAT(user.name,"'sageis",user.age)
CONCAT()isafunctionthatcanjoinelementstogethertoastring.Weuseitheretoreturnastatementforeveryuser.Asyoucansee,theresultsetdoesnotalwayshavetobeanarrayofobjects:
[
"JamesHendrix'sageis69",
"JohnSmith'sageis32",
"KatieFoster'sageis40"
]
Nowlet'sdosomethingcrazy:foreverydocumentintheuserscollection,iterateoveralluserdocumentsagainandreturnuserpairs,e.g.JohnandKatie.Wecanusealoopinsidealoopforthistogetthecrossproduct(everypossiblecombinationofalluserrecords,3*3=9).Wedon'twantpairingslikeJohn+Johnhowever,solet'seliminatethemwithafiltercondition:
FORuser1INusers
FORuser2INusers
FILTERuser1!=user2
RETURN[user1.name,user2.name]
Weget6pairings.PairslikeJames+JohnandJohn+Jamesarebasicallyredundant,butfairenough:
[
["JamesHendrix","JohnSmith"],
["JamesHendrix","KatieFoster"],
["JohnSmith","JamesHendrix"],
["JohnSmith","KatieFoster"],
["KatieFoster","JamesHendrix"],
["KatieFoster","JohnSmith"]
]
Wecouldcalculatethesumofbothagesandcomputesomethingnewthisway:
FORuser1INusers
FORuser2INusers
FILTERuser1!=user2
RETURN{
pair:[user1.name,user2.name],
sumOfAges:user1.age+user2.age
}
WeintroduceanewattributesumOfAgesandaddupbothagesforthevalue:
[
GettingStarted
15
{
"pair":["JamesHendrix","JohnSmith"],
"sumOfAges":101
},
{
"pair":["JamesHendrix","KatieFoster"],
"sumOfAges":109
},
{
"pair":["JohnSmith","JamesHendrix"],
"sumOfAges":101
},
{
"pair":["JohnSmith","KatieFoster"],
"sumOfAges":72
},
{
"pair":["KatieFoster","JamesHendrix"],
"sumOfAges":109
},
{
"pair":["KatieFoster","JohnSmith"],
"sumOfAges":72
}
]
Ifwewantedtopost-filteronthenewattributetoonlyreturnpairswithasumlessthan100,weshoulddefineavariabletotemporarilystorethesum,sothatwecanuseitinaFILTERstatementaswellasintheRETURNstatement:
FORuser1INusers
FORuser2INusers
FILTERuser1!=user2
LETsumOfAges=user1.age+user2.age
FILTERsumOfAges<100
RETURN{
pair:[user1.name,user2.name],
sumOfAges:sumOfAges
}
TheLETkeywordisfollowedbythedesignatedvariablename(sumOfAges),thenthere'sa=symbolandthevalueoranexpressiontodefinewhatvaluethevariableissupposedtohave.Were-useourexpressiontocalculatethesumhere.WethenhaveanotherFILTERtoskiptheunwantedpairingsandmakeuseofthevariablewedeclaredbefore.Wereturnaprojectionwithanarrayoftheusernamesandthecalculatedage,forwhichweusethevariableagain:
[
{
"pair":["JohnSmith","KatieFoster"],
"sumOfAges":72
},
{
"pair":["KatieFoster","JohnSmith"],
"sumOfAges":72
}
]
Protip:whendefiningobjects,ifthedesiredattributekeyandthevariabletousefortheattributevaluearethesame,youcanuseashorthandnotation:{sumOfAges}insteadof{sumOfAges:sumOfAges}.
Finally,let'sdeleteoneoftheuserdocuments:
REMOVE"9883"INusers
ItdeletestheuserJohn(_key:"9883").Wecouldalsoremovedocumentsinaloop(samegoesforINSERT,UPDATEandREPLACE):
FORuserINusers
FILTERuser.age>=30
REMOVEuserINusers
GettingStarted
16
Thequerydeletesalluserswhoseageisgreaterthanorequalto30.
Howtocontinue
ThereisalotmoretodiscoverinAQLandmuchmorefunctionalitythatArangoDBoffers.Continuereadingtheotherchaptersandexperimentwithatestdatabasetofosteryourknowledge.
IfyouwanttowritemoreAQLqueriesrightnow,havealookhere:
DataQueries:dataaccessandmodificationqueriesHigh-leveloperations:detaileddescriptionsofFOR,FILTERandmoreoperationsnotshowninthisintroductionFunctions:areferenceofallprovidedfunctions
ArangoDBprogramsTheArangoDBpackagecomeswiththefollowingprograms:
arangod:TheArangoDBdatabasedaemon.ThisserverprogramisintendedtorunasadaemonprocessandtoservethevariousclientsconnectiontotheserverviaTCP/HTTP.
arangosh:TheArangoDBshell.Aclientthatimplementsaread-eval-printloop(REPL)andprovidesfunctionstoaccessandadministratetheArangoDBserver.
arangoimp:AbulkimporterfortheArangoDBserver.ItsupportsJSONandCSV.
arangodump:AtooltocreatebackupsofanArangoDBdatabaseinJSONformat.
arangorestore:AtooltoloaddataofabackupbackintoanArangoDBdatabase.
arango-dfdb:AdatafiledebuggerforArangoDB.ItisprimarilyintendedtobeusedduringdevelopmentofArangoDB.
arangobench:Abenchmarkandtesttool.Itcanbeusedforperformanceandserverfunctiontesting.
GettingStarted
17
InstallingFirstofall,downloadandinstallthecorrespondingRPMorDebianpackageorusehomebrewonMacOSX.Youcanfindpackagesforvariousoperationsystemsatourinstallsection,includinginstallersforWindows.
Howtodothatindetailisdescribedinthesubchaptersofthissection.
Onhowtosetupacluster,checkouttheDeploymentchapter.
Installing
18
LinuxVisittheofficialArangoDBdownloadpageanddownloadthecorrectpackageforyourLinuxdistribution.Youcanfindbinarypackagesforthemostcommondistributionsthere.Followtheinstructionstouseyourfavoritepackagemanagerforthemajordistributions.AftersettinguptheArangoDBrepositoryyoucaneasilyinstallArangoDBusingyum,aptitude,urpmiorzypper.Debianbasedpackageswillaskforapasswordduringinstallation.ForanunattendedinstallationforDebian,seebelow.Red-Hatbasedpackageswillsetarandompasswordduringinstallation.Forotherdistributionsortochangethepassword,runarango-secure-installationtosetarootpassword.Alternatively,seeCompilingifyouwanttobuildArangoDByourself.Startupthedatabaseserver.
Normally,thisisdonebyexecutingthefollowingcommand:
unix>/etc/init.d/arangodstart
Itwillstarttheserver,anddothataswellatsystemboottime.
Tostoptheserveryoucanusethefollowingcommand:
unix>/etc/init.d/arangodstop
TheexactcommandsdependonyourLinuxdistribution.Youmayrequirerootprivilegestoexecutethesecommands.
LinuxMint
PleaseusethecorrespondingUbuntuorDebianpackages.
UnattendedInstallation
Debianbasedpackagewillaskforapasswordduringinstallation.Forunattendedinstallation,youcansetthepasswordusingthedebconfhelpers.
echoarangodb3arangodb3/passwordpasswordNEWPASSWORD|debconf-set-selections
echoarangodb3arangodb3/password_againpasswordNEWPASSWORD|debconf-set-selections
Thecommandsshouldbeexecutedpriortotheinstallation.
Red-Hatbasedpackageswillsetarandompasswordduringinstallation.Ifyouwanttoforceapassword,execute
ARANGODB_DEFAULT_ROOT_PASSWORD=NEWPASSWORDarango-secure-installation
Thecommandshouldbeexecutedaftertheinstallation.
Non-StandardInstallationIfyoucompiledArangoDBfromsourceanddidnotuseanyinstallationpackage–orusingnon-defaultlocationsand/ormultipleArangoDBinstancesonthesamehost–youmaywanttostarttheserverprocessmanually.Youcandosobyinvokingthearangodbinaryfromthecommandlineasshownbelow:
unix>/usr/local/sbin/arangod/tmp/vocbase
20ZZ-XX-YYT12:37:08Z[8145]INFOusingbuilt-inJavaScriptstartupfiles
20ZZ-XX-YYT12:37:08Z[8145]INFOArangoDB(version1.x.y)isreadyforbusiness
20ZZ-XX-YYT12:37:08Z[8145]INFOHaveFun!
Linux
19
Tostopthedatabaseservergracefully,youcaneitherpressCTRL-CorbysendtheSIGINTsignaltotheserverprocess.Onmanysystemsthiscanbeachievedwiththefollowingcommand:
unix>kill-2`pidofarangod`
Onceyoustartedtheserver,thereshouldbearunninginstanceofarangod-theArangoDBdatabaseserver.
unix>psauxw|fgreparangod
arangodb145360.10.6530726423464s002S1:21pm0:00.18/usr/local/sbin/arangod
Ifthereisnosuchprocess,checkthelogfile/var/log/arangodb/arangod.logforerrors.Ifyouseealogmessagelike
2012-12-03T11:35:29Z[12882]ERRORDatabasedirectoryversion(1)islowerthanserverversion(1.2).
2012-12-03T11:35:29Z[12882]ERRORItseemslikeyouhaveupgradedtheArangoDBbinary.Ifthisiswhatyouwantedtodo,pleas
erestartwiththe--database.auto-upgradeoptiontoupgradethedatainthedatabasedirectory.
2012-12-03T11:35:29Z[12882]FATALDatabaseversioncheckfailed.Pleasestarttheserverwiththe--database.auto-upgradeopti
on
makesuretostarttheserveroncewiththe--database.auto-upgradeoption.
Notethatyoumayhavetoenableloggingfirst.Ifyoustarttheserverinashell,youshouldseeerrorsloggedthereaswell.
Linux
20
MacOSXThepreferredmethodforinstallingArangoDBunderMacOSXishomebrew.However,incaseyouarenotusinghomebrew,weprovideacommand-lineapporgraphicalappwhichcontainsalltheexecutables.
Homebrew
Ifyouareusinghomebrew,thenyoucaninstallthelatestreleasedstableversionofArangoDBusingbrewasfollows:
brewinstallarangodb
ThiswillinstallthecurrentstableversionofArangoDBandalldependencieswithinyourHomebrewtree.Notethattheserverwillbeinstalledas:
/usr/local/sbin/arangod
Youcanstarttheserverbyrunningthecommand/usr/local/sbin/arangod&.
Configurationfileislocatedat
/usr/local/etc/arangodb3/arangod.conf
TheArangoDBshellwillbeinstalledas:
/usr/local/bin/arangosh
YoucanuninstallArangoDBusing:
brewuninstallarangodb
However,incaseyoustartedArangoDBusingthelaunchctl,youneedtounloaditbeforeuninstallingtheserver:
launchctlunload~/Library/LaunchAgents/homebrew.mxcl.arangodb.plist
ThenremovetheLaunchAgent:
rm~/Library/LaunchAgents/homebrew.mxcl.arangodb.plist
Note:IfthelatestArangoDBVersionisnotshowninhomebrew,youalsoneedtoupdatehomebrew:
brewupdate
Knownissues
Performance-theLLVMdeliveredasofMacOSXElCapitanbuildsslowbinaries.UseGCCinstead,untilthisissuehasbeenfixedbyApple.theCommandlineargumentparsingdoesn'tacceptblanksinfilenames;theCLIversionbelowdoes.ifyouneedtochangeserverendpointwhilestartinghomebrewversion,youcaneditarangod.conffileanduncommentlinewithendpointneeded,e.g.:
[server]
endpoint=tcp://0.0.0.0:8529
MacOSX
21
GraphicalApp
Incaseyouarenotusinghomebrew,wealsoprovideagraphicalapp.Youcandownloaditfromhere.
ChooseMacOSX.DownloadandinstalltheapplicationArangoDBinyourapplicationfolder.
CommandlineAppIncaseyouarenotusinghomebrew,wealsoprovideacommand-lineapp.Youcandownloaditfromhere.
ChooseMacOSX.DownloadandinstalltheapplicationArangoDB-CLIinyourapplicationfolder.
Startingtheapplicationwillstarttheserverandopenaterminalwindowshowingyouthelog-file.
ArangoDBserverhasbeenstarted
Thedatabasedirectoryislocatedat
'/Applications/ArangoDB-CLI.app/Contents/MacOS/opt/arangodb/var/lib/arangodb'
Thelogfileislocatedat
'/Applications/ArangoDB-CLI.app/Contents/MacOS/opt/arangodb/var/log/arangodb/arangod.log'
Youcanaccesstheserverusingabrowserat'http://127.0.0.1:8529/'
orstarttheArangoDBshell
'/Applications/ArangoDB-CLI.app/Contents/MacOS/arangosh'
Switchingtolog-filenow,killingthiswindowswillNOTstoptheserver.
2013-10-27T19:42:04Z[23840]INFOArangoDB(version1.4.devel[darwin])isreadyforbusiness.Havefun!
Notethatitispossibletoinstallboth,thehomebrewversionandthecommand-lineapp.Youshould,however,edittheconfigurationfilesofoneversionandchangetheportused.
MacOSX
22
WindowsThedefaultinstallationdirectoryisC:\ProgramFiles\ArangoDB-3.x.x.Duringtheinstallationprocessyoumaychangethis.InthefollowingdescriptionwewillassumethatArangoDBhasbeeninstalledinthelocation<ROOTDIR>.
Youhavetobecarefulwhenchoosinganinstallationdirectory.Youneedeitherwritepermissiontothisdirectoryoryouneedtomodifytheconfigurationfilefortheserverprocess.InthelattercasethedatabasedirectoryandtheFoxxdirectoryhavetobewritablebytheuser.
Single-andMultiuserInstallation
TherearetwomainmodesfortheinstallerofArangoDB.Theinstallerletsyouselect:
multiuserinstallation(default;adminprivilegesrequired)WillinstallArangoDBasservice.singleuserinstallationAllowtoinstallArangodbasnormaluser.Requiresmanualstartingofthedatabaseserver.
CheckBoxesThecheckboxesallowyoutochoseweatheryouwantto:
chosecustominstallpathsdoanautomaticupgradekeepanbackupofyourdataaddexecutablestopathcreateadesktopicon
ornot.
CustomInstallPaths
Thischeckboxcontrolsifyouwillbeabletooverridethedefaultpathsfortheinstallationinsubsequentsteps.
Thedefaultinstallationpathsare:
MultiUserDefault:
Installation:C:\ProgramFiles\ArangoDB-3.x.xDataBase:C:\ProgramData\ArangoDBFoxxService:C:\ProgramData\ArangoDB-apps
SingleUserDefault:
Installation:C:\Users\\\AppData\Local\ArangoDB-3.x.xDataBase:C:\Users\\\AppData\Local\ArangoDBFoxxService:C:\Users\\\AppData\Local\ArangoDB-apps
Wearenotusingtheroamingpartoftheuser'sprofile,becausedoingsoavoidsthedatabeingsyncedtothewindowsdomaincontroller.
AutomaticUpgrade
Ifthischeckboxisselectedtheinstallerwillattempttoperformanautomaticupdate.FormoreinformationpleaseseeUpgradingfromPreviousVersion.
KeepBackup
Selectthistocreateabackupofyourdatabasedirectoryduringautomaticupgrade.Thebackupwillbecreatednexttoyourcurrentdatabasedirectorysuffixedbyatimestamp.
Windows
23
AddtoPath
Selectthistoaddthebinarydirectorytoyoursystem'spath(multiuserinstallation)oruser'spath(singleuserinstallation).
DesktopIcon
SelectifyouwanttheinstallertocreateDesktopIconsthatletyou:
accessthewebintefacestartthecommandlineclient(arangosh)startthedatabaseserver(singleuserinstallationonly)
UpgradingfromPreviousVersionIfyouareupgradingArangoDBfromanearlierversionyouneedtocopyyourolddatabasedirectorytothenewdefaultpaths.Upgradingwillkeepyourolddata,passwordandchoiceofstorageengineasitis.SwitchingtotheRocksDBstorageenginerequiresaexportandreimportofyourdata.
StartingIfyouinstalledArangoDBformultipleusers(asaservice)itisautomaticallystarted.OtherwiseyouneedtousethelinkthatwascreatedonyouDesktopifyouchosetolettheinstallercreatedesktopiconsor
theexecutablearangod.exelocatedin<ROOTDIR>\bin.Thiswillusetheconfigurationfilearangod.conflocatedin<ROOTDIR>\etc\arangodb,whichyoucanadjusttoyourneedsandusethedatadirectory<ROOTDIR>\var\lib\arangodb.Thisistheplacewhereallyourdata(databasesandcollections)willbestoredbydefault.
Pleasechecktheoutputofthearangod.exeexecutablebeforegoingon.Iftheserverstartedsuccessfully,youshouldseealineArangoDBisreadyforbusiness.Havefun!attheendofitsoutput.
Wenowwishtocheckthattheinstallationisworkingcorrectlyandtodothiswewillbeusingtheadministrationwebinterface.Executearangod.exeifyouhavenotalreadydoneso,thenopenupyourwebbrowserandpointittothepage:
http://127.0.0.1:8529/
AdvancedStarting
Ifyouwanttoprovideourownstartscripts,youcansettheenvironmentvariableARANGODB_CONFIG_PATH.Thisvariableshouldpointtoadirectorycontainingtheconfigurationfiles.
UsingtheClient
ToconnecttoanalreadyrunningArangoDBserverinstance,thereisashellarangosh.exelocatedin<ROOTDIR>\bin.Thisstartsashellwhichcanbeused–amongstotherthings–toadministerandqueryalocalorremoteArangoDBserver.
Notethatarangosh.exedoesNOTstartaseparateserver,itonlystartstheshell.Touseityoumusthaveaserverrunningsomewhere,e.g.byusingthearangod.exeexecutable.
arangosh.exeusesconfigurationfromthefilearangosh.conflocatedin<ROOTDIR>\etc\arangodb\.Pleaseadjustthistoyourneedsifyouwanttousedifferentconnectionsettingsetc.
Uninstalling
TouninstalltheArangoserverapplicationyoucanusethewindowscontrolpanel(asyouwouldnormallyuninstallanapplication).Notehowever,thatanydatafilescreatedbytheArangoserverwillremainaswellasthe<ROOTDIR>directory.Tocompletetheuninstallationprocess,removethedatafilesandthe<ROOTDIR>directorymanually.
Windows
24
LimitationsforCygwin
PleasenotesomeimportantlimitationswhenrunningArangoDBunderCygwin:StartingArangoDBcanbestartedfromoutofaCygwinterminal,butpressingCTRL-Cwillforcefullykilltheserverprocesswithoutgivingitachancetohandlethekillsignal.Inthiscase,aregularservershutdownisnotpossible,whichmayleaveafileLOCKaroundintheserver'sdatadirectory.ThisfileneedstoberemovedmanuallytomakeArangoDBstartagain.Additionally,asArangoDBdoesnothaveachancetohandlethekillsignal,theservercannotforcefullyflushanydatatodiskonshutdown,leadingtopotentialdataloss.WhenstartingArangoDBfromaCygwinterminalitmightalsohappenthatnoerrorsareprintedintheterminaloutput.StartingArangoDBfromanMS-DOScommandpromptdoesnotimposetheselimitationsandisthusthepreferredmethod.
PleasenotethatArangoDBusesUTF-8asitsinternalencodingandthatthesystemconsolemustsupportaUTF-8codepage(65001)andfont.ItmaybenecessarytomanuallyswitchtheconsolefonttoafontthatsupportsUTF-8.
Windows
25
CompilingArangoDBfromscratchThefollowingsectionsdescribehowtocompileandbuildtheArangoDBfromscratch.ArangoDBwillcompileonmostLinuxandMacOSXsystems.WeassumethatyouusetheGNUC/C++compilerorclang/clang++tocompilethesource.ArangoDBhasbeentestedwiththesecompilers,butshouldbeabletocompilewithanyPosix-compliant,C++11-enabledcompiler.PleaseletusknowwhetheryousuccessfullycompileditwithanotherC/C++compiler.
Bydefault,cloningthegithubrepositorywillcheckoutdevel.ThisversioncontainsthedevelopmentversionoftheArangoDB.UsethisbranchifyouwanttomakechangestotheArangoDBsource.
OnWindowsyoufirstneedtoallowandenablesymlinksforyouruser.
PleasecheckoutthecookbookonhowtocompileArangoDB.
Compiling
26
AuthenticationArangoDBallowstorestrictaccesstodatabasestocertainusers.Allusersofthesystemdatabaseareconsideredadministrators.Duringinstallationadefaultuserrootiscreated,whichhasaccesstoalldatabases.
Youshouldcreateadatabaseforyourapplicationtogetherwithauserthathasaccessrightstothisdatabase.SeeManagingUsers.
Usethearangoshtocreateanewdatabaseanduser.
arangosh>db._createDatabase("example");
arangosh>varusers=require("@arangodb/users");
arangosh>users.save("root@example","password");
arangosh>users.grantDatabase("root@example","example");
Youcannowconnecttothenewdatabaseusingtheuserroot@example.
shell>arangosh--server.username"root@example"--server.databaseexample
Authentication
27
AccessingtheWebInterfaceArangoDBcomeswithabuilt-inwebinterfaceforadministration.ThewebinterfacecanbeaccessedviatheURL:
http://127.0.0.1:8529
Ifeverythingworksasexpected,youshouldseetheloginview:
FormoreinformationontheArangoDBwebinterface,seeWebInterface
AccessingtheWebInterface
28
ComingfromSQLIfyouworkedwitharelationaldatabasemanagementsystem(RDBMS)suchasMySQL,MariaDBorPostgreSQL,youwillbefamiliarwithitsquerylanguage,adialectofSQL(StructuredQueryLanguage).
ArangoDB'squerylanguageiscalledAQL.Therearesomesimilaritiesbetweenbothlanguagesdespitethedifferentdatamodelsofthedatabasesystems.ThemostnotabledifferenceisprobablytheconceptofloopsinAQL,whichmakesitfeelmorelikeaprogramminglanguage.Itsuitstheschema-lessmodelmorenaturalandmakesthequerylanguageverypowerfulwhileremainingeasytoreadandwrite.
TogetstartedwithAQL,havealookatourdetailedcomparisonofSQLandAQL.ItwillalsohelpyoutotranslateSQLqueriestoAQLwhenmigratingtoArangoDB.
Howdobrowsevectorstranslateintodocumentqueries?
IntraditionalSQLyoumayeitherfetchallcolumnsofatablerowbyrow,usingSELECT*FROMtable,orselectasubsetofthecolumns.Thelistoftablecolumnstofetchiscommonlycalledcolumnlistorbrowsevector:
SELECTcolumnA,columnB,columnZFROMtable
Sincedocumentsaren'ttwo-dimensional,andneitherdoyouwanttobelimitedtoreturningtwo-dimensionallists,therequirementsforaquerylanguagearehigher.AQListhusalittlebitmorecomplexthanplainSQLatfirst,butoffersmuchmoreflexibilityinthelongrun.Itletsyouhandlearbitrarilystructureddocumentsinconvenientways,mostlyleanedonthesyntaxusedinJavaScript.
Composingthedocumentstobereturned
TheAQLRETURNstatementreturnsoneitemperdocumentitishanded.Youcanreturnthewholedocument,orjustpartsofit.GiventhatoneDocumentisadocument(retrievedlikeLEToneDocument=DOCUMENT("myusers/3456789")forinstance),itcanbereturnedas-islikethis:
RETURNoneDocument
[
{
"_id":"myusers/3456789",
"_key":"3456789"
"_rev":"14253647",
"firstName":"John",
"lastName":"Doe",
"address":{
"city":"Gotham",
"street":"RoadToNowhere1"
},
"hobbies":[
{name:"swimming",howFavorite:10},
{name:"biking",howFavorite:6},
{name:"programming",howFavorite:4}
]
}
]
Returnthehobbiessub-structureonly:
RETURNoneDocument.hobbies
[
[
{name:"swimming",howFavorite:10},
{name:"biking",howFavorite:6},
{name:"programming",howFavorite:4}
]
ComingfromSQL
29
]
Returnthehobbiesandtheaddress:
RETURN{
hobbies:oneDocument.hobbies,
address:oneDocument.address
}
[
{
hobbies:[
{name:"swimming",howFavorite:10},
{name:"biking",howFavorite:6},
{name:"programming",howFavorite:4}
],
address:{
"city":"Gotham",
"street":"RoadToNowhere1"
}
}
]
Returnthefirsthobbyonly:
RETURNoneDocument.hobbies[0].name
[
"swimming"
]
Returnalistofallhobbystrings:
RETURN{hobbies:oneDocument.hobbies[*].name}
[
{hobbies:["swimming","biking","porgramming"]}
]
MorecomplexarrayandobjectmanipulationscanbedoneusingAQLfunctionsandoperators.
ComingfromSQL
30
StartArangoDBonKubernetesin5minutesStartinganArangoDBdatabase(eithersingleserverorfullblowncluster)onKubernetesinvolvesalotofresources.
TheserversneedstoruninPods,youneedSecretsforauthentication,TLScertificatesandServicestoenablecommunicationwiththedatabase.
Usekube-arangodb,theArangoDBKubernetesOperatortogreatlysimplifythisprocess.
Inthisguide,wewillexplainwhattheArangoDBKubernetesOperatoris,howtoinstallitandhowuseittodeployyourfirstArangoDBdatabaseinaKubernetescluster.
Whatis kube-arangodb
kube-arangodbisasetoftwooperatorsthatyoudeployinyourKubernetesclusterto(1)managedeploymentsoftheArangoDBdatabaseand(2)providePersistentVolumesonlocalstorageofyournodesforoptimalstorageperformance.
NotethattheoperatorthatprovidesPersistentVolumesisnotneededtorunArangoDBdeployments.YoucanalsousePersistentVolumesprovidedbyothercontrollers.
InthisguidewewillfocusontheArangoDeploymentoperator.
Installing kube-arangodb
Toinstallkube-arangodbinyourKubernetescluster,makesureyouhaveaccesstothisclusterandtherightstodeployresourcesatclusterlevel.
Fornow,anyrecentKubernetesclusterwilldo(e.g.minikube).
Thenrun(replace<version>withtheversionoftheoperatorthatyouwanttoinstall):
kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb/<version>/manifests/crd.yaml
kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb/<version>/manifests/arango-deployment.yaml
#Optional
kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb/<version>/manifests/arango-storage.yaml
ThefirstcommandinstallstwoCustomResourceDefinitionsinyourKubernetescluster:
ArangoDeploymentistheresourceusedtodeployArangoDBdatabase.ArangoLocalStorageistheresourceusedtoprovisionPersistentVolumesonlocalstorage.
ThesecondcommandinstallsaDeploymentthatrunstheoperatorthatcontrolsArangoDeploymentresources.
TheoptionalthirdcommandinstallsaDeploymentthatrunstheoperatorthatprovidesPersistentVolumesonlocaldisksoftheclusternodes.Usethiswhenrunningonbare-metalorifthereisnoprovisionerforfaststorageinyourKubernetescluster.
DeployingyourfirstArangoDBdatabaseThefirstdatabasewearegoingtodeployisasingleserverdatabase.
Createafilecalledsingle-server.yamlwiththefollowingcontent.
apiVersion:"database.arangodb.com/v1alpha"
kind:"ArangoDeployment"
metadata:
name:"single-server"
spec:
mode:Single
Kubernetes
32
NowinsertthisresourceinyourKubernetesclusterusing:
kubectlapply-fsingle-server.yaml
TheArangoDeploymentoperatorinkube-arangodbwillnowinspecttheresourceyoujustdeployedandstarttheprocesstorunasingleserverdatabase.
Toinspectthecurrentstatusofyourdeployment,run:
kubectldescribeArangoDeploymentsingle-server
#orshorter
kubectldescribearangosingle-server
Toinspectthepodscreatedforthisdeployment,run:
kubectlgetpods--selector=arango_deployment=single-server
Theresultwilllooksimilartothis:
NAMEREADYSTATUSRESTARTSAGE
single-server-sngl-cjtdxrgl-fe06f01/1Running01m
OncethepodreportsthatitishasaRunningstatusandisready,yourdatabasesavailable.
Connectingtoyourdatabase
ThesingleserverdatabaseyoudeployedinthepreviouschapterisnowavailablefromwithintheKubernetesclusteraswellasoutsideit.
AccesstothedatabasefromoutsidetheKubernetesclusterisprovidedusinganexternal-accessservice.BydefaultthisserviceisoftypeLoadBalancer.IfthistypeofserviceisnotsupportedbyyourKubernetescluster,itwillbereplacedbyaserviceoftypeNodePortafteraminute.
Toseethetypeofservicethathasbeencreated,run:
kubectlgetservicesingle-server-ea
WhentheserviceisoftheLoadBalancertype,usetheIPaddresslistedintheEXTERNAL-IPcolumnwithport8529.WhentheserviceisoftheNodePorttype,usetheIPaddressofanyofthenodesofthecluster,combinewiththehigh(>30000)portlistedinthePORT(S)column.
Nowyoucanconnectyourbrowsertohttps://<ip>:<port>/.
Yourbrowserwillshowawarningaboutanunknowncertificate.Acceptthecertificatefornow.
Thenloginusingusernamerootandanemptypassword.
IfyouwanttodeleteyoursingleserverArangoDBdatabase,justrun:
kubectldeleteArangoDeploymentsingle-server
DeployingafullblownArangoDBclusterdatabase
Thedeploymentofafullblownclusterisverysimilartodeployingasingleserverdatabase.ThedifferenceisinthemodefieldoftheArangoDeploymentspecification.
Createafilecalledcluster.yamlwiththefollowingcontent.
apiVersion:"database.arangodb.com/v1alpha"
kind:"ArangoDeployment"
Kubernetes
33
metadata:
name:"cluster"
spec:
mode:Cluster
NowinsertthisresourceinyourKubernetesclusterusing:
kubectlapply-fcluster.yaml
Thesamecommandsusedinthesingleserverdeploymentcanbeusedtoinspectyourcluster.Justusethecorrectdeploymentname(clusterinsteadofsingle-server).
Wheretogofromhere
ArangoDBKubernetesOperator
Kubernetes
34
StartArangoDBClustertoClusterSynchronizationonKubernetesThistutorialguidesyouthroughthestepsneededtoconfigureanArangoDBdatacentertodatacenterreplicationbetweentwoArangoDBclustersrunninginKubernetes.
Requirements
1. Thistutorialassumesthatyouhave2ArangoDBclustersrunningin2differentKubernetesclusters.2. BothKubernetesclustersareequippedwithsupportforServicesoftypeLoadBalancer.3. Youcancreate(global)DNSnamesforconfiguredServiceswithlowpropagationtimes.E.g.useCloudflare.4. Youhave4DNSnamesavailable:
OneforthedatabaseinthesourceArangoDBcluster.E.g.src-db.mycompany.comOnefortheArangoDBsyncmastersinthesourceArangoDBcluster.E.g.src-sync.mycompany.comOneforthedatabaseinthedestinationArangoDBcluster.E.g.dst-db.mycompany.comOnefortheArangoDBsyncmastersinthedestinationArangoDBcluster.E.g.dst-sync.mycompany.com
Step1:EnableDatacenterReplicationSupportonsourceArangoDBcluster
SetyourcurrentKubernetescontexttotheKubernetessourcecluster.
EdittheArangoDeploymentofthesourceArangoDBclusters.
Set:
spec.tls.altNamesto["src-db.mycompany.com"](canincludemorenames/IPaddresses)spec.sync.enabledtotruespec.sync.externalAccess.masterEndpointto["https://src-sync.mycompany.com:8629"]spec.sync.externalAccess.accessPackageSecretNamesto["src-accesspackage"]
Step2:Extractaccess-packagefromsourceArangoDBcluster
Run:
kubectlgetsecretsrc-accesspackage--template='{{index.data"accessPackage.yaml"}}'|\
base64-D>accessPackage.yaml
Step3:ConfiguresourceDNSnames
Run:
kubectlgetservice
FindtheIPaddresscontainedintheLoadBalancercolumnforthefollowingServices:
<deployment-name>-eaUsethisIPaddressforthesrc-db.mycompany.comDNSname.<deployment-name>-syncUsethisIPaddressforthesrc-sync.mycompany.comDNSname.
TheprocessforconfiguringDNSnamesisspecifictoeachDNSprovider.
DatacentertodatacenterReplicationonKubernetes
35
Step4:EnableDatacenterReplicationSupportondestinationArangoDBcluster
SetyourcurrentKubernetescontexttotheKubernetesdestinationcluster.
EdittheArangoDeploymentofthesourceArangoDBclusters.
Set:
spec.tls.altNamesto["dst-db.mycompany.com"](canincludemorenames/IPaddresses)spec.sync.enabledtotruespec.sync.externalAccess.masterEndpointto["https://dst-sync.mycompany.com:8629"]
Step5:ImportaccesspackageindestinationclusterRun:
kubectlapply-faccessPackage.yaml
Note:ThisimportstwoSecrets,containingTLSinformationaboutthesourcecluster,intothedestinationcluster
Step6:ConfiguredestinationDNSnames
Run:
kubectlgetservice
FindtheIPaddresscontainedintheLoadBalancercolumnforthefollowingServices:
<deployment-name>-eaUsethisIPaddressforthedst-db.mycompany.comDNSname.<deployment-name>-syncUsethisIPaddressforthedst-sync.mycompany.comDNSname.
TheprocessforconfiguringDNSnamesisspecifictoeachDNSprovider.
Step7:Createan ArangoDeploymentReplicationresourceCreateayamlfile(e.g.calledsrc-to-dst-repl.yaml)withthefollowingcontent:
apiVersion:"replication.database.arangodb.com/v1alpha"
kind:"ArangoDeploymentReplication"
metadata:
name:"replication-src-to-dst"
spec:
source:
masterEndpoint:["https://src-sync.mycompany.com:8629"]
auth:
keyfileSecretName:src-accesspackage-auth
tls:
caSecretName:src-accesspackage-ca
destination:
deploymentName:<dst-deployment-name>
Step8:WaitforDNSnamestopropagateWaituntiltheDNSnamesconfiguredinstep3and6resolvetotheirconfiguredIPaddresses.
DependingonyourDNSprovidesthiscantakeafewminutesupto24hours.
DatacentertodatacenterReplicationonKubernetes
36
Step9:Activatereplication
Run:
kubectlapply-fsrc-to-dst-repl.yaml
Replicationfromthesourceclustertothedestinationclusterwillnowbeconfigured.
CheckthestatusofthereplicationbyinspectingthestatusoftheArangoDeploymentReplicationresourceusing:
kubectldescribeArangoDeploymentReplicationreplication-src-to-dst
Assoonasthereplicationisconfigured,theAddcollectionbuttonintheCollectionspageofthewebUI(ofthedestinationcluster)willbegrayedout.
DatacentertodatacenterReplicationonKubernetes
37
Highlights
Version3.3
EnterpriseEdition
DatacentertoDatacenterReplication:ReplicatetheentirestructureandcontentofanArangoDBclusterasynchronouslytoanotherclusterinadifferentdatacenterwithArangoSync.Multi-datacentersupportmeansyoucanfallbacktoareplicaofyourclusterincaseofadisasterinonedatacenter.
EncryptedBackups:ArangodumpcancreatebackupsencryptedwithasecretkeyusingAES256blockcipher.
AllEditions
Server-levelReplication:Inadditiontoper-databasereplication,thereisnowanadditionalglobalApplier.Starttheglobalreplicationontheslaveonceandallcurrentandfuturedatabaseswillbereplicatedfromthemastertotheslaveautomatically.
AsynchronousFailover:Makeasingleserverinstanceresilientwithasecondserverinstance,oneasmasterandtheotherasasynchronouslyreplicatingslave,withautomaticfailovertotheslaveifthemastergoesdown.
AlsoseeWhat'sNewin3.3.
Version3.2
RocksDBStorageEngine:YoucannowuseasmuchdatainArangoDBasyoucanfitonyourdisk.Plus,youcanenjoyperformanceboostsonwritesbyhavingonlydocument-levellocks
Pregel:WeimplementeddistributedgraphprocessingwithPregeltodiscoverhiddenpatterns,identifycommunitiesandperformin-depthanalyticsoflargegraphdatasets.
Fault-TolerantFoxx:TheFoxxmanagementinternalshavebeenrewrittenfromthegrounduptomakesuremulti-coordinatorclustersetupsalwayskeeptheirservicesinsyncandnewcoordinatorsarefullyinitializedevenwhenallexistingcoordinatorsareunavailable.
Enterprise:Workingwithsomeofourlargestcustomers,we’veaddedfurthersecurityandscalabilityfeaturestoArangoDBEnterpriselikeLDAPintegration,EncryptionatRest,andthebrandnewSatelliteCollections.
AlsoseeWhat'sNewin3.2.
Version3.1SmartGraphs:Scalewithgraphstoaclusterandstayperformant.WithSmartGraphsyoucanusethe"smartness"ofyourapplicationlayertoshardyourgraphefficientlytoyourmachinesandlettraversalsrunlocally.
EncryptionControl:ChooseyourlevelofSSLencryption
Auditing:KeepadetailedlogofalltheimportantthingsthathappenedinArangoDB.
AlsoseeWhat'sNewin3.1.
Version3.0
self-organizingclusterwithsynchronousreplication,master/mastersetup,sharednothingarchitecture,clustermanagementagency.
Deeplyintegrated,nativeAQLgraphtraversal
Highlights
38
VelocyPackasnewinternalbinarystorageformataswellasforintermediateAQLvalues.
PersistentindexesviaRocksDBsuitableforsortingandrangequeries.
Foxx3.0:overhauledJSframeworkfordata-centricmicroservices
SignificantlyimprovedWebInterface
AlsoseeWhat'sNewin3.0.
Highlights
39
ScalabilityArangoDBisadistributeddatabasesupportingmultipledatamodels,andcanthusbescaledhorizontally,thatis,byusingmanyservers,typicallybasedoncommodityhardware.Thisapproachnotonlydeliversperformanceaswellascapacityimprovements,butalsoachievesresiliencebymeansofreplicationandautomaticfail-over.Furthermore,onecanbuildsystemsthatscaletheircapacitydynamicallyupanddownautomaticallyaccordingtodemand.
OnecanalsoscaleArangoDBvertically,thatis,byusingeverlargerservers.ThereisnobuiltinlimitationinArangoDB,forexample,theserverwillautomaticallyusemorethreadsifmoreCPUsarepresent.
However,scalingverticallyhasthedisadvantagethatthecostsgrowfasterthanlinearwiththesizeoftheserver,andnoneoftheresilienceanddynamicalcapabilitiescanbeachievedinthisway.
InthischapterweexplainthedistributedarchitectureofArangoDBanddiscussitsscalabilityfeaturesandlimitations:
ArangoDB'sdistributedarchitectureDifferentdatamodelsandscalabilityLimitations
Scalability
40
Architecture
TheclusterarchitectureofArangoDBisaCPmaster/mastermodelwithnosinglepointoffailure.With"CP"wemeanthatinthepresenceofanetworkpartition,thedatabaseprefersinternalconsistencyoveravailability.With"master/master"wemeanthatclientscansendtheirrequeststoanarbitrarynode,andexperiencethesameviewonthedatabaseregardless."Nosinglepointoffailure"meansthattheclustercancontinuetoserverequests,evenifonemachinefailscompletely.
Inthisway,ArangoDBhasbeendesignedasadistributedmulti-modeldatabase.Thissectiongivesashortoutlineontheclusterarchitectureandhowtheabovefeaturesandcapabilitiesareachieved.
StructureofanArangoDBcluster
AnArangoDBclusterconsistsofanumberofArangoDBinstanceswhichtalktoeachotheroverthenetwork.Theyplaydifferentroles,whichwillbeexplainedindetailbelow.Thecurrentconfigurationoftheclusterisheldinthe"Agency",whichisahighly-availableresilientkey/valuestorebasedonanoddnumberofArangoDBinstancesrunningRaftConsensusProtocol.
ForthevariousinstancesinanArangoDBclusterthereare4distinctroles:Agents,Coordinators,PrimaryandSecondaryDBservers.Inthefollowingsectionswewillshedlightoneachofthem.NotethatthetasksforallrolesrunthesamebinaryfromthesameDockerimage.
Agents
OneormultipleAgentsformtheAgencyinanArangoDBcluster.TheAgencyisthecentralplacetostoretheconfigurationinacluster.Itperformsleaderelectionsandprovidesothersynchronizationservicesforthewholecluster.WithouttheAgencynoneoftheothercomponentscanoperate.
Whilegenerallyinvisibletotheoutsideitistheheartofthecluster.Assuch,faulttoleranceisofcourseamusthavefortheAgency.ToachievethattheAgentsareusingtheRaftConsensusAlgorithm.ThealgorithmformallyguaranteesconflictfreeconfigurationmanagementwithintheArangoDBcluster.
AtitscoretheAgencymanagesabigconfigurationtree.Itsupportstransactionalreadandwriteoperationsonthistree,andotherserverscansubscribetoHTTPcallbacksforallchangestothetree.
Coordinators
Coordinatorsshouldbeaccessiblefromtheoutside.Thesearetheonestheclientstalkto.TheywillcoordinateclustertaskslikeexecutingqueriesandrunningFoxxservices.Theyknowwherethedataisstoredandwilloptimizewheretorunusersuppliedqueriesorpartsthereof.Coordinatorsarestatelessandcanthuseasilybeshutdownandrestartedasneeded.
PrimaryDBservers
PrimaryDBserversaretheoneswherethedataisactuallyhosted.Theyhostshardsofdataandusingsynchronousreplicationaprimarymayeitherbeleaderorfollowerforashard.
Theyshouldnotbeaccessedfromtheoutsidebutindirectlythroughthecoordinators.Theymayalsoexecutequeriesinpartorasawholewhenaskedbyacoordinator.
Secondaries
SecondaryDBserversareasynchronousreplicasofprimaries.Ifoneisusingonlysynchronousreplication,onedoesnotneedsecondariesatall.Foreachprimary,therecanbeoneormoresecondaries.Sincethereplicationworksasynchronously(eventualconsistency),thereplicationdoesnotimpedetheperformanceoftheprimaries.Ontheotherhand,theirreplicaofthedatacanbeslightlyoutofdate.Thesecondariesareperfectlysuitableforbackupsastheydon'tinterferewiththenormalclusteroperation.
ClusterID
Everynon-AgencyArangoDBinstanceinaclusterisassignedauniqueIDduringitsstartup.UsingitsIDanodeisidentifiablethroughoutthecluster.AllclusteroperationswillcommunicateviathisID.
Architecture
41
Sharding
UsingtherolesoutlinedaboveanArangoDBclusterisabletodistributedatainsocalledshardsacrossmultipleprimaries.Fromtheoutsidethisprocessisfullytransparentandassuchweachievethegoalsofwhatothersystemscall"master-masterreplication".InanArangoDBclusteryoutalktoanycoordinatorandwheneveryoureadorwritedataitwillautomaticallyfigureoutwherethedataisstored(read)ortobestored(write).TheinformationabouttheshardsissharedacrossthecoordinatorsusingtheAgency.
AlsoseeShardingintheAdministrationchapter.
Manysensibleconfigurations
Thisarchitectureisveryflexibleandthusallowsmanyconfigurations,whicharesuitablefordifferentusagescenarios:
1. ThedefaultconfigurationistorunexactlyonecoordinatorandoneprimaryDBserveroneachmachine.Thisachievestheclassicalmaster/mastersetup,sincethereisaperfectsymmetrybetweenthedifferentnodes,clientscanequallywelltalktoanyoneofthecoordinatorsandallexposethesameviewtothedatastore.
2. OnecandeploymorecoordinatorsthanDBservers.ThisisasensibleapproachifoneneedsalotofCPUpowerfortheFoxxservices,becausetheyrunonthecoordinators.
3. OnecandeploymoreDBserversthancoordinatorsifmoredatacapacityisneededandthequeryperformanceisthelesserbottleneck
4. Onecandeployacoordinatoroneachmachinewhereanapplicationserver(e.g.anode.jsserver)runs,andtheAgentsandDBserversonaseparatesetofmachineselsewhere.Thisavoidsanetworkhopbetweentheapplicationserverandthedatabaseandthusdecreaseslatency.Essentially,thismovessomeofthedatabasedistributionlogictothemachinewheretheclientruns.
Theseshallsufficefornow.TheimportantpieceofinformationhereisthatthecoordinatorlayercanbescaledanddeployedindependentlyfromtheDBserverlayer.
Replication
ArangoDBofferstwowaysofdatareplicationwithinacluster,synchronousandasynchronous.Inthissectionweexplainsomedetailsandhighlighttheadvantagesanddisadvantagesrespectively.
Synchronousreplicationwithautomaticfail-over
Synchronousreplicationworksonaper-shardbasis.Oneconfiguresforeachcollection,howmanycopiesofeachshardarekeptinthecluster.Atanygiventime,oneofthecopiesisdeclaredtobethe"leader"andallotherreplicasare"followers".WriteoperationsforthisshardarealwayssenttotheDBserverwhichhappenstoholdtheleadercopy,whichinturnreplicatesthechangestoallfollowersbeforetheoperationisconsideredtobedoneandreportedbacktothecoordinator.Readoperationsareallservedbytheserverholdingtheleadercopy,thisallowstoprovidesnapshotsemanticsforcomplextransactions.
IfaDBserverfailsthatholdsafollowercopyofashard,thentheleadercannolongersynchronizeitschangestothatfollower.Afterashorttimeout(3seconds),theleadergivesuponthefollower,declaresittobeoutofsync,andcontinuesservicewithoutthefollower.Whentheserverwiththefollowercopycomesback,itautomaticallyresynchronizesitsdatawiththeleaderandsynchronousreplicationisrestored.
IfaDBserverfailsthatholdsaleadercopyofashard,thentheleadercannolongerserveanyrequests.ItwillnolongersendaheartbeattotheAgency.Therefore,asupervisionprocessrunningintheRaftleaderoftheAgency,cantakethenecessaryaction(after15secondsofmissingheartbeats),namelytopromoteoneoftheserversthatholdin-syncreplicasoftheshardtoleaderforthatshard.ThisinvolvesareconfigurationintheAgencyandleadstothefactthatcoordinatorsnowcontactadifferentDBserverforrequeststothisshard.Serviceresumes.Theothersurvivingreplicasautomaticallyresynchronizetheirdatawiththenewleader.WhentheDBserverwiththeoriginalleadercopycomesback,itnoticesthatitnowholdsafollowerreplica,resynchronizesitsdatawiththenewleaderandorderisrestored.
Allsharddatasynchronizationsaredoneinanincrementalway,suchthatresynchronizationsarequick.Thistechnologyallowstomoveshards(followerandleaderones)betweenDBserverswithoutserviceinterruptions.Therefore,anArangoDBclustercanmoveallthedataonaspecificDBservertootherDBserversandthenshutdownthatserverinacontrolledway.ThisallowstoscaledownanArangoDBclusterwithoutserviceinterruption,lossoffaulttoleranceordataloss.Furthermore,onecanre-balancethedistributionoftheshards,eithermanuallyorautomatically.
AlltheseoperationscanbetriggeredviaaREST/JSONAPIorviathegraphicalwebUI.Allfail-overoperationsarecompletelyhandledwithintheArangoDBcluster.
Architecture
42
Obviously,synchronousreplicationinvolvesacertainincreasedlatencyforwriteoperations,simplybecausethereisonemorenetworkhopwithintheclusterforeveryrequest.Thereforetheusercansetthereplicationfactorto1,whichmeansthatonlyonecopyofeachshardiskept,therebyswitchingoffsynchronousreplication.Thisisasuitablesettingforlessimportantoreasilyrecoverabledataforwhichlowlatencywriteoperationsmatter.
Asynchronousreplicationwithautomaticfail-over
Asynchronousreplicationworksdifferently,inthatitisorganizedusingprimaryandsecondaryDBservers.Eachsecondaryserverreplicatesallthedataheldonaprimarybypollinginanasynchronousway.Thisprocesshasverylittleimpactontheperformanceoftheprimary.Thedisadvantageisthatthereisadelaybetweentheconfirmationofawriteoperationthatissenttotheclientandtheactualreplicationofthedata.Ifthemasterserverfailsduringthisdelay,thencommittedandconfirmeddatacanbelost.
Nevertheless,wealsoofferautomaticfail-overwiththissetup.Contrarytothesynchronouscase,herethefail-overmanagementisdonefromoutsidetheArangoDBcluster.InafutureversionwemightmovethismanagementintothesupervisionprocessintheAgency,butasofnow,themanagementisdoneviatheMesosframeworkschedulerforArangoDB(seebelow).
ThegranularityofthereplicationisawholeArangoDBinstancewithalldatathatresidesonthatinstance,whichmeansthatyouneedtwiceasmanyinstancesaswithoutasynchronousreplication.Synchronousreplicationismoreflexibleinthatrespect,youcanhavesmallerandlargerinstances,andifonefails,thedatacanberebalancedacrosstheremainingones.
Microservicesandzeroadministation
ThedesignandcapabilitiesofArangoDBaregearedtowardsusageinmodernmicroservicearchitecturesofapplications.WiththeFoxxservicesitisveryeasytodeployadatacentricmicroservicewithinanArangoDBcluster.
Inaddition,onecandeploymultipleinstancesofArangoDBwithinthesameproject.Onepartoftheprojectmightneedascalabledocumentstore,anothermightneedagraphdatabase,andyetanothermightneedthefullpowerofamulti-modeldatabaseactuallymixingthevariousdatamodels.Thereareenormousefficiencybenefitstobereapedbybeingabletouseasingletechnologyforvariousrolesinaproject.
TosimplifylifeofthedevopsinsuchascenariowetryasmuchaspossibletouseazeroadministrationapproachforArangoDB.ArunningArangoDBclusterisresilientagainstfailuresandessentiallyrepairsitselfincaseoftemporaryfailures.Seethenextsectionforfurthercapabilitiesinthisdirection.
ApacheMesosintegration
Forthedistributedsetup,weusetheApacheMesosinfrastructurebydefault.ArangoDBisafullycertifiedpackageforDC/OSandcanthusbedeployedessentiallywithafewmouseclicksorasinglecommand,onceyouhaveanexistingDC/OScluster.ButevenonaplainApacheMesosclusteronecandeployArangoDBviaMarathonwithasingleAPIcallandsomeJSONconfiguration.
Theadvantageofthisapproachisthatwecannotonlyimplementtheinitialdeployment,butalsothelatermanagementofautomaticreplacementoffailedinstancesandthescalingoftheArangoDBcluster(triggeredmanuallyorevenautomatically).SinceallmanipulationsareeitherviathegraphicalwebUIorviaJSON/RESTcalls,onecanevenimplementauto-scalingveryeasily.
ADC/OSclusterisaverynaturalenvironmenttodeploymicroservicearchitectures,sinceitissoconvenienttodeployvariousservices,includingpotentiallymultipleArangoDBclusterinstanceswithinthesameDC/OScluster.Thebuilt-inservicediscoverymakesitextremelysimpletoconnectthevariousmicroservicesandMesosautomaticallytakescareofthedistributionanddeploymentofthevarioustasks.
SeetheDeploymentchapteranditssubsectionsforinstructions.
ItispossibletodeployanArangoDBclusterbysimplylaunchingabunchofDockercontainerswiththerightcommandlineoptionstolinkthemup,orevenonasinglemachinestartingmultipleArangoDBprocesses.Inthatcase,synchronousreplicationwillworkwithinthedeployedArangoDBcluster,andautomaticfail-overinthesensethatthedutiesofafailedserverwillautomaticallybeassignedtoanother,survivingone.However,sincetheArangoDBclustercannotwithinitselflaunchadditionalinstances,replacementoffailednodesisnotautomaticandscalingupanddownhastobemanagedmanually.Thisiswhywedonotrecommendthissetupforproductiondeployment.
Architecture
43
Differentdatamodelsandscalability
InthissectionwediscussscalabilityinthecontextofthedifferentdatamodelssupportedbyArangoDB.
Key/valuepairs
Thekey/valuestoredatamodelistheeasiesttoscale.InArangoDB,thisisimplementedinthesensethatadocumentcollectionalwayshasaprimarykey_keyattributeandintheabsenceoffurthersecondaryindexesthedocumentcollectionbehaveslikeasimplekey/valuestore.
Theonlyoperationsthatarepossibleinthiscontextaresinglekeylookupsandkey/valuepairinsertionsandupdates.If_keyistheonlyshardingattributethentheshardingisdonewithrespecttotheprimarykeyandalltheseoperationsscalelinearly.Iftheshardingisdoneusingdifferentshardkeys,thenalookupofasinglekeyinvolvesaskingallshardsandthusdoesnotscalelinearly.
Documentstore
Forthedocumentstorecaseeveninthepresenceofsecondaryindexesessentiallythesameargumentsapply,sinceanindexforashardedcollectionissimplythesameasalocalindexforeachshard.Therefore,singledocumentoperationsstillscalelinearlywiththesizeofthecluster,unlessaspecialshardingconfigurationmakeslookupsorwriteoperationsmoreexpensive.
ForadeeperanalysisofthistopicseethisblogpostinwhichgoodlinearscalabilityofArangoDBforsingledocumentoperationsisdemonstrated.
Complexqueriesandjoins
TheAQLquerylanguageallowscomplexqueries,usingmultiplecollections,secondaryindexesaswellasjoins.Inparticularwiththelatter,scalingcanbeachallenge,sinceifthedatatobejoinedresidesondifferentmachines,alotofcommunicationhastohappen.TheAQLqueryexecutionengineorganizesadatapipelineacrosstheclustertoputtogethertheresultsinthemostefficientway.Thequeryoptimizerisawareoftheclusterstructureandknowswhatdataiswhereandhowitisindexed.Therefore,itcanarriveataninformeddecisionaboutwhatpartsofthequeryoughttorunwhereinthecluster.
Nevertheless,forcertaincomplicatedjoins,therearelimitsastowhatcanbeachieved.
Graphdatabase
Graphdatabasesareparticularlygoodatqueriesongraphsthatinvolvepathsinthegraphofanaprioriunknownlength.Forexample,findingtheshortestpathbetweentwoverticesinagraph,orfindingallpathsthatmatchacertainpatternstartingatagivenvertexaresuchexamples.
However,iftheverticesandedgesalongtheoccurringpathsaredistributedacrossthecluster,thenalotofcommunicationisnecessarybetweennodes,andperformancesuffers.Toachievegoodperformanceatscale,itisthereforenecessarytogetthedistributionofthegraphdataacrosstheshardsintheclusterright.Mostofthetime,theapplicationdevelopersandusersofArangoDBknowbest,howtheirgraphsarastructured.Therefore,ArangoDBallowsuserstospecify,accordingtowhichattributesthegraphdataissharded.Ausefulfirststepisusuallytomakesurethattheedgesoriginatingatavertexresideonthesameclusternodeasthevertex.
Datamodels
44
Limitations
ArangoDBhasnobuilt-inlimitationstohorizontalscalability.ThecentralresilientAgencywilleasilysustainhundredsofDBserversandcoordinators,andtheusualdatabaseoperationsworkcompletelydecentrallyanddonotrequireassistanceoftheAgency.
Likewise,thesupervisionprocessintheAgencycaneasilydealwithlotsofservers,sinceallitsactivitiesarenotperformancecritical.
Obviously,anArangoDBclusterislimitedbytheavailableresourcesofCPU,memory,diskandnetworkbandwidthandlatency.
Limitations
45
Datamodels&modelingThischapterintroducesArangoDB'scoreconceptsandcovers
itsdatamodel(ordatamodelsrespectively),theterminologyusedthroughoutthedatabasesystemandinthisdocumentation,aswellasaspectstoconsiderwhenmodelingyourdatatostrikeabalancebetweennaturaldatastructuresandgreatperformance
Youwillalsofindusageexamplesonhowtointeractwiththedatabasesystemusingarangosh,e.g.howtocreateanddropdatabases/collections,orhowtosave,update,replaceandremovedocuments.Youcandoallthisusingthewebinterfaceaswellandmaythereforeskipthesesectionsasbeginner.
Datamodels&modeling
46
Concepts
DatabaseInteraction
ArangoDBisadatabasethatservesdocumentstoclients.ThesedocumentsaretransportedusingJSONviaaTCPconnection,usingtheHTTPprotocol.ARESTAPIisprovidedtointeractwiththedatabasesystem.
ThewebinterfacethatcomeswithArangoDB,calledAardvark,providesgraphicaluserinterfacethatiseasytouse.Aninteractiveshell,calledArangosh,isalsoshipped.Inaddition,therearesocalleddriversthatmakeiteasytousethedatabasesysteminvariousenvironmentsandprogramminglanguages.AllthesetoolsusetheHTTPinterfaceoftheserverandremovethenecessitytorollownlow-levelcodeforbasiccommunicationinmostcases.
DatamodelThedocumentsyoucanstoreinArangoDBcloselyfollowtheJSONformat,althoughtheyarestoredinabinaryformatcalledVelocyPack.Adocumentcontainszeroormoreattributes,eachoftheseattributeshavingavalue.Avaluecaneitherbeanatomictype,i.e.number,string,booleanornull,oracompoundtype,i.e.anarrayorembeddeddocument/object.Arraysandsub-objectscancontainallofthesetypes,whichmeansthatarbitrarilynesteddatastructurescanberepresentedinasingledocument.
Documentsaregroupedintocollections.Acollectioncontainszeroormoredocuments.Ifyouarefamiliarwithrelationaldatabasemanagementsystems(RDBMS)thenitissafetocomparecollectionstotablesanddocumentstorows.ThedifferenceisthatinatraditionalRDBMS,youhavetodefinecolumnsbeforeyoucanstorerecordsinatable.Suchdefinitionsarealsoknownasschemas.ArangoDBisschema-less,whichmeansthatthereisnoneedtodefinewhatattributesadocumentcanhave.Everysingledocumentcanhaveacompletelydifferentstructureandstillbestoredtogetherwithotherdocumentsinasinglecollection.Inpractice,therewillbecommondenominatorsamongthedocumentsinacollection,butthedatabasesystemitselfdoesn'tforceyoutolimityourselftoacertaindatastructure.
Therearetwotypesofcollections:documentcollection(alsoreferedtoasvertexcollectionsinthecontextofgraphs)aswellasedgecollections.Edgecollectionsstoredocumentsaswell,buttheyincludetwospecialattributes,_fromand_to,whichareusedtocreaterelationsbetweendocuments.Usually,twodocuments(vertices)storedindocumentcollectionsarelinkedbyadocument(edge)storedinanedgecollection.ThisisArangoDB'sgraphdatamodel.Itfollowsthemathematicalconceptofadirected,labeledgraph,exceptthatedgesdon'tjusthavelabels,butarefull-blowndocuments.
Collectionsexistinsideofdatabases.Therecanbeoneormanydatabases.Differentdatabasesareusuallyusedformultitenantsetups,asthedatainsidethem(collections,documentsetc.)isisolatedfromoneanother.Thedefaultdatabase_systemisspecial,becauseitcannotberemoved.Databaseusersaremanagedinthisdatabase,andtheircredentialsarevalidforalldatabasesofaserverinstance.
DataRetrievalQueriesareusedtofilterdocumentsbasedoncertaincriteria,tocomputenewdata,aswellastomanipulateordeleteexistingdocuments.Queriescanbeassimpleasa"querybyexample"orascomplexas"joins"usingmanycollectionsortraversinggraphstructures.TheyarewrittenintheArangoDBQueryLanguage(AQL).
Cursorsareusedtoiterateovertheresultofqueries,sothatyougeteasilyprocessablebatchesinsteadofonebighunk.
Indexesareusedtospeedupsearches.Therearevarioustypesofindexes,suchashashindexesandgeoindexes.
Concepts
47
HandlingDatabasesThisisanintroductiontomanagingdatabasesinArangoDBfromwithinJavaScript.
WhenyouhaveanestablishedconnectiontoArangoDB,thecurrentdatabasecanbechangedexplicitlyusingthedb._useDatabase()method.Thiswillswitchtothespecifieddatabase(provideditexistsandtheusercanconnecttoit).Fromthispointon,anyfollowingactioninthesameshellorconnectionwillusethespecifieddatabase,unlessotherwisespecified.
Note:Ifthedatabaseischanged,clientdriversneedtostorethecurrentdatabasenameontheirside,too.ThisisbecauseconnectionsinArangoDBdonotcontainanystateinformation.AllstateinformationiscontainedintheHTTPrequest/responsedata.
Toconnecttoaspecificdatabaseafterarangoshhasstartedusethecommanddescribedabove.Itisalsopossibletospecifyadatabasenamewheninvokingarangosh.Forthispurpose,usethecommand-lineparameter--server.database,e.g.
>arangosh--server.databasetest
Pleasenotethatcommands,actions,scriptsorAQLqueriesshouldneveraccessmultipledatabases,eveniftheyexist.TheonlyintendedandsupportedwayinArangoDBistouseonedatabaseatatimeforacommand,anaction,ascriptoraquery.Operationsstartedinonedatabasemustnotswitchthedatabaselaterandcontinueoperatinginanother.
Databases
48
WorkingwithDatabases
DatabaseMethods
ThefollowingmethodsareavailabletomanagedatabasesviaJavaScript.Pleasenotethatseveralofthesemethodscanbeusedfromthe_systemdatabaseonly.
Name
returnthedatabasenamedb._name()
Returnsthenameofthecurrentdatabaseasastring.
Examples
arangosh>require("@arangodb").db._name();
_system
ID
returnthedatabaseiddb._id()
Returnstheidofthecurrentdatabaseasastring.
Examples
arangosh>require("@arangodb").db._id();
1
Path
returnthepathtodatabasefilesdb._path()
Returnsthefilesystempathofthecurrentdatabaseasastring.
Examples
arangosh>require("@arangodb").db._path();
/tmp/arangosh_aFcSh5/tmp-9239-3002331047/data/databases/database-1
isSystem
returnthedatabasetypedb._isSystem()
Returnswhetherthecurrentlyuseddatabaseisthe_systemdatabase.Thesystemdatabasehassomespecialprivilegesandproperties,forexample,databasemanagementoperationssuchascreateordropcanonlybeexecutedfromwithinthisdatabase.Additionally,the_systemdatabaseitselfcannotbedropped.
UseDatabase
changethecurrentdatabasedb._useDatabase(name)
Changesthecurrentdatabasetothedatabasespecifiedbyname.Notethatthedatabasespecifiedbynamemustalreadyexist.
Changingthedatabasemightbedisallowedinsomecontexts,forexampleserver-sideactions(includingFoxx).
WorkingwithDatabases
49
Whenperformingthiscommandfromarangosh,thecurrentcredentials(usernameandpassword)willbere-used.Thesecredentialsmightnotbevalidtoconnecttothedatabasespecifiedbyname.Additionally,thedatabaseonlybeaccessedfromcertainendpointsonly.Inthiscase,switchingthedatabasemightnotwork,andtheconnection/sessionshouldbeclosedandrestartedwithdifferentusernameandpasswordcredentialsand/orendpointdata.
ListDatabases
returnthelistofallexistingdatabasesdb._databases()
Returnsthelistofalldatabases.Thismethodcanonlybeusedfromwithinthe_systemdatabase.
CreateDatabase
createanewdatabasedb._createDatabase(name,options,users)
Createsanewdatabasewiththenamespecifiedbyname.Therearerestrictionsfordatabasenames(seeDatabaseNames).
Notethatevenifthedatabaseiscreatedsuccessfully,therewillbenochangeintothecurrentdatabasetothenewdatabase.Changingthecurrentdatabasemustexplicitlyberequestedbyusingthedb._useDatabasemethod.
Theoptionsattributecurrentlyhasnomeaningandisreservedforfutureuse.
Theoptionalusersattributecanbeusedtocreateinitialusersforthenewdatabase.Ifspecified,itmustbealistofuserobjects.Eachuserobjectcancontainthefollowingattributes:
username:theusernameasastring.Thisattributeismandatory.passwd:theuserpasswordasastring.Ifnotspecified,thenitdefaultstoanemptystring.active:abooleanflagindicatingwhethertheuseraccountshouldbeactiveornot.Thedefaultvalueistrue.extra:anoptionalJSONobjectwithextrauserinformation.ThedatacontainedinextrawillbestoredfortheuserbutnotbeinterpretedfurtherbyArangoDB.
Ifnoinitialusersarespecified,adefaultuserrootwillbecreatedwithanemptystringpassword.ThisensuresthatthenewdatabasewillbeaccessibleviaHTTPafteritiscreated.
Youcancreateusersinadatabaseifnoinitialuserisspecified.Switchintothenewdatabase(usernameandpasswordmustbeidenticaltothecurrentsession)andaddormodifyuserswiththefollowingcommands.
require("@arangodb/users").save(username,password,true);
require("@arangodb/users").update(username,password,true);
require("@arangodb/users").remove(username);
Alternatively,youcanspecifyuserdatadirectly.Forexample:
db._createDatabase("newDB",{},[{username:"newUser",passwd:"123456",active:true}])
Thosemethodscanonlybeusedfromwithinthe_systemdatabase.
DropDatabase
dropanexistingdatabasedb._dropDatabase(name)
Dropsthedatabasespecifiedbyname.Thedatabasespecifiedbynamemustexist.
Note:Droppingdatabasesisonlypossiblefromwithinthe_systemdatabase.The_systemdatabaseitselfcannotbedropped.
Databasesaredroppedasynchronously,andwillbephysicallyremovedifallclientshavedisconnectedandreferenceshavebeengarbage-collected.
Enginestatistics
retrievestatisticsrelatedtothestorageengine-rocksdbdb._engineStats()
Returnssomestatisticsrelatedtostorageengineactivity,includingfiguresaboutdatasize,cacheusage,etc.
WorkingwithDatabases
50
NotesaboutDatabasesPleasekeepinmindthateachdatabasecontainsitsownsystemcollections,whichneedtobesetupwhenadatabaseiscreated.Thiswillmakethecreationofadatabasetakeawhile.
Replicationisconfiguredonaper-databaselevel,meaningthatanyreplicationloggingorapplyingforanewdatabasemustbeconfiguredexplicitlyafteranewdatabasehasbeencreated.
Foxxapplicationsarealsoavailableonlyinthecontextofthedatabasetheyhavebeeninstalledin.AnewdatabasewillonlyprovideaccesstothesystemapplicationsshippedwithArangoDB(thatisthewebinterfaceatthemoment)andnootherFoxxapplicationsuntiltheyareexplicitlyinstalledfortheparticulardatabase.
NotesaboutDatabases
52
JavaScriptInterfacetoCollectionsThisisanintroductiontoArangoDB'sinterfaceforcollectionsandhowtohandlecollectionsfromtheJavaScriptshellarangosh.ForotherlanguagesseethecorrespondinglanguageAPI.
Themostimportantcallisthecalltocreateanewcollection.
AddressofaCollection
AllcollectionsinArangoDBhaveauniqueidentifierandauniquename.ArangoDBinternallyusesthecollection'suniqueidentifiertolookupcollections.Thisidentifier,however,ismanagedbyArangoDBandtheuserhasnocontroloverit.Inordertoallowuserstousetheirownnames,eachcollectionalsohasauniquenamewhichisspecifiedbytheuser.Toaccessacollectionfromtheuserperspective,thecollectionnameshouldbeused,i.e.:
Collection
db._collection(collection-name)
Acollectioniscreatedbya"db._create"call.
Forexample:Assumethatthecollectionidentifieris7254820andthenameisdemo,thenthecollectioncanbeaccessedas:
db._collection("demo")
Ifnocollectionwithsuchanameexists,thennullisreturned.
Thereisashort-cutthatcanbeusedfornon-systemcollections:
Collectionname
db.collection-name
Thiscallwilleitherreturnthecollectionnameddb.collection-nameorcreateanewonewiththatnameandasetofdefaultproperties.
Note:Creatingacollectionontheflyusingdb.collection-nameisnotrecommendanddoesnotworkinarangosh.Tocreateanewcollection,pleaseuse
Create
db._create(collection-name)
Thiscallwillcreateanewcollectioncalledcollection-name.ThismethodisadatabasemethodandisdocumentedindetailatDatabaseMethods
Synchronousreplication
StartinginArangoDB3.0,thedistributedversionofferssynchronousreplication,whichmeansthatthereistheoptiontoreplicatealldataautomaticallywithintheArangoDBcluster.Thisisconfiguredforshardedcollectionsonapercollectionbasisbyspecifyinga"replicationfactor"whenthecollectioniscreated.Areplicationfactorofkmeansthataltogetherkcopiesofeachshardarekeptintheclusteronkdifferentservers,andarekeptinsync.Thatis,everywriteoperationisautomaticallyreplicatedonallcopies.
Thisisorganisedusingaleader/followermodel.Atalltimes,oneoftheserversholdingreplicasforashardis"theleader"andallothersare"followers",thisconfigurationisheldintheAgency(seeScalabilityfordetailsoftheArangoDBclusterarchitecture).Everywriteoperationissenttotheleaderbyoneofthecoordinators,andthenreplicatedtoallfollowersbeforetheoperationisreportedtohavesucceeded.Theleaderkeepsarecordofwhichfollowersarecurrentlyinsync.Incaseofnetworkproblemsorafailureofafollower,aleadercanandwilldropafollowertemporarilyafter3seconds,suchthatservicecanresume.Induecourse,thefollowerwillautomaticallyresynchronizewiththeleadertorestoreresilience.
Collections
53
Ifaleaderfails,theclusterAgencyautomaticallyinitiatesafailoverroutineafteraround15seconds,promotingoneofthefollowerstoleader.Theotherfollowers(andtheformerleader,whenitcomesback),automaticallyresynchronizewiththenewleadertorestoreresilience.Usually,thiswholefailoverprocedurecanbehandledtransparentlyforthecoordinator,suchthattheusercodedoesnotevenseeanerrormessage.
Obviously,thisfaulttolerancecomesatacostofincreasedlatency.Eachwriteoperationneedsanadditionalnetworkroundtripforthesynchronousreplicationofthefollowers,butallreplicationoperationstoallfollowershappenconcurrently.Thisis,whythedefaultreplicationfactoris1,whichmeansnoreplication.
Fordetailsonhowtoswitchonsynchronousreplicationforacollection,seethedatabasemethoddb._create(collection-name)inthesectionaboutDatabaseMethods.
Collections
54
CollectionMethods
Drop
dropsacollectioncollection.drop(options)
Dropsacollectionandallitsindexesanddata.Inordertodropasystemcollection,anoptionsobjectwithattributeisSystemsettotruemustbespecified.
Note:droppingacollectioninacluster,whichisprototypeforsharinginothercollectionsisprohibited.Inordertobeabletodropsuchacollection,alldependentcollectionsmustbedroppedfirst.
Examples
arangosh>col=db.example;
[ArangoCollection15512,"example"(typedocument,statusloaded)]
arangosh>col.drop();
arangosh>col;
[ArangoCollection15512,"example"(typedocument,statusdeleted)]
arangosh>col=db._example;
[ArangoCollection15516,"_example"(typedocument,statusloaded)]
arangosh>col.drop({isSystem:true});
arangosh>col;
[ArangoCollection15516,"_example"(typedocument,statusdeleted)]
Truncate
truncatesacollectioncollection.truncate()
Truncatesacollection,removingalldocumentsbutkeepingallitsindexes.
Examples
Truncatesacollection:
arangosh>col=db.example;
arangosh>col.save({"Hello":"World"});
arangosh>col.count();
arangosh>col.truncate();
arangosh>col.count();
showexecutionresults
Properties
getsorsetsthepropertiesofacollectioncollection.properties()
Returnsanobjectcontainingallcollectionproperties.
waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.
journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
isVolatile:IftruethenthecollectiondatawillbekeptinmemoryonlyandArangoDBwillnotwriteorsyncthedatatodisk.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
CollectionMethods
55
keyOptions(optional)additionaloptionsforkeygeneration.ThisisaJSONarraycontainingthefollowingattributes(note:someoftheattributesareoptional):
type:thetypeofthekeygeneratorusedforthecollection.allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.
indexBuckets:numberofbucketsintowhichindexesusingahashtablearesplit.Thedefaultis16andthisnumberhastobeapowerof2andlessthanorequalto1024.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
Forverylargecollectionsoneshouldincreasethistoavoidlongpauseswhenthehashtablehastobeinitiallybuiltorresized,sincebucketsareresizedindividuallyandcanbeinitiallybuiltinparallel.Forexample,64mightbeasensiblevalueforacollectionwith100000000documents.Currently,onlytheedgeindexrespectsthisvalue,butotherindextypesmightfollowinfutureArangoDBversions.Changes(seebelow)areappliedwhenthecollectionisloadedthenexttime.
Inaclustersetup,theresultwillalsocontainthefollowingattributes:
numberOfShards:thenumberofshardsofthecollection.
shardKeys:containsthenamesofdocumentattributesthatareusedtodeterminethetargetshardfordocuments.
replicationFactor:determineshowmanycopiesofeachshardarekeptondifferentDBServers.
collection.properties(properties)
Changesthecollectionproperties.propertiesmustbeanobjectwithoneormoreofthefollowingattribute(s):
waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.
journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
indexBuckets:Seeabove,changesareonlyappliedwhenthecollectionisloadedthenexttime.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
replicationFactor:ChangethenumberofshardcopieskeptondifferentDBServers,validvaluesareintegernumbersintherangeof1-10(Clusteronly)
Note:itisnotpossibletochangethejournalsizeafterthejournalordatafilehasbeencreated.Changingthisparameterwillonlyeffectnewlycreatedjournals.Alsonotethatyoucannotlowerthejournalsizetolessthensizeofthelargestdocumentalreadystoredinthecollection.
Note:someothercollectionproperties,suchastype,isVolatile,orkeyOptionscannotbechangedoncethecollectioniscreated.
Examples
Readallproperties
arangosh>db.example.properties();
showexecutionresultsChangeaproperty
arangosh>db.example.properties({waitForSync:true});
showexecutionresults
Figures
returnsthefiguresofacollectioncollection.figures()
Returnsanobjectcontainingstatisticsaboutthecollection.Note:Retrievingthefigureswillalwaysloadthecollectionintomemory.
alive.count:Thenumberofcurrentlyactivedocumentsinalldatafilesandjournalsofthecollection.Documentsthatarecontainedin
CollectionMethods
56
thewrite-aheadlogonlyarenotreportedinthisfigure.
alive.size:Thetotalsizeinbytesusedbyallactivedocumentsofthecollection.Documentsthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.dead.count:Thenumberofdeaddocuments.Thisincludesdocumentversionsthathavebeendeletedorreplacedbyanewerversion.Documentsdeletedorreplacedthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.dead.size:Thetotalsizeinbytesusedbyalldeaddocuments.dead.deletion:Thetotalnumberofdeletionmarkers.Deletionmarkersonlycontainedinthewrite-aheadlogarenotreportinginthisfigure.datafiles.count:Thenumberofdatafiles.datafiles.fileSize:Thetotalfilesizeofdatafiles(inbytes).journals.count:Thenumberofjournalfiles.journals.fileSize:Thetotalfilesizeofthejournalfiles(inbytes).compactors.count:Thenumberofcompactorfiles.compactors.fileSize:Thetotalfilesizeofthecompactorfiles(inbytes).shapefiles.count:Thenumberofshapefiles.Thisvalueisdeprecatedandkeptforcompatibilityreasonsonly.Thevaluewillalwaysbe0sinceArangoDB2.0andhigher.shapefiles.fileSize:Thetotalfilesizeoftheshapefiles.Thisvalueisdeprecatedandkeptforcompatibilityreasonsonly.Thevaluewillalwaysbe0inArangoDB2.0andhigher.shapes.count:Thetotalnumberofshapesusedinthecollection.Thisincludesshapesthatarenotinuseanymore.Shapesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.shapes.size:Thetotalsizeofallshapes(inbytes).Thisincludesshapesthatarenotinuseanymore.Shapesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.attributes.count:Thetotalnumberofattributesusedinthecollection.Note:thevalueincludesdataofattributesthatarenotinuseanymore.Attributesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.attributes.size:Thetotalsizeoftheattributedata(inbytes).Note:thevalueincludesdataofattributesthatarenotinuseanymore.Attributesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.indexes.count:Thetotalnumberofindexesdefinedforthecollection,includingthepre-definedindexes(e.g.primaryindex).indexes.size:Thetotalmemoryallocatedforindexesinbytes.lastTick:Thetickofthelastmarkerthatwasstoredinajournalofthecollection.Thismightbe0ifthecollectiondoesnotyethaveajournal.uncollectedLogfileEntries:Thenumberofmarkersinthewrite-aheadlogforthiscollectionthathavenotbeentransferredtojournalsordatafiles.documentReferences:ThenumberofreferencestodocumentsindatafilesthatJavaScriptcodecurrentlyholds.Thisinformationcanbeusedfordebuggingcompactionandunloadissues.waitingFor:Anoptionalstringvaluethatcontainsinformationaboutwhichobjecttypeisattheheadofthecollection'scleanupqueue.Thisinformationcanbeusedfordebuggingcompactionandunloadissues.compactionStatus.time:Thepointintimethecompactionforthecollectionwaslastexecuted.Thisinformationcanbeusedfordebuggingcompactionissues.compactionStatus.message:Theactionthatwasperformedwhenthecompactionwaslastrunforthecollection.Thisinformationcanbeusedfordebuggingcompactionissues.
Note:collectiondatathatarestoredinthewrite-aheadlogonlyarenotreportedintheresults.Whenthewrite-aheadlogiscollected,documentsmightbeaddedtojournalsanddatafilesofthecollection,whichmaymodifythefiguresofthecollection.AlsonotethatwaitingForandcompactionStatusmaybeemptywhencalledonacoordinatorinacluster.
Additionally,thefilesizesofcollectionandindexparameterJSONfilesarenotreported.Thesefilesshouldnormallyhaveasizeofafewbyteseach.PleasealsonotethatthefileSizevaluesarereportedinbytesandreflectthelogicalfilesizes.Somefilesystemsmayuseoptimisations(e.g.sparsefiles)sothattheactualphysicalfilesizeissomewhatdifferent.Directoriesandsub-directoriesmayalsorequirespaceinthefilesystem,butthisspaceisnotreportedinthefileSizeresults.
Thatmeansthatthefiguresreporteddonotreflecttheactualdiskusageofthecollectionwith100%accuracy.TheactualdiskusageofacollectionisnormallyslightlyhigherthanthesumofthereportedfileSizevalues.StillthesumofthefileSizevaluescanstillbeusedasalowerboundapproximationofthediskusage.
Examples
arangosh>db.demo.figures()
CollectionMethods
57
showexecutionresults
Load
loadsacollectioncollection.load()
Loadsacollectionintomemory.
Note:clustercollectionsareloadedatalltimes.
Examples
arangosh>col=db.example;
[ArangoCollection15586,"example"(typedocument,statusloaded)]
arangosh>col.load();
arangosh>col;
[ArangoCollection15586,"example"(typedocument,statusloaded)]
Revision
returnstherevisionidofacollectioncollection.revision()
Returnstherevisionidofthecollection
Therevisionidisupdatedwhenthedocumentdataismodified,eitherbyinserting,deleting,updatingorreplacingdocumentsinit.
Therevisionidofacollectioncanbeusedbyclientstocheckwhetherdatainacollectionhaschangedorifitisstillunmodifiedsinceapreviousfetchoftherevisionid.
Therevisionidreturnedisastringvalue.Clientsshouldtreatthisvalueasanopaquestring,andonlyuseitforequality/non-equalitycomparisons.
Path
returnsthephysicalpathofthecollectioncollection.path()
Thepathoperationreturnsastringwiththephysicalstoragepathforthecollectiondata.
Note:thismethodwillreturnnothingmeaningfulinacluster.Inasingle-serverArangoDB,thismethodwillonlyreturnmeaningfuldatafortheMMFilesengine.
Checksum
calculatesachecksumforthedatainacollectioncollection.checksum(withRevisions,withData)
Thechecksumoperationcalculatesanaggregatehashvalueforalldocumentkeyscontainedincollectioncollection.
IftheoptionalargumentwithRevisionsissettotrue,thentherevisionidsofthedocumentsarealsoincludedinthehashcalculation.
IftheoptionalargumentwithDataissettotrue,thenalluser-defineddocumentattributesarealsochecksummed.Includingthedocumentdatainchecksummingwillmakethecalculationslower,butismoreaccurate.
ThechecksumcalculationalgorithmchangedinArangoDB3.0,sochecksumsfrom3.0andearlierversionsforthesamedatawilldiffer.
Note:thismethodisnotavailableinacluster.
Unload
unloadsacollectioncollection.unload()
Startsunloadingacollectionfrommemory.Notethatunloadingisdeferreduntilallqueryhavefinished.
Note:clustercollectionscannotbeunloaded.
Examples
CollectionMethods
58
arangosh>col=db.example;
[ArangoCollection7427,"example"(typedocument,statusloaded)]
arangosh>col.unload();
arangosh>col;
[ArangoCollection7427,"example"(typedocument,statusunloaded)]
Rename
renamesacollectioncollection.rename(new-name)
Renamesacollectionusingthenew-name.Thenew-namemustnotalreadybeusedforadifferentcollection.new-namemustalsobeavalidcollectionname.Formoreinformationonvalidcollectionnamespleaserefertothenamingconventions.
Ifrenamingfailsforanyreason,anerroristhrown.Ifrenamingthecollectionsucceeds,thenthecollectionisalsorenamedinallgraphdefinitionsinsidethe_graphscollectioninthecurrentdatabase.
Note:thismethodisnotavailableinacluster.
Examples
arangosh>c=db.example;
[ArangoCollection15669,"example"(typedocument,statusloaded)]
arangosh>c.rename("better-example");
arangosh>c;
[ArangoCollection15669,"better-example"(typedocument,statusloaded)]
Rotate
rotatesthecurrentjournalofacollectioncollection.rotate()
Rotatesthecurrentjournalofacollection.Thisoperationmakesthecurrentjournalofthecollectionaread-onlydatafilesoitmaybecomeacandidateforgarbagecollection.Ifthereiscurrentlynojournalavailableforthecollection,theoperationwillfailwithanerror.
Note:thismethodisspecificfortheMMFilesstorageengine,andthereitisnotavailableinacluster.
Note:pleasenotethatyouneedappropriateuserpermissionstoexecutethis.
TodotherenamecollectionsinfirstplaceyouneedtohaveadministrativerightsonthedatabaseTohaveaccesstotheresultingrenamedcollectionyoueitherneedtohaveaccesstoallcollectionsofthatdatabase(*)oramainsystemadministratorhastogiveyouaccesstothenewlynamedone.
CollectionMethods
59
DatabaseMethods
Collection
returnsasinglecollectionornulldb._collection(collection-name)
Returnsthecollectionwiththegivennameornullifnosuchcollectionexists.
db._collection(collection-identifier)
Returnsthecollectionwiththegivenidentifierornullifnosuchcollectionexists.Accessingcollectionsbyidentifierisdiscouragedforendusers.Endusersshouldaccesscollectionsusingthecollectionname.
Examples
Getacollectionbyname:
arangosh>db._collection("demo");
[ArangoCollection92,"demo"(typedocument,statusloaded)]
Getacollectionbyid:
arangosh>db._collection(123456);
[ArangoCollection123456,"demo"(typedocument,statusloaded)]
Unknowncollection:
arangosh>db._collection("unknown");
null
Create
createsanewdocumentoredgecollectiondb._create(collection-name)
Createsanewdocumentcollectionnamedcollection-name.Ifthecollectionnamealreadyexistsorifthenameformatisinvalid,anerroristhrown.Formoreinformationonvalidcollectionnamespleaserefertothenamingconventions.
db._create(collection-name,properties)
propertiesmustbeanobjectwiththefollowingattributes:
waitForSync(optional,defaultfalse):Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.
journalSize(optional,defaultisaconfigurationparameter:Themaximalsizeofajournalordatafile.Notethatthisalsolimitsthemaximalsizeofasingleobject.Mustbeatleast1MB.
isSystem(optional,defaultisfalse):Iftrue,createasystemcollection.Inthiscasecollection-nameshouldstartwithanunderscore.Endusersshouldnormallycreatenon-systemcollectionsonly.APIimplementorsmayberequiredtocreatesystemcollectionsinveryspecialoccasions,butnormallyaregularcollectionwilldo.
isVolatile(optional,defaultisfalse):Iftruethenthecollectiondataiskeptin-memoryonlyandnotmadepersistent.Unloadingthecollectionwillcausethecollectiondatatobediscarded.Stoppingorre-startingtheserverwillalsocausefulllossofdatainthecollection.Thecollectionitselfwillremainhowever(onlythedataisvolatile).SettingthisoptionwillmaketheresultingcollectionbeslightlyfasterthanregularcollectionsbecauseArangoDBdoesnotenforceanysynchronizationtodiskanddoesnotcalculateanyCRCchecksumsfordatafiles(astherearenodatafiles).ThisoptionismeaningfulfortheMMFilesstorageengineonly.
keyOptions(optional):additionaloptionsforkeygeneration.Ifspecified,thenkeyOptionsshouldbeaJSONobjectcontainingthefollowingattributes(note:someofthemareoptional):
type:specifiesthetypeofthekeygenerator.Thecurrentlyavailablegeneratorsaretraditionalandautoincrement.(note:autoincrementiscurrentlyonlysupportedfornon-shardedcollections)
DatabaseMethods
60
allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.
numberOfShards(optional,defaultis1):inacluster,thisvaluedeterminesthenumberofshardstocreateforthecollection.Inasingleserversetup,thisoptionismeaningless.
shardKeys(optional,defaultis["_key"]):inacluster,thisattributedetermineswhichdocumentattributesareusedtodeterminethetargetshardfordocuments.Documentsaresenttoshardsbasedonthevaluestheyhaveintheirshardkeyattributes.Thevaluesofallshardkeyattributesinadocumentarehashed,andthehashvalueisusedtodeterminethetargetshard.Notethatvaluesofshardkeyattributescannotbechangedonceset.Thisoptionismeaninglessinasingleserversetup.
Whenchoosingtheshardkeys,onemustbeawareofthefollowingrulesandlimitations:InashardedcollectionwithmorethanonesharditisnotpossibletosetupauniqueconstraintonanattributethatisnottheoneandonlyshardkeygiveninshardKeys.Thisisbecauseenforcingauniqueconstraintwouldotherwisemakeaglobalindexnecessaryorneedextensivecommunicationforeverysinglewriteoperation.Furthermore,if_keyisnottheoneandonlyshardkey,thenitisnotpossibletosetthe_keyattributewheninsertingadocument,providedthecollectionhasmorethanoneshard.Again,thisisbecausethedatabasehastoenforcetheuniqueconstraintonthe_keyattributeandthiscanonlybedoneefficientlyifthisistheonlyshardkeybydelegatingtotheindividualshards.
replicationFactor(optional,defaultis1):inacluster,thisattributedetermineshowmanycopiesofeachshardarekeptondifferentDBServers.Thevalue1meansthatonlyonecopy(nosynchronousreplication)iskept.Avalueofkmeansthatk-1replicasarekept.AnytwocopiesresideondifferentDBServers.Replicationbetweenthemissynchronous,thatis,everywriteoperationtothe"leader"copywillbereplicatedtoall"follower"replicas,beforethewriteoperationisreportedsuccessful.
Ifaserverfails,thisisdetectedautomaticallyandoneoftheserversholdingcopiestakeover,usuallywithoutanerrorbeingreported.
WhenusingtheEnterpriseversionofArangoDBthereplicationFactormaybesetto"satellite"makingthecollectionlocallyjoinableoneverydatabaseserver.ThisreducesthenumberofnetworkhopsdramaticallywhenusingjoinsinAQLatthecostsofreducedwriteperformanceonthesecollections.
distributeShardsLikedistributetheshardsofthiscollectioncloningthesharddistributionofanother.IfthisvalueissetitwillcopyreplicationFactorandnumberOfShardsfromtheothercollection,theattributesinthiscollectionwillbeignoredandcanbeommited.
db._create(collection-name,properties,type)
Specifiestheoptionaltypeofthecollection,itcaneitherbedocumentoredge.Ondefaultitisdocument.Insteadofgivingatypeyoucanalsousedb._createEdgeCollectionordb._createDocumentCollection.
db._create(collection-name,properties[,type],options)
Asanoptionalthird(ifthetypestringisbeingomitted)orfourthparameteryoucanspecifyanoptionaloptionsmapthatcontrolshowtheclusterwillcreatethecollection.Theseoptionsareonlyrelevantatcreationtimeandwillnotbepersisted:
waitForSyncReplication(default:true)Whenenabledtheserverwillonlyreportsuccessbacktotheclientifallreplicashavecreatedthecollection.Settofalseifyouwantfasterserverresponsesanddon'tcareaboutfullreplication.
enforceReplicationFactor(default:true)Whenenabledwhichmeanstheserverwillcheckifthereareenoughreplicasavailableatcreationtimeandbailoutotherwise.Settofalsetodisablethisextracheck.
Examples
Withdefaults:
arangosh>c=db._create("users");
arangosh>c.properties();
showexecutionresultsWithproperties:
arangosh>c=db._create("users",{waitForSync:true,
DatabaseMethods
61
........>journalSize:1024*1204});
arangosh>c.properties();
showexecutionresultsWithakeygenerator:
arangosh>db._create("users",
........>{keyOptions:{type:"autoincrement",offset:10,increment:5}});
arangosh>db.users.save({name:"user1"});
arangosh>db.users.save({name:"user2"});
arangosh>db.users.save({name:"user3"});
showexecutionresultsWithaspecialkeyoption:
arangosh>db._create("users",{keyOptions:{allowUserKeys:false}});
arangosh>db.users.save({name:"user1"});
arangosh>db.users.save({name:"user2",_key:"myuser"});
arangosh>db.users.save({name:"user3"});
showexecutionresultscreatesanewedgecollectiondb._createEdgeCollection(collection-name)
Createsanewedgecollectionnamedcollection-name.Ifthecollectionnamealreadyexistsanerroristhrown.ThedefaultvalueforwaitForSyncisfalse.
db._createEdgeCollection(collection-name,properties)
propertiesmustbeanobjectwiththefollowingattributes:
waitForSync(optional,defaultfalse):Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.journalSize(optional,defaultis"configurationparameter"):Themaximalsizeofajournalordatafile.Notethatthisalsolimitsthemaximalsizeofasingleobjectandmustbeatleast1MB.
createsanewdocumentcollectiondb._createDocumentCollection(collection-name)
Createsanewdocumentcollectionnamedcollection-name.Ifthedocumentnamealreadyexistsanderroristhrown.
AllCollections
returnsallcollectionsdb._collections()
Returnsallcollectionsofthegivendatabase.
Examples
arangosh>db._collections();
showexecutionresults
CollectionName
selectsacollectionfromthevocbasedb.collection-name
Returnsthecollectionwiththegivencollection-name.Ifnosuchcollectionexists,createacollectionnamedcollection-namewiththedefaultproperties.
Examples
arangosh>db.example;
DatabaseMethods
62
[ArangoCollection15403,"example"(typedocument,statusloaded)]
Drop
dropsacollectiondb._drop(collection)
Dropsacollectionandallitsindexesanddata.
db._drop(collection-identifier)
Dropsacollectionidentifiedbycollection-identifierwithallitsindexesanddata.Noerroristhrownifthereisnosuchcollection.
db._drop(collection-name)
Dropsacollectionnamedcollection-nameandallitsindexes.Noerroristhrownifthereisnosuchcollection.
db._drop(collection-name,options)
Inordertodropasystemcollection,onemustspecifyanoptionsobjectwithattributeisSystemsettotrue.Otherwiseitisnotpossibletodropsystemcollections.
Note:clustercollection,whichareprototypesforcollectionswithdistributeShardsLikeparameter,cannotbedropped.
Examples
Dropsacollection:
arangosh>col=db.example;
[ArangoCollection15449,"example"(typedocument,statusloaded)]
arangosh>db._drop(col);
arangosh>col;
[ArangoCollection15449,"example"(typedocument,statusloaded)]
Dropsacollectionidentifiedbyname:
arangosh>col=db.example;
[ArangoCollection15453,"example"(typedocument,statusloaded)]
arangosh>db._drop("example");
arangosh>col;
[ArangoCollection15453,"example"(typedocument,statusdeleted)]
Dropsasystemcollection
arangosh>col=db._example;
[ArangoCollection15457,"_example"(typedocument,statusloaded)]
arangosh>db._drop("_example",{isSystem:true});
arangosh>col;
[ArangoCollection15457,"_example"(typedocument,statusdeleted)]
Truncate
truncatesacollectiondb._truncate(collection)
Truncatesacollection,removingalldocumentsbutkeepingallitsindexes.
db._truncate(collection-identifier)
Truncatesacollectionidentifiedbycollection-identified.Noerroristhrownifthereisnosuchcollection.
db._truncate(collection-name)
Truncatesacollectionnamedcollection-name.Noerroristhrownifthereisnosuchcollection.
Examples
DatabaseMethods
63
Truncatesacollection:
arangosh>col=db.example;
arangosh>col.save({"Hello":"World"});
arangosh>col.count();
arangosh>db._truncate(col);
arangosh>col.count();
showexecutionresultsTruncatesacollectionidentifiedbyname:
arangosh>col=db.example;
arangosh>col.save({"Hello":"World"});
arangosh>col.count();
arangosh>db._truncate("example");
arangosh>col.count();
showexecutionresults
DatabaseMethods
64
DocumentsThisisanintroductiontoArangoDB'sinterfaceforworkingwithdocumentsfromtheJavaScriptshellarangoshorinJavaScriptcodeintheserver.ForotherlanguagesseethecorrespondinglanguageAPI.
BasicsandTerminology:sectiononthebasicapproachCollectionMethods:detailedAPIdescriptionforcollectionobjectsDatabaseMethods:detailedAPIdescriptionfordatabaseobjects
Documents
65
BasicsandTerminologyDocumentsinArangoDBareJSONobjects.Theseobjectscanbenested(toanydepth)andmaycontainlists.Eachdocumenthasauniqueprimarykeywhichidentifiesitwithinitscollection.Furthermore,eachdocumentisuniquelyidentifiedbyitsdocumenthandleacrossallcollectionsinthesamedatabase.Differentrevisionsofthesamedocument(identifiedbyitshandle)canbedistinguishedbytheirdocumentrevision.Anytransactiononlyeverseesasinglerevisionofadocument.Forexample:
{
"_id":"myusers/3456789",
"_key":"3456789",
"_rev":"14253647",
"firstName":"John",
"lastName":"Doe",
"address":{
"street":"RoadToNowhere1",
"city":"Gotham"
},
"hobbies":[
{name:"swimming",howFavorite:10},
{name:"biking",howFavorite:6},
{name:"programming",howFavorite:4}
]
}
Alldocumentscontainspecialattributes:thedocumenthandleisstoredasastringin_id,thedocument'sprimarykeyin_keyandthedocumentrevisionin_rev.Thevalueofthe_keyattributecanbespecifiedbytheuserwhencreatingadocument._idand_keyvaluesareimmutableoncethedocumenthasbeencreated.The_revvalueismaintainedbyArangoDBautomatically.
DocumentHandle
Adocumenthandleuniquelyidentifiesadocumentinthedatabase.Itisastringandconsistsofthecollection'snameandthedocumentkey(_keyattribute)separatedby/.
DocumentKey
Adocumentkeyuniquelyidentifiesadocumentinthecollectionitisstoredin.Itcanandshouldbeusedbyclientswhenspecificdocumentsarequeried.Thedocumentkeyisstoredinthe_keyattributeofeachdocument.ThekeyvaluesareautomaticallyindexedbyArangoDBinacollection'sprimaryindex.Thuslookingupadocumentbyitskeyisafastoperation.The_keyvalueofadocumentisimmutableoncethedocumenthasbeencreated.Bydefault,ArangoDBwillauto-generateadocumentkeyifno_keyattributeisspecified,andusetheuser-specified_keyotherwise.Thegenerated_keyisguaranteedtobeuniqueinthecollectionitwasgeneratedfor.Thisalsoappliestoshardedcollectionsinacluster.Itcan'tbeguaranteedthatthe_keyisuniquewithinadatabaseoracrossawholenodeorinstancehowever.
Thisbehaviorcanbechangedonaper-collectionlevelbycreatingcollectionswiththekeyOptionsattribute.
UsingkeyOptionsitispossibletodisallowuser-specifiedkeyscompletely,ortoforceaspecificregimeforauto-generatingthe_keyvalues.
DocumentRevision
AsArangoDBsupportsMVCC(MultipleVersionConcurrencyControl),documentscanexistinmorethanonerevision.ThedocumentrevisionistheMVCCtokenusedtospecifyaparticularrevisionofadocument(identifiedbyits_id).Itisastringvaluethatcontained(uptoArangoDB3.0)anintegernumberandisuniquewithinthelistofdocumentrevisionsforasingledocument.InArangoDB>=3.1the_revstringsareinfacttimestamps.TheyusethelocalclockoftheDBserverthatactuallywritesthedocumentandhavemillisecondaccuracy.Actually,a"HybridLogicalClock"isused(forthisconceptseethispaper).
Withinonesharditisguaranteedthattwodifferentdocumentrevisionshaveadifferent_revstring,eveniftheyarewritteninthesamemillisecond,andthatthesestampsareascending.
BasicsandTerminology
66
Notehoweverthatdifferentserversinyourclustermighthaveaclockskew,andthereforebetweendifferentshardsorevenbetweendifferentcollectionsthetimestampsarenotguaranteedtobecomparable.
TheHybridLogicalClockfeaturedoesonethingtoaddressthisissue:WheneveramessageissentfromsomeserverAinyourclustertoanotheroneB,itisensuredthatanytimestamptakenonBafterthemessagehasarrivedisgreaterthananytimestamptakenonAbeforethemessagewassent.Thisensuresthatifthereissome"causality"betweeneventsondifferentservers,timestampsincreasefromcausetoeffect.Adirectconsequenceofthisisthatsometimesaserverhastotaketimestampsthatseemtocomefromthefutureofitsownclock.Itwillhoweverstillproduceeverincreasingtimestamps.Iftheclockskewissmall,thenyourtimestampswillrelativelyaccuratelydescribethetimewhenthedocumentrevisionwasactuallywritten.
ArangoDBuses64bitunsignedintegervaluestomaintaindocumentrevisionsinternally.Atthisstageweintentionallydonotdocumenttheexactformatoftherevisionvalues.Whenreturningdocumentrevisionstoclients,ArangoDBwillputthemintoastringtoensuretherevisionisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreattherevisionreturnedbyArangoDBasanopaquestringwhentheystoreoruseitlocally.ThiswillallowArangoDBtochangetheformatofrevisionslaterifthisshouldberequired(ashashappenedwith3.1withtheHybridLogicalClock).Clientscanuserevisionstoperformsimpleequality/non-equalitycomparisons(e.g.tocheckwhetheradocumenthaschangedornot),buttheyshouldnotuserevisionidstoperformgreater/lessthancomparisonswiththemtocheckifadocumentrevisionisolderthanoneanother,evenifthismightworkforsomecases.
Documentrevisionscanbeusedtoconditionallyquery,update,replaceordeletedocumentsinthedatabase.Inordertofindaparticularrevisionofadocument,youneedthedocumenthandleorkey,andthedocumentrevision.
MultipleDocumentsinasingleCommand
BeginningwithArangoDB3.0thebasicdocumentAPIhasbeenextendedtohandlenotonlysingledocumentsbutmultipledocumentsinasinglecommand.Thisiscrucialforperformance,inparticularintheclustersituation,inwhichasinglerequestcaninvolvemultiplenetworkhopswithinthecluster.Anotheradvantageisthatitreducestheoverheadofindividualnetworkroundtripsbetweentheclientandtheserver.ThegeneralideatoperformmultipledocumentoperationsinasinglecommandistouseJSONarraysofobjectsintheplaceofasingledocument.Asaconsequence,documentkeys,handlesandrevisionsforpreconditionshavetobesuppliedembeddedintheindividualdocumentsgiven.Multipledocumentoperationsarerestrictedtoasingledocumentoredgecollection.SeetheAPIdescriptionsforcollectionobjectsfordetails.NotethattheAPIfordatabaseobjectsdonotoffertheseoperations.
BasicsandTerminology
67
CollectionMethods
All
collection.all()
Fetchesalldocumentsfromacollectionandreturnsacursor.YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.
Examples
UsetoArraytogetalldocumentsatonce:
arangosh>db.five.save({name:"one"});
arangosh>db.five.save({name:"two"});
arangosh>db.five.save({name:"three"});
arangosh>db.five.save({name:"four"});
arangosh>db.five.save({name:"five"});
arangosh>db.five.all().toArray();
showexecutionresultsUselimittorestrictthedocuments:
arangosh>db.five.save({name:"one"});
arangosh>db.five.save({name:"two"});
arangosh>db.five.save({name:"three"});
arangosh>db.five.save({name:"four"});
arangosh>db.five.save({name:"five"});
arangosh>db.five.all().limit(2).toArray();
showexecutionresults
Querybyexamplecollection.byExample(example)
Fetchesalldocumentsfromacollectionthatmatchthespecifiedexampleandreturnsacursor.
YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.
Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.Ifyouuse
{"a":{"c":1}}
asexample,thenyouwillfindalldocuments,suchthattheattributeacontainsadocumentoftheform{c:1}.Forexamplethedocument
{"a":{"c":1},"b":1}
willmatch,butthedocument
{"a":{"c":1,"b":1}}
willnot.
However,ifyouuse
{"a.c":1}
CollectionMethods
68
thenyouwillfindalldocuments,whichcontainasub-documentinathathasanattributecofvalue1.Boththefollowingdocuments
{"a":{"c":1},"b":1}
and
{"a":{"c":1,"b":1}}
willmatch.
collection.byExample(path1,value1,...)
Asalternativeyoucansupplyanarrayofpathsandvalues.
Examples
UsetoArraytogetalldocumentsatonce:
arangosh>db.users.save({name:"Gerhard"});
arangosh>db.users.save({name:"Helmut"});
arangosh>db.users.save({name:"Angela"});
arangosh>db.users.all().toArray();
arangosh>db.users.byExample({"_id":"users/20"}).toArray();
arangosh>db.users.byExample({"name":"Gerhard"}).toArray();
arangosh>db.users.byExample({"name":"Helmut","_id":"users/15"}).toArray();
showexecutionresultsUsenexttoloopoveralldocuments:
arangosh>db.users.save({name:"Gerhard"});
arangosh>db.users.save({name:"Helmut"});
arangosh>db.users.save({name:"Angela"});
arangosh>vara=db.users.byExample({"name":"Angela"});
arangosh>while(a.hasNext())print(a.next());
showexecutionresults
FirstExample
collection.firstExample(example)
Returnssomedocumentofacollectionthatmatchesthespecifiedexample.Ifnosuchdocumentexists,nullwillbereturned.Theexamplehastobespecifiedaspathsandvalues.SeebyExamplefordetails.
collection.firstExample(path1,value1,...)
Asalternativeyoucansupplyanarrayofpathsandvalues.
Examples
arangosh>db.users.firstExample("name","Angela");
showexecutionresults
Range
collection.range(attribute,left,right)
CollectionMethods
69
Returnsalldocumentsfromacollectionsuchthattheattributeisgreaterorequalthanleftandstrictlylessthanright.
YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.
Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.
Note:therangesimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionwithinaspecificrangeistouseanAQLqueryasfollows:
FORdocIN@@collection
FILTERdoc.value>=@left&&doc.value<@right
LIMIT@skip,@limit
RETURNdoc
Examples
UsetoArraytogetalldocumentsatonce:
arangosh>db.old.ensureIndex({type:"skiplist",fields:["age"]});
arangosh>db.old.save({age:15});
arangosh>db.old.save({age:25});
arangosh>db.old.save({age:30});
arangosh>db.old.range("age",10,30).toArray();
showexecutionresults
Closedrange
collection.closedRange(attribute,left,right)
Returnsalldocumentsofacollectionsuchthattheattributeisgreaterorequalthanleftandlessorequalthanright.
YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.
Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.
Note:theclosedRangesimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionwithinaspecificrangeistouseanAQLqueryasfollows:
FORdocIN@@collection
FILTERdoc.value>=@left&&doc.value<=@right
LIMIT@skip,@limit
RETURNdoc
Examples
UsetoArraytogetalldocumentsatonce:
arangosh>db.old.ensureIndex({type:"skiplist",fields:["age"]});
arangosh>db.old.save({age:15});
arangosh>db.old.save({age:25});
arangosh>db.old.save({age:30});
arangosh>db.old.closedRange("age",10,30).toArray();
showexecutionresults
Any
collection.any()
Returnsarandomdocumentfromthecollectionornullifnoneexists.
Note:thismethodisexpensivewhenusingtheRocksDBstorageengine.
CollectionMethods
70
Count
collection.count()
Returnsthenumberoflivingdocumentsinthecollection.
Examples
arangosh>db.users.count();
0
toArray
collection.toArray()
Convertsthecollectionintoanarrayofdocuments.NeverusethiscallinaproductionenvironmentasitwillbasicallycreateacopyofyourcollectioninRAMwhichwilluseresourcesdependingonthenumberandsizeofthedocumentsinyourcollecion.
Documentcollection.document(object)
Thedocumentmethodfindsadocumentgivenanobjectobjectcontainingthe_idor_keyattribute.Themethodreturnsthedocumentifitcanbefound.Ifbothattributesaregiven,the_idtakesprecedence,itisanerror,ifthecollectionpartofthe_iddoesnotmatchthecollection.
Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.Anerrorisalsothrownifnodocumentexistswiththegiven_idor_keyvalue.
Pleasenotethatifthemethodisexecutedonthearangodserver(e.g.frominsideaFoxxapplication),animmutabledocumentobjectwillbereturnedforperformancereasons.Itisnotpossibletochangeattributesofthisimmutableobject.Toupdateorpatchthereturneddocument,itneedstobecloned/copiedintoaregularJavaScriptobjectfirst.Thisisnotnecessaryifthedocumentmethodiscalledfromoutofarangoshorfromanyotherclient.
collection.document(document-handle)
Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.Norevisioncanbespecifiedinthiscase.
collection.document(document-key)
Asbefore.Insteadofobjectadocument-keycanbepassedasfirstargument.
collection.document(array)
Thisvariantallowstoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifdocumentwouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.
Examples
Returnsthedocumentforadocument-handle:
arangosh>db.example.document("example/2873916");
showexecutionresultsReturnsthedocumentforadocument-key:
arangosh>db.example.document("2873916");
showexecutionresultsReturnsthedocumentforanobject:
arangosh>db.example.document({_id:"example/2873916"});
CollectionMethods
71
showexecutionresultsReturnsthedocumentforanarrayoftwokeys:
arangosh>db.example.document(["2873916","2873917"]);
showexecutionresultsAnerrorisraisedifthedocumentisunknown:
arangosh>db.example.document("example/4472917");
[ArangoError1202:documentnotfound]
Anerrorisraisedifthehandleisinvalid:
arangosh>db.example.document("");
[ArangoError1205:illegaldocumenthandle]
Changesin3.0from2.8:
documentcannowquerymultipledocumentswithonecall.
Exists
checkswhetheradocumentexistscollection.exists(object)
Theexistsmethoddetermineswhetheradocumentexistsgivenanobjectobjectcontainingthe_idor_keyattribute.Ifbothattributesaregiven,the_idtakesprecedence,itisanerror,ifthecollectionpartofthe_iddoesnotmatchthecollection.
Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.
Insteadofreturningthefounddocumentoranerror,thismethodwillonlyreturnanobjectwiththeattributes_id,_keyand_rev,orfalseifnodocumentwiththegiven_idor_keyexists.Itcanthusbeusedforeasyexistencechecks.
Thismethodwillthrowanerrorifusedimproperly,e.g.whencalledwithanon-documenthandle,anon-document,orwhenacross-collectionrequestisperformed.
collection.exists(document-handle)
Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.
collection.exists(document-key)
Asbefore.Insteadofobjectadocument-keycanbepassedasfirstargument.
collection.exists(array)
Thisvariantallowstoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifexistswouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,theoperationstopsimmediatelyreturningonlyanerrorobject.
Changesin3.0from2.8:
Inthecaseofarevisionmismatchexistsnowthrowsanerrorinsteadofsimplyreturningfalse.Thisistomakeitpossibletotellthedifferencebetweenarevisionmismatchandanon-existingdocument.
existscannowquerymultipledocumentswithonecall.
LookupByKeyscollection.documents(keys)
CollectionMethods
72
Looksupthedocumentsinthespecifiedcollectionusingthearrayofkeysprovided.Alldocumentsforwhichamatchingkeywasspecifiedinthekeysarrayandthatexistinthecollectionwillbereturned.Keysforwhichnodocumentcanbefoundintheunderlyingcollectionareignored,andnoexceptionwillbethrownforthem.
Thismethodisdeprecatedinfavourofthearrayvariantofdocument.
Examples
arangosh>keys=[];
arangosh>for(vari=0;i<10;++i){
........>db.example.insert({_key:"test"+i,value:i});
........>keys.push("test"+i);
........>}
arangosh>db.example.documents(keys);
showexecutionresults
Insert
collection.insert(data)
Createsanewdocumentinthecollectionfromthegivendata.Thedatamustbeanobject.Theattributes_idand_revareignoredandareautomaticallygenerated.Auniquevaluefortheattribute_keywillbeautomaticallygeneratedifnotspecified.Ifspecified,theremustnotbeadocumentwiththegiven_keyinthecollection.
Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleofthenewlycreateddocument,theattribute_keythedocumentkeyandtheattribute_revcontainsthedocumentrevision.
collection.insert(data,options)
Createsanewdocumentinthecollectionfromthegivendataasabove.Theoptionaloptionsparametermustbeanobjectandcanbeusedtospecifythefollowingoptions:
waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.silent:Ifthisflagissettotrue,themethoddoesnotreturnanyoutput.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.
Note:sinceArangoDB2.2,insertisanaliasforsave.
collection.insert(array)
collection.insert(array,options)
Thesetwovariantsallowtoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifinsertwouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.
Changesin3.0from2.8:
TheoptionssilentandreturnNewarenew.Themethodcannowinsertmultipledocumentswithonecall.
Examples
arangosh>db.example.insert({Hello:"World"});
arangosh>db.example.insert({Hello:"World"},{waitForSync:true});
showexecutionresults
arangosh>db.example.insert([{Hello:"World"},{Hello:"there"}])
CollectionMethods
73
arangosh>db.example.insert([{Hello:"World"},{}],{waitForSync:true});
showexecutionresults
Replace
collection.replace(selector,data)
Replacesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenreplacedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.
Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowreplaced)document.
Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.
collection.replace(selector,data,options)
Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:
waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.
collection.replace(document-handle,data)
collection.replace(document-handle,data,options)
Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.
collection.replace(document-key,data)
collection.replace(document-key,data,options)
Asbefore.Insteadofselectoradocument-keycanbepassedasfirstargument.Norevisionpreconditionistested.
collection.replace(selectorarray,dataarray)
collection.replace(selectorarray,dataarray,options)
Thesetwovariantsallowtoperformtheoperationonawholearrayofselector/datapairs.Thetwoarraysgivenasselectorarrayanddataarraymusthavethesamelength.Thebehaviorisexactlyasifreplacewouldhavebeencalledonallrespectivemembersofthetwoarraysandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.
Examples
Createandupdateadocument:
arangosh>a1=db.example.insert({a:1});
arangosh>a2=db.example.replace(a1,{a:2});
arangosh>a3=db.example.replace(a1,{a:3});
showexecutionresultsUseadocumenthandle:
arangosh>a1=db.example.insert({a:1});
CollectionMethods
74
arangosh>a2=db.example.replace("example/3903044",{a:2});
showexecutionresults
Changesin3.0from2.8:
Theoptionssilent,returnNewandreturnOldarenew.Themethodcannowreplacemultipledocumentswithonecall.
Update
collection.update(selector,data)
Updatesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenpatchedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.
Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowupdated)document.
Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.
collection.update(selector,data,options)
Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:
waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.keepNull:TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.mergeObjects:Controlswhetherobjects(notarrays)willbemergedifpresentinboththeexistingandthepatchdocument.Ifsettofalse,thevalueinthepatchdocumentwilloverwritetheexistingdocument'svalue.Ifsettotrue,objectswillbemerged.Thedefaultistrue.
collection.update(document-handle,data)
collection.update(document-handle,data,options)
Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.
collection.update(document-key,data)
collection.update(document-key,data,options)
Asbefore.Insteadofselectoradocument-keycanbepassedasfirstargument.Norevisionpreconditionistested.
collection.update(selectorarray,dataarray)
collection.update(selectorarray,dataarray,options)
Thesetwovariantsallowtoperformtheoperationonawholearrayofselector/datapairs.Thetwoarraysgivenasselectorarrayanddataarraymusthavethesamelength.Thebehaviorisexactlyasifupdatewouldhavebeencalledonallrespectivemembersofthetwoarraysandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.
Examples
Createandupdateadocument:
CollectionMethods
75
arangosh>a1=db.example.insert({"a":1});
arangosh>a2=db.example.update(a1,{"b":2,"c":3});
arangosh>a3=db.example.update(a1,{"d":4});
arangosh>a4=db.example.update(a2,{"e":5,"f":6});
arangosh>db.example.document(a4);
arangosh>a5=db.example.update(a4,{"a":1,c:9,e:42});
arangosh>db.example.document(a5);
showexecutionresultsUseadocumenthandle:
arangosh>a1=db.example.insert({"a":1});
arangosh>a2=db.example.update("example/18612115",{"x":1,"y":2});
showexecutionresultsUsethekeepNullparametertoremoveattributeswithnullvalues:
arangosh>db.example.insert({"a":1});
arangosh>db.example.update("example/19988371",
........>{"b":null,"c":null,"d":3});
arangosh>db.example.document("example/19988371");
arangosh>db.example.update("example/19988371",{"a":null},false,false);
arangosh>db.example.document("example/19988371");
arangosh>db.example.update("example/19988371",
........>{"b":null,"c":null,"d":null},false,false);
arangosh>db.example.document("example/19988371");
showexecutionresultsPatchingarrayvalues:
arangosh>db.example.insert({"a":{"one":1,"two":2,"three":3},
........>"b":{}});
arangosh>db.example.update("example/20774803",{"a":{"four":4},
........>"b":{"b1":1}});
arangosh>db.example.document("example/20774803");
arangosh>db.example.update("example/20774803",{"a":{"one":null},
........>"b":null},
........>false,false);
arangosh>db.example.document("example/20774803");
showexecutionresults
Changesin3.0from2.8:
Theoptionssilent,returnNewandreturnOldarenew.Themethodcannowupdatemultipledocumentswithonecall.
Remove
collection.remove(selector)
Removesadocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenremoved.
Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleoftheremoveddocument,theattribute_revcontainsthedocumentrevisionoftheremoveddocument.
CollectionMethods
76
Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.
collection.remove(selector,options)
Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:
waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.
collection.remove(document-handle)
collection.remove(document-handle,options)
Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisioncheckisperformed.
collection.remove(document-key)
collection.remove(document-handle,options)
Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisioncheckisperformed.
collection.remove(selectorarray)
collection.remove(selectorarray,options)
Thesetwovariantsallowtoperformtheoperationonawholearrayofselectors.Thebehaviorisexactlyasifremovewouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.
Examples
Removeadocument:
arangosh>a1=db.example.insert({a:1});
arangosh>db.example.document(a1);
arangosh>db.example.remove(a1);
arangosh>db.example.document(a1);
showexecutionresultsRemoveadocumentwithaconflict:
arangosh>a1=db.example.insert({a:1});
arangosh>a2=db.example.replace(a1,{a:2});
arangosh>db.example.remove(a1);
arangosh>db.example.remove(a1,true);
arangosh>db.example.document(a1);
showexecutionresults
Changesin3.0from2.8:
Themethodnowreturnsnotonlytruebutinformationabouttheremoveddocument(s).TheoptionssilentandreturnOldarenew.Themethodcannowremovemultipledocumentswithonecall.
RemoveByKeys
collection.removeByKeys(keys)
CollectionMethods
77
Looksupthedocumentsinthespecifiedcollectionusingthearrayofkeysprovided,andremovesalldocumentsfromthecollectionwhosekeysarecontainedinthekeysarray.Keysforwhichnodocumentcanbefoundintheunderlyingcollectionareignored,andnoexceptionwillbethrownforthem.
Themethodwillreturnanobjectcontainingthenumberofremoveddocumentsintheremovedsub-attribute,andthenumberofnot-removed/ignoreddocumentsintheignoredsub-attribute.
Thismethodisdeprecatedinfavourofthearrayvariantofremove.
Examples
arangosh>keys=[];
arangosh>for(vari=0;i<10;++i){
........>db.example.insert({_key:"test"+i,value:i});
........>keys.push("test"+i);
........>}
arangosh>db.example.removeByKeys(keys);
showexecutionresults
RemoveByExamplecollection.removeByExample(example)
Removesalldocumentsmatchinganexample.
collection.removeByExample(document,waitForSync)
TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentdeletionoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.
collection.removeByExample(document,waitForSync,limit)
Theoptionallimitparametercanbeusedtorestrictthenumberofremovalstothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsareremoved.
Examples
arangosh>db.example.removeByExample({Hello:"world"});
1
ReplaceByExample
collection.replaceByExample(example,newValue)
Replacesalldocumentsmatchinganexamplewithanewdocumentbody.TheentiredocumentbodyofeachdocumentmatchingtheexamplewillbereplacedwithnewValue.Thedocumentmeta-attributes_id,_keyand_revwillnotbereplaced.
collection.replaceByExample(document,newValue,waitForSync)
TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentreplacementoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.
collection.replaceByExample(document,newValue,waitForSync,limit)
Theoptionallimitparametercanbeusedtorestrictthenumberofreplacementstothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsarereplaced.
CollectionMethods
78
Examples
arangosh>db.example.save({Hello:"world"});
arangosh>db.example.replaceByExample({Hello:"world"},{Hello:"mars"},false,5);
showexecutionresults
UpdateByExamplecollection.updateByExample(example,newValue)
Partiallyupdatesalldocumentsmatchinganexamplewithanewdocumentbody.SpecificattributesinthedocumentbodyofeachdocumentmatchingtheexamplewillbeupdatedwiththevaluesfromnewValue.Thedocumentmeta-attributes_id,_keyand_revcannotbeupdated.
Partialupdatecouldalsobeusedtoappendnewfields,iftherewerenooldfieldwithsamename.
collection.updateByExample(document,newValue,keepNull,waitForSync)
TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.
TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentreplacementoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.
collection.updateByExample(document,newValue,keepNull,waitForSync,limit)
Theoptionallimitparametercanbeusedtorestrictthenumberofupdatestothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsareupdated.
collection.updateByExample(document,newValue,options)
Usingthisvariant,theoptionsfortheoperationcanbepassedusinganobjectwiththefollowingsub-attributes:
keepNullwaitForSynclimitmergeObjects
Examples
arangosh>db.example.save({Hello:"world",foo:"bar"});
arangosh>db.example.updateByExample({Hello:"world"},{Hello:"foo",World:"bar"},
false);
arangosh>db.example.byExample({Hello:"foo"}).toArray()
showexecutionresults
Collectiontypecollection.type()
Returnsthetypeofacollection.Possiblevaluesare:
2:documentcollection3:edgecollection
GettheVersionofArangoDBdb._version()
CollectionMethods
79
Returnstheserverversionstring.Notethatthisisnottheversionofthedatabase.
Examples
arangosh>require("@arangodb").db._version();
3.3.10
Edges
Edgesarenormaldocumentsthatalwayscontaina_fromanda_toattribute.Therefore,youcanusethedocumentmethodstooperateonedges.Thefollowingmethods,however,arespecifictoedges.
edge-collection.edges(vertex)
Theedgesoperatorfindsalledgesstartingfrom(outbound)orendingin(inbound)vertex.
edge-collection.edges(vertices)
Theedgesoperatorfindsalledgesstartingfrom(outbound)orendingin(inbound)adocumentfromvertices,whichmustbealistofdocumentsordocumenthandles.
arangosh>db._create("vertex");
arangosh>db._createEdgeCollection("relation");
arangosh>varmyGraph={};
arangosh>myGraph.v1=db.vertex.insert({name:"vertex1"});
arangosh>myGraph.v2=db.vertex.insert({name:"vertex2"});
arangosh>myGraph.e1=db.relation.insert(myGraph.v1,myGraph.v2,
........>{label:"knows"});
arangosh>db._document(myGraph.e1);
arangosh>db.relation.edges(myGraph.e1._id);
showexecutionresultsedge-collection.inEdges(vertex)
Theedgesoperatorfindsalledgesendingin(inbound)vertex.
edge-collection.inEdges(vertices)
Theedgesoperatorfindsalledgesendingin(inbound)adocumentfromvertices,whichmustalistofdocumentsordocumenthandles.
Examples
arangosh>db._create("vertex");
arangosh>db._createEdgeCollection("relation");
arangosh>myGraph.v1=db.vertex.insert({name:"vertex1"});
arangosh>myGraph.v2=db.vertex.insert({name:"vertex2"});
arangosh>myGraph.e1=db.relation.insert(myGraph.v1,myGraph.v2,
........>{label:"knows"});
arangosh>db._document(myGraph.e1);
arangosh>db.relation.inEdges(myGraph.v1._id);
arangosh>db.relation.inEdges(myGraph.v2._id);
showexecutionresultsedge-collection.outEdges(vertex)
Theedgesoperatorfindsalledgesstartingfrom(outbound)vertices.
edge-collection.outEdges(vertices)
Theedgesoperatorfindsalledgesstartingfrom(outbound)adocumentfromvertices,whichmustalistofdocumentsordocumenthandles.
Examples
CollectionMethods
80
arangosh>db._create("vertex");
arangosh>db._createEdgeCollection("relation");
arangosh>myGraph.v1=db.vertex.insert({name:"vertex1"});
arangosh>myGraph.v2=db.vertex.insert({name:"vertex2"});
arangosh>myGraph.e1=db.relation.insert(myGraph.v1,myGraph.v2,
........>{label:"knows"});
arangosh>db._document(myGraph.e1);
arangosh>db.relation.outEdges(myGraph.v1._id);
arangosh>db.relation.outEdges(myGraph.v2._id);
showexecutionresults
Misc
collection.iterate(iterator,options)
Iteratesoversomeelementsofthecollectionandapplythefunctioniteratortotheelements.Thefunctionwillbecalledwiththedocumentasfirstargumentandthecurrentnumber(startingwith0)assecondargument.
optionsmustbeanobjectwiththefollowingattributes:
limit(optional,defaultnone):useatmostlimitdocuments.
probability(optional,defaultall):anumberbetween0and1.Documentsarechosenwiththisprobability.
Examples
arangosh>for(i=-90;i<=90;i+=10){
........>for(j=-180;j<=180;j+=10){
........>db.example.save({name:"Name/"+i+"/"+j,
........>home:[i,j],
........>work:[-i,-j]});
........>}
........>}
........>
arangosh>db.example.ensureIndex({type:"geo",fields:["home"]});
arangosh>items=db.example.getIndexes().map(function(x){returnx.id;});
........>db.example.index(items[1]);
showexecutionresults
CollectionMethods
81
DatabaseMethods
Document
db._document(object)
The_documentmethodfindsadocumentgivenanobjectobjectcontainingthe_idattribute.Themethodreturnsthedocumentifitcanbefound.
Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.Anerrorisalsothrownifnodocumentexistswiththegiven_id.
Pleasenotethatifthemethodisexecutedonthearangodserver(e.g.frominsideaFoxxapplication),animmutabledocumentobjectwillbereturnedforperformancereasons.Itisnotpossibletochangeattributesofthisimmutableobject.Toupdateorpatchthereturneddocument,itneedstobecloned/copiedintoaregularJavaScriptobjectfirst.Thisisnotnecessaryifthe_documentmethodiscalledfromoutofarangoshorfromanyotherclient.
db._document(document-handle)
Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.Norevisioncanbespecifiedinthiscase.
Examples
Returnsthedocument:
arangosh>db._document("example/12345");
showexecutionresults
Existsdb._exists(object)
The_existsmethoddetermineswhetheradocumentexistsgivenanobjectobjectcontainingthe_idattribute.
Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.
Insteadofreturningthefounddocumentoranerror,thismethodwillonlyreturnanobjectwiththeattributes_id,_keyand_rev,orfalseifnodocumentwiththegiven_idor_keyexists.Itcanthusbeusedforeasyexistencechecks.
Thismethodwillthrowanerrorifusedimproperly,e.g.whencalledwithanon-documenthandle,anon-document,orwhenacross-collectionrequestisperformed.
db._exists(document-handle)
Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.
Changesin3.0from2.8:
Inthecaseofarevisionmismatch_existsnowthrowsanerrorinsteadofsimplyreturningfalse.Thisistomakeitpossibletotellthedifferencebetweenarevisionmismatchandanon-existingdocument.
Replace
db._replace(selector,data)
Replacesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenreplacedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.
Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowreplaced)document.
DatabaseMethods
82
Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.
collection.replace(selector,data,options)
Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:
waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.
db._replace(document-handle,data)
db._replace(document-handle,data,options)
Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.
Examples
Createandreplaceadocument:
arangosh>a1=db.example.insert({a:1});
arangosh>a2=db._replace(a1,{a:2});
arangosh>a3=db._replace(a1,{a:3});
showexecutionresults
Changesin3.0from2.8:
Theoptionssilent,returnNewandreturnOldarenew.
Update
db._update(selector,data)
Updatesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenpatchedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.
Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowupdated)document.
Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.
db._update(selector,data,options)
Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:
waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.
DatabaseMethods
83
silent:Ifthisflagissettotrue,nooutputisreturned.keepNull:TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.mergeObjects:Controlswhetherobjects(notarrays)willbemergedifpresentinboththeexistingandthepatchdocument.Ifsettofalse,thevalueinthepatchdocumentwilloverwritetheexistingdocument'svalue.Ifsettotrue,objectswillbemerged.Thedefaultistrue.
db._update(document-handle,data)
db._update(document-handle,data,options)
Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.
Examples
Createandupdateadocument:
arangosh>a1=db.example.insert({a:1});
arangosh>a2=db._update(a1,{b:2});
arangosh>a3=db._update(a1,{c:3});
showexecutionresults
Changesin3.0from2.8:
Theoptionssilent,returnNewandreturnOldarenew.
Removedb._remove(selector)
Removesadocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenremoved.
Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleoftheremoveddocument,theattribute_revcontainsthedocumentrevisionoftheremovedeocument.
Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.
db._remove(selector,options)
Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:
waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.
db._remove(document-handle)
db._remove(document-handle,options)
Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisioncheckisperformed.
Examples
Removeadocument:
arangosh>a1=db.example.insert({a:1});
DatabaseMethods
84
arangosh>db._remove(a1);
arangosh>db._remove(a1);
arangosh>db._remove(a1,{overwrite:true});
showexecutionresultsRemovethedocumentintherevisiona1withaconflict:
arangosh>a1=db.example.insert({a:1});
arangosh>a2=db._replace(a1,{a:2});
arangosh>db._remove(a1);
arangosh>db._remove(a1,{overwrite:true});
arangosh>db._document(a1);
showexecutionresultsRemoveadocumentusingnewsignature:
arangosh>db.example.insert({_key:"11265325374",a:1});
arangosh>db.example.remove("example/11265325374",
........>{overwrite:true,waitForSync:false})
showexecutionresults
Changesin3.0from2.8:
Themethodnowreturnsnotonlytruebutinformationabouttheremoveddocument(s).TheoptionssilentandreturnOldarenew.
DatabaseMethods
85
Graphs,Vertices&EdgesGraphs,vertices&edgesaredefinedintheGraphschapterindetails.
Relatedblogposts:
Graphsindatamodeling-istheemperornaked?IndexFreeAdjacencyorHybridIndexesforGraphDatabases
Graphs,Vertices&Edges
86
NamingConventionsinArangoDBThefollowingnamingconventionsshouldbefollowedbyuserswhencreatingdatabases,collectionsanddocumentsinArangoDB.
NamingConventions
87
DatabaseNamesArangoDBwillalwaysstartupwithadefaultdatabase,named_system.UserscancreateadditionaldatabasesinArangoDB,providedthedatabasenamesconformtothefollowingconstraints:
Databasenamesmustonlyconsistofthelettersatoz(bothloweranduppercaseallowed),thenumbers0to9,andtheunderscore(_)ordash(-)symbolsThisalsomeansthatanynon-ASCIIdatabasenamesarenotallowedDatabasenamesmustalwaysstartwithaletter.Databasenamesstartingwithanunderscoreareconsideredtobesystemdatabases,andusersshouldnotcreateordeletethoseThemaximumallowedlengthofadatabasenameis64bytesDatabasenamesarecase-sensitive
DatabaseNames
88
CollectionNamesUserscanpicknamesfortheircollectionsasdesired,providedthefollowingnamingconstraintsarenotviolated:
Collectionnamesmustonlyconsistofthelettersatoz(bothinloweranduppercase),thenumbers0to9,andtheunderscore(_)ordash(-)symbols.Thisalsomeansthatanynon-ASCIIcollectionnamesarenotallowedUser-definedcollectionnamesmustalwaysstartwithaletter.Systemcollectionnamesmuststartwithanunderscore.AllcollectionnamesstartingwithanunderscoreareconsideredtobesystemcollectionsthatareforArangoDB'sinternaluseonly.SystemcollectionnamesshouldnotbeusedbyendusersfortheirowncollectionsThemaximumallowedlengthofacollectionnameis64bytesCollectionnamesarecase-sensitive
CollectionNames
89
DocumentKeysUserscandefinetheirownkeysfordocumentstheysave.Thedocumentkeywillbesavedalongwithadocumentinthe_keyattribute.Userscanpickkeyvaluesasrequired,providedthatthevaluesconformtothefollowingrestrictions:
Thekeymustbeastringvalue.Numerickeysarenotallowed,butanynumericvaluecanbeputintoastringandcanthenbeusedasdocumentkey.Thekeymustbeatleast1byteandatmost254byteslong.Emptykeysaredisallowedwhenspecified(thoughitmaybevalidtocompletelyomitthe_keyattributefromadocument)Itmustconsistofthelettersa-z(loweroruppercase),thedigits0-9oranyofthefollowingpunctuationcharacters:_-:.@()+,=;$!*'%Anyothercharacters,especiallymulti-byteUTF-8sequences,whitespaceorpunctuationcharacterscannotbeusedinsidekeyvaluesThekeymustbeuniquewithinthecollectionitisused
Keysarecase-sensitive,i.e.myKeyandMyKEYareconsideredtobedifferentkeys.
Specifyingadocumentkeyisoptionalwhencreatingnewdocuments.Ifnodocumentkeyisspecifiedbytheuser,ArangoDBwillcreatethedocumentkeyitselfaseachdocumentisrequiredtohaveakey.
Therearenoguaranteesabouttheformatandpatternofauto-generateddocumentkeysotherthantheaboverestrictions.Clientsshouldthereforetreatauto-generateddocumentkeysasopaquevaluesandnotrelyontheirformat.
Thecurrentformatforgeneratedkeysisastringcontainingnumericdigits.Thenumericvaluesreflectchronologicaltimeinthesensethat_keyvaluesgeneratedlaterwillcontainhighernumbersthan_keyvaluesgeneratedearlier.Buttheexactvaluethatwillbegeneratedbytheserverisnotpredictable.Notethatifyousortonthe_keyattribute,stringcomparisonwillbeused,whichmeans"100"islessthan"99"etc.
DocumentKeys
90
AttributeNamesUserscanpickattributenamesfordocumentattributesasdesired,providedthefollowingattributenamingconstraintsarenotviolated:
AttributenamesstartingwithanunderscoreareconsideredtobesystemattributesforArangoDB'sinternaluse.SuchattributenamesarealreadyusedbyArangoDBforspecialpurposes:
_idisusedtocontainadocument'shandle_keyisusedtocontainadocument'suser-definedkey_revisusedtocontainthedocument'srevisionnumberInedgecollections,the
_from_to
attributesareusedtoreferenceotherdocuments.
Moresystemattributesmaybeaddedinthefuturewithoutfurthernoticesoendusersshouldtrytoavoidusingtheirownattributenamesstartingwithunderscores.
Theoretically,attributenamescanincludepunctuationandspecialcharactersasdesired,providedthenameisavalidUTF-8string.Formaximumportability,specialcharactersshouldbeavoidedthough.Forexample,attributenamesmaycontainthedotsymbol,butthedothasaspecialmeaninginJavaScriptandalsoinAQL,sowhenusingsuchattributenamesinoneoftheselanguages,theattributenameneedstobequotedbytheenduser.Overallitmightbebettertouseattributenameswhichdon'trequireanyquoting/escapinginalllanguagesused.Thisincludeslanguagesusedbytheclient(e.g.Ruby,PHP)iftheattributesaremappedtoobjectmembersthere.
Attributenamesstartingwithanat-mark(@)willneedtobeenclosedinbacktickswhenusedinanAQLquerytotellthemapartfrombindvariables.Thereforewedonotencouragetheuseofattributesstartingwithat-marks,thoughtheywillworkwhenusedproperly.ArangoDBdoesnotenforcealengthlimitforattributenames.However,longattributenamesmayusemorememoryinresultsetsetc.Thereforetheuseoflongattributenamesisdiscouraged.Attributenamesarecase-sensitive.Attributeswithemptynames(anemptystring)aredisallowed.
AttributeNames
91
HandlingIndexesThisisanintroductiontoArangoDB'sinterfaceforindexesingeneral.Therearespecialsectionsfor
IndexBasics:IntroductiontoallindextypesWhichindextousewhen:IndextypeandoptionsadviserIndexUtilization:HowArangoDBusesindexesWorkingwithIndexes:Howtohandleindexesprogrammaticallyusingthedbobject
HashIndexesSkiplistsPersistentIndexesFulltextIndexesGeo-spatialIndexesVertex-centricIndexes
Indexing
92
Indexbasics
Indexesallowfastaccesstodocuments,providedtheindexedattribute(s)areusedinaquery.WhileArangoDBautomaticallyindexessomesystemattributes,usersarefreetocreateextraindexesonnon-systemattributesofdocuments.
User-definedindexescanbecreatedoncollectionlevel.Mostuser-definedindexescanbecreatedbyspecifyingthenamesoftheindexattributes.Someindextypesallowindexingjustoneattribute(e.g.fulltextindex)whereasotherindextypesallowindexingmultipleattributesatthesametime.
Thesystemattributes_id,_key,_fromand_toareautomaticallyindexedbyArangoDB,withouttheuserbeingrequiredtocreateextraindexesforthem._idand_keyarecoveredbyacollection'sprimarykey,and_fromand_toarecoveredbyanedgecollection'sedgeindexautomatically.
Usingthesystemattribute_idinuser-definedindexesisnotpossible,butindexing_key,_rev,_from,and_tois.
ArangoDBprovidesthefollowingindextypes:
PrimaryIndex
Foreachcollectiontherewillalwaysbeaprimaryindexwhichisahashindexforthedocumentkeys(_keyattribute)ofalldocumentsinthecollection.Theprimaryindexallowsquickselectionofdocumentsinthecollectionusingeitherthe_keyor_idattributes.ItwillbeusedfromwithinAQLqueriesautomaticallywhenperformingequalitylookupson_keyor_id.
Therearealsodedicatedfunctionstofindadocumentgivenits_keyor_idthatwillalwaysmakeuseoftheprimaryindex:
db.collection.document("<document-key>");
db._document("<document-id>");
Astheprimaryindexisanunsortedhashindex,itcannotbeusedfornon-equalityrangequeriesorforsorting.
Theprimaryindexofacollectioncannotbedroppedorchanged,andthereisnomechanismtocreateuser-definedprimaryindexes.
EdgeIndex
Everyedgecollectionalsohasanautomaticallycreatededgeindex.Theedgeindexprovidesquickaccesstodocumentsbyeithertheir_fromor_toattributes.Itcanthereforebeusedtoquicklyfindconnectionsbetweenvertexdocumentsandisinvokedwhentheconnectingedgesofavertexarequeried.
EdgeindexesareusedfromwithinAQLwhenperformingequalitylookupson_fromor_tovaluesinanedgecollections.Therearealsodedicatedfunctionstofindedgesgiventheir_fromor_tovaluesthatwillalwaysmakeuseoftheedgeindex:
db.collection.edges("<from-value>");
db.collection.edges("<to-value>");
db.collection.outEdges("<from-value>");
db.collection.outEdges("<to-value>");
db.collection.inEdges("<from-value>");
db.collection.inEdges("<to-value>");
Internally,theedgeindexisimplementedasahashindex,whichstorestheunionofall_fromand_toattributes.Itcanbeusedforequalitylookups,butnotforrangequeriesorforsorting.Edgeindexesareautomaticallycreatedforedgecollections.Itisnotpossibletocreateuser-definededgeindexes.However,itispossibletofreelyusethe_fromand_toattributesinuser-definedindexes.
Anedgeindexcannotbedroppedorchanged.
HashIndex
Ahashindexcanbeusedtoquicklyfinddocumentswithspecificattributevalues.Thehashindexisunsorted,soitsupportsequalitylookupsbutnorangequeriesorsorting.
IndexBasics
93
Ahashindexcanbecreatedononeormultipledocumentattributes.Ahashindexwillonlybeusedbyaqueryifallindexattributesarepresentinthesearchcondition,andifallattributesarecomparedusingtheequality(==)operator.HashindexesareusedfromwithinAQLandseveralqueryfunctions,e.g.byExample,firstExampleetc.
Hashindexescanoptionallybedeclaredunique,thendisallowingsavingthesamevalue(s)intheindexedattribute(s).Hashindexescanoptionallybesparse.
Thedifferenttypesofhashindexeshavethefollowingcharacteristics:
uniquehashindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Tryingtoinsertadocumentwiththesamekeyvalueasanalreadyexistingdocumentwillleadtoauniqueconstraintviolation.
Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Akeyvalueofnullmayonlyoccuronceintheindex,sothistypeofindexcannotbeusedforoptionalattributes.
Theuniqueoptioncanalsobeusedtoensurethatnoduplicateedgesarecreated,byaddingacombinedindexforthefields_fromand_totoanedgecollection.
unique,sparsehashindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Documentsinwhichatleastoneoftheindexattributesisnotsetorhasavalueofnullarenotincludedintheindex.Thistypeofindexcanbeusedtoensurethattherearenoduplicatekeysinthecollectionfordocumentswhichhavetheindexedattributesset.Astheindexwillexcludedocumentsforwhichtheindexedattributesarenullornotset,itcanbeusedforoptionalattributes.
non-uniquehashindex:alldocumentsinthecollectionwillbeindexed.Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Duplicatekeyvaluescanoccuranddonotleadtouniqueconstraintviolations.
non-unique,sparsehashindex:onlythosedocumentswillbeindexedthathavealltheindexedattributessettoavalueotherthannull.Itcanbeusedforoptionalattributes.
Theamortizedcomplexityoflookup,insert,update,andremovaloperationsinuniquehashindexesisO(1).
Non-uniquehashindexeshaveanamortizedcomplexityofO(1)forinsert,update,andremovaloperations.Thatmeansnon-uniquehashindexescanbeusedonattributeswithlowcardinality.
Ifahashindexiscreatedonanattributethatismissinginallormanyofthedocuments,thebehaviorisasfollows:
iftheindexissparse,thedocumentsmissingtheattributewillnotbeindexedandnotuseindexmemory.Thesedocumentswillnotinfluencetheupdateorremovalperformancefortheindex.
iftheindexisnon-sparse,thedocumentsmissingtheattributewillbecontainedintheindexwithakeyvalueofnull.
Hashindexessupportindexingarrayvaluesiftheindexattributenameisextendedwitha[*].
SkiplistIndex
Askiplistisasortedindexstructure.Itcanbeusedtoquicklyfinddocumentswithspecificattributevalues,forrangequeriesandforreturningdocumentsfromtheindexinsortedorder.SkiplistswillbeusedfromwithinAQLandseveralqueryfunctions,e.g.byExample,firstExampleetc.
Skiplistindexeswillbeusedforlookups,rangequeriesandsortingonlyifeitherallindexattributesareprovidedinaquery,orifaleftmostprefixoftheindexattributesisspecified.
Forexample,ifaskiplistindexiscreatedonattributesvalue1andvalue2,thefollowingfilterconditionscanusetheindex(note:the<=and>=operatorsareintentionallyomittedhereforthesakeofbrevity):
FILTERdoc.value1==...
FILTERdoc.value1<...
FILTERdoc.value1>...
FILTERdoc.value1>...&&doc.value1<...
FILTERdoc.value1==...&&doc.value2==...
FILTERdoc.value1==...&&doc.value2>...
FILTERdoc.value1==...&&doc.value2>...&&doc.value2<...
IndexBasics
94
Inordertouseaskiplistindexforsorting,theindexattributesmustbespecifiedintheSORTclauseofthequeryinthesameorderastheyappearintheindexdefinition.Skiplistindexesarealwayscreatedinascendingorder,buttheycanbeusedtoaccesstheindexedelementsinbothascendingordescendingorder.However,foracombinedindex(anindexonmultipleattributes)thisrequiresthatthesortordersinasinglequeryasspecifiedintheSORTclausemustbeeitherallascending(optionallyommittedasascendingisthedefault)oralldescending.
Forexample,iftheskiplistindexiscreatedonattributesvalue1andvalue2(inthisorder),thenthefollowingsortsclausescanusetheindexforsorting:
SORTvalue1ASC,value2ASC(anditsequivalentSORTvalue1,value2)SORTvalue1DESC,value2DESC
SORTvalue1ASC(anditsequivalentSORTvalue1)SORTvalue1DESC
Thefollowingsortclausescannotmakeuseoftheindexorder,andrequireanextrasortstep:
SORTvalue1ASC,value2DESC
SORTvalue1DESC,value2ASC
SORTvalue2(anditsequivalentSORTvalue2ASC)SORTvalue2DESC(becausefirstindexedattributevalue1isnotusedinsortclause)
Note:thelattertwosortclausescannotusetheindexbecausethesortclausedoesnotrefertoaleftmostprefixoftheindexattributes.
Skiplistscanoptionallybedeclaredunique,disallowingsavingthesamevalueintheindexedattribute.Theycanbesparseornon-sparse.
Thedifferenttypesofskiplistindexeshavethefollowingcharacteristics:
uniqueskiplistindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Tryingtoinsertadocumentwiththesamekeyvalueasanalreadyexistingdocumentwillleadtoauniqueconstraintviolation.
Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Akeyvalueofnullmayonlyoccuronceintheindex,sothistypeofindexcannotbeusedforoptionalattributes.
unique,sparseskiplistindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Documentsinwhichatleastoneoftheindexattributesisnotsetorhasavalueofnullarenotincludedintheindex.Thistypeofindexcanbeusedtoensurethattherearenoduplicatekeysinthecollectionfordocumentswhichhavetheindexedattributesset.Astheindexwillexcludedocumentsforwhichtheindexedattributesarenullornotset,itcanbeusedforoptionalattributes.
non-uniqueskiplistindex:alldocumentsinthecollectionwillbeindexed.Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Duplicatekeyvaluescanoccuranddonotleadtouniqueconstraintviolations.
non-unique,sparseskiplistindex:onlythosedocumentswillbeindexedthathavealltheindexedattributessettoavalueotherthannull.Itcanbeusedforoptionalattributes.
Theoperationalamortizedcomplexityforskiplistindexesislogarithmicallycorrelatedwiththenumberofdocumentsintheindex.
Skiplistindexessupportindexingarrayvaluesiftheindexattributenameisextendedwitha[*]`.
PersistentIndex
Thepersistentindexisasortedindexwithpersistence.Theindexentriesarewrittentodiskwhendocumentsarestoredorupdated.Thatmeanstheindexentriesdonotneedtoberebuiltfromthecollectiondatawhentheserverisrestartedortheindexedcollectionisinitiallyloaded.Thususingpersistentindexesmayreducecollectionloadingtimes.
Thepersistentindextypecanbeusedforsecondaryindexesatthemoment.Thatmeansthepersistentindexcurrentlycannotbemadetheonlyindexforacollection,becausetherewillalwaysbethein-memoryprimaryindexforthecollectioninaddition,andpotentiallymoreindexes(suchastheedgesindexforanedgecollection).
TheindeximplementationisusingtheRocksDBengine,anditprovideslogarithmiccomplexityforinsert,update,andremoveoperations.Asthepersistentindexisnotanin-memoryindex,itdoesnotstorepointersintotheprimaryindexasallthein-memoryindexesdo,butinsteaditstoresadocument'sprimarykey.Toretrieveadocumentviaapersistentindexviaanindexvaluelookup,therewillthereforebe
IndexBasics
95
anadditionalO(1)lookupintotheprimaryindextofetchtheactualdocument.
Asthepersistentindexissorted,itcanbeusedforpointlookups,rangequeriesandsortingoperations,butonlyifeitherallindexattributesareprovidedinaquery,orifaleftmostprefixoftheindexattributesisspecified.
GeoIndex
Userscancreateadditionalgeoindexesononeormultipleattributesincollections.Ageoindexisusedtofindplacesonthesurfaceoftheearthfast.
Thegeoindexstorestwo-dimensionalcoordinates.Itcanbecreatedoneithertwoseparatedocumentattributes(latitudeandlongitude)orasinglearrayattributethatcontainsbothlatitudeandlongitude.Latitudeandlongitudemustbenumericvalues.
Thegeoindexprovidesoperationstofinddocumentswithcoordinatesnearesttoagivencomparisoncoordinate,andtofinddocumentswithcoordinatesthatarewithinaspecifiableradiusaroundacomparisoncoordinate.
ThegeoindexisusedviadedicatedfunctionsinAQL,thesimplequeriesfunctionsanditisimplicitlyappliedwheninAQLaSORTorFILTERisusedwiththedistancefunction.Otherwiseitwillnotbeusedforothertypesofqueriesorconditions.
FulltextIndex
Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.Afulltextindexcanbecreatedonasingleattributeonly,andwillindexallwordscontainedindocumentsthathaveatextualvalueinthatattribute.Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.Wordsareindexedintheirlower-casedform.Theindexsupportscompletematchqueries(fullwords)andprefixqueries,plusbasiclogicaloperationssuchasand,orandnotforcombiningpartialresults.
Thefulltextindexissparse,meaningitwillonlyindexdocumentsforwhichtheindexattributeissetandcontainsastringvalue.Additionally,onlywordswithaconfigurableminimumlengthwillbeincludedintheindex.
ThefulltextindexisusedviadedicatedfunctionsinAQLorthesimplequeries,butwillnotbeenabledforothertypesofqueriesorconditions.
Indexingattributesandsub-attributes
Top-levelaswellasnestedattributescanbeindexed.Forattributesatthetoplevel,theattributenamesalonearerequired.Toindexasinglefield,passanarraywithasingleelement(stringoftheattributekey)tothefieldsparameteroftheensureIndex()method.Tocreateacombinedindexovermultiplefields,simplyaddmorememberstothefieldsarray:
//{name:"Smith",age:35}
db.posts.ensureIndex({type:"hash",fields:["name"]})
db.posts.ensureIndex({type:"hash",fields:["name","age"]})
Toindexsub-attributes,specifytheattributepathusingthedotnotation:
//{name:{last:"Smith",first:"John"}}
db.posts.ensureIndex({type:"hash",fields:["name.last"]})
db.posts.ensureIndex({type:"hash",fields:["name.last","name.first"]})
Indexingarrayvalues
Ifanindexattributecontainsanarray,ArangoDBwillstoretheentirearrayastheindexvaluebydefault.Accessingindividualmembersofthearrayviatheindexisnotpossiblethisway.
Tomakeanindexinserttheindividualarraymembersintotheindexinsteadoftheentirearrayvalue,aspecialarrayindexneedstobecreatedfortheattribute.Arrayindexescanbesetuplikeregularhashorskiplistindexesusingthecollection.ensureIndex()function.Tomakeahashorskiplistindexanarrayindex,theindexattributenameneedstobeextendedwith[*]whencreatingtheindexandwhenfilteringinanAQLqueryusingtheINoperator.
Thefollowingexamplecreatesanarrayhashindexonthetagsattributeinacollectionnamedposts:
IndexBasics
96
db.posts.ensureIndex({type:"hash",fields:["tags[*]"]});
db.posts.insert({tags:["foobar","baz","quux"]});
ThisarrayindexcanthenbeusedforlookingupindividualtagsvaluesfromAQLqueriesviatheINoperator:
FORdocINposts
FILTER'foobar'INdoc.tags
RETURNdoc
Itispossibletoaddthearrayexpansionoperator[*],butitisnotmandatory.Youmayuseittoindicatethatanarrayindexisused,itispurelycosmetichowever:
FORdocINposts
FILTER'foobar'INdoc.tags[*]
RETURNdoc
ThefollowingFILTERconditionswillnotusethearrayindex:
FILTERdoc.tagsANY=='foobar'
FILTERdoc.tagsANYIN'foobar'
FILTERdoc.tagsIN'foobar'
FILTERdoc.tags=='foobar'
FILTER'foobar'==doc.tags
Itisalsopossibletocreateanindexonsubattributesofarrayvalues.Thismakessenseiftheindexattributeisanarrayofobjects,e.g.
db.posts.ensureIndex({type:"hash",fields:["tags[*].name"]});
db.posts.insert({tags:[{name:"foobar"},{name:"baz"},{name:"quux"}]});
Thefollowingquerywillthenusethearrayindex(thisdoesrequirethearrayexpansionoperator):
FORdocINposts
FILTER'foobar'INdoc.tags[*].name
RETURNdoc
Ifyoustoreadocumenthavingthearraywhichdoescontainelementsnothavingthesubattributesthisdocumentwillalsobeindexedwiththevaluenull,whichinArangoDBisequaltoattributenotexisting.
ArangoDBsupportscreatingarrayindexeswithasingle[*]operatorperindexattribute.Forexample,creatinganindexasfollowsisnotsupported:
db.posts.ensureIndex({type:"hash",fields:["tags[*].name[*].value"]});
Arrayvalueswillautomaticallybede-duplicatedbeforebeinginsertedintoanarrayindex.Forexample,ifthefollowingdocumentisinsertedintothecollection,theduplicatearrayvaluebarwillbeinsertedonlyonce:
db.posts.insert({tags:["foobar","bar","bar"]});
Thisisdonetoavoidredudantstorageofthesameindexvalueforthesamedocument,whichwouldnotprovideanybenefit.
Ifanarrayindexisdeclaredunique,thede-duplicationofarrayvalueswillhappenbeforeinsertingthevaluesintotheindex,sotheaboveinsertoperationwithtwoidenticalvaluesbarwillnotnecessarilyfail
Itwillalwaysfailiftheindexalreadycontainsaninstanceofthebarvalue.However,ifthevaluebarisnotalreadypresentintheindex,thenthede-duplicationofthearrayvalueswilleffectivelyleadtobarbeinginsertedonlyonce.
Toturnoffthededuplicationofarrayvalues,itispossibletosetthededuplicateattributeonthearrayindextofalse.Thedefaultvaluefordeduplicateistruehowever,sode-duplicationwilltakeplaceifnotexplicitlyturnedoff.
db.posts.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:false});
IndexBasics
97
//willfailnow
db.posts.insert({tags:["foobar","bar","bar"]});
Ifanarrayindexisdeclaredandyoustoredocumentsthatdonothaveanarrayatthespecifiedattributethisdocumentwillnotbeinsertedintheindex.Hencethefollowingobjectswillnotbeindexed:
db.posts.ensureIndex({type:"hash",fields:["tags[*]"]});
db.posts.insert({something:"else"});
db.posts.insert({tags:null});
db.posts.insert({tags:"thisisnoarray"});
db.posts.insert({tags:{content:[1,2,3]}});
Anarrayindexisabletoindexexplicitnullvalues.Whenqueriedfornullvalues,itwillonlyreturnthosedocumentshavingexplicitlynullstoredinthearray,itwillnotreturnanydocumentsthatdonothavethearrayatall.
db.posts.ensureIndex({type:"hash",fields:["tags[*]"]});
db.posts.insert({tags:null})//Willnotbeindexed
db.posts.insert({tags:[]})//Willnotbeindexed
db.posts.insert({tags:[null]});//Willbeindexedfornull
db.posts.insert({tags:[null,1,2]});//Willbeindexedfornull,1and2
Declaringanarrayindexassparsedoesnothaveaneffectonthearraypartoftheindex,thisinparticularmeansthatexplicitnullvaluesarealsoindexedinthesparseversion.Ifanindexiscombinedfromanarrayandanormalattributethesparsitywillapplyfortheattributee.g.:
db.posts.ensureIndex({type:"hash",fields:["tags[*]","name"],sparse:true});
db.posts.insert({tags:null,name:"alice"})//Willnotbeindexed
db.posts.insert({tags:[],name:"alice"})//Willnotbeindexed
db.posts.insert({tags:[1,2,3]})//Willnotbeindexed
db.posts.insert({tags:[1,2,3],name:null})//Willnotbeindexed
db.posts.insert({tags:[1,2,3],name:"alice"})
//Willbeindexedfor[1,"alice"],[2,"alice"],[3,"alice"]
db.posts.insert({tags:[null],name:"bob"})
//Willbeindexedfor[null,"bob"]
PleasenotethatfilteringusingarrayindexesonlyworksfromwithinAQLqueriesandonlyifthequeryfiltersontheindexedattributeusingtheINoperator.Theothercomparisonoperators(==,!=,>,>=,<,<=,ANY,ALL,NONE)currentlycannotusearrayindexes.
Vertexcentricindexes
Asmentionedabove,themostimportantindexesforgraphsaretheedgeindexes,indexingthe_fromand_toattributesofedgecollections.Theyprovideveryquickaccesstoalledgesoriginatinginorarrivingatagivenvertex,whichallowstoquicklyfindallneighboursofavertexinagraph.
Inmanycasesonewouldliketorunmorespecificqueries,forexamplefindingamongsttheedgesoriginatinginagivenvertexonlythosewiththe20latesttimestamps.Exactlythisisachievedwith"vertexcentricindexes".Inasensethesearelocalizedindexesforanedgecollection,whichsitateverysinglevertex.
Technically,theyareimplementedinArangoDBasindexes,whichsortthecompleteedgecollectionfirstby_fromandthenbyotherattributes.Ifweforexamplehaveaskiplistindexontheattributes_fromandtimestampofanedgecollection,wecananswertheabovequestionveryquicklywithasinglerangelookupintheindex.
SinceArangoDB3.0onecancreatesortedindexes(type"skiplist"and"persistent")thatindexthespecialedgeattributes_fromor_toandadditionallyotherattributes.SinceArangoDB3.1,theseareusedingraphtraversals,whenappropriateFILTERstatementsarefoundbytheoptimizer.
Forexample,tocreateavertexcentricindexoftheabovetype,youwouldsimplydo
db.edges.ensureIndex({"type":"skiplist","fields":["_from","timestamp"]});
Then,querieslike
IndexBasics
98
FORv,e,pIN1..1OUTBOUND"V/1"edges
FILTERe.timestampALL>="2016-11-09"
RETURNp
willbeconsiderablyfasterincasetherearemanyedgesoriginatinginvertex"V/1"butonlyfewwitharecenttimestamp.
IndexBasics
99
WhichIndextousewhen
ArangoDBautomaticallyindexesthe_keyattributeineachcollection.Thereisnoneedtoindexthisattributeseparately.Pleasenotethatadocument's_idattributeisderivedfromthe_keyattribute,andisthusimplicitlyindexed,too.
ArangoDBwillalsoautomaticallycreateanindexon_fromand_toinanyedgecollection,meaningincomingandoutgoingconnectionscanbedeterminedefficiently.
Indextypes
Userscandefineadditionalindexesononeormultipledocumentattributes.SeveraldifferentindextypesareprovidedbyArangoDB.Theseindexeshavedifferentusagescenarios:
hashindex:providesquickaccesstoindividualdocumentsif(andonlyif)allindexedattributesareprovidedinthesearchquery.Theindexwillonlybeusedforequalitycomparisons.Itdoesnotsupportrangequeriesandcannotbeusedforsorting.
Thehashindexisagoodcandidateifallormostqueriesontheindexedattribute(s)areequalitycomparisons.TheuniquehashindexprovidesanamortizedcomplexityofO(1)forinsert,update,removeandlookupoperations.Thenon-uniquehashindexprovidesO(1)inserts,updatesandremoves,andwillallowlookingupdocumentsbyindexvaluewithamortizedO(n)complexity,withnbeingthenumberofdocumentswiththatindexvalue.
Anon-uniquehashindexonanoptionaldocumentattributeshouldbedeclaredsparsesothatitwillnotindexdocumentsforwhichtheindexattributeisnotset.
skiplistindex:skiplistskeeptheindexedvaluesinanorder,sotheycanbeusedforequalitylookups,rangequeriesandforsorting.Forhighselectivityattributes,skiplistindexeswillhaveahigheroverheadthanhashindexes.Forlowselectivityattributes,skiplistindexeswillbemoreefficientthannon-uniquehashindexes.
Additionally,skiplistindexesallowmoreusecases(e.g.rangequeries,sorting)thanhashindexes.Furthermore,theycanbeusedforlookupsbasedonaleftmostprefixoftheindexattributes.
persistentindex:apersistentindexbehavesmuchlikethesortedskiplistindex,exceptthatallindexvaluesarepersistedondiskanddonotneedtoberebuiltinmemorywhentheserverisrestartedortheindexedcollectionisreloaded.Theoperationsinapersistentindexhavelogarithmiccomplexity,butoperationshavemayhaveahigherconstantfactorthantheoperationsinaskiplistindex,becausethepersistentindexmayneedtomakeextraroundtripstotheprimaryindextofetchtheactualdocuments.
Apersistentindexcanbeusedforequalitylookups,rangequeriesandforsorting.Forhighselectivityattributes,persistentindexeswillhaveahigheroverheadthanskiplistorhashindexes.
Persistentindexesallowmoreusecases(e.g.rangequeries,sorting)thanhashindexes.Furthermore,theycanbeusedforlookupsbasedonaleftmostprefixoftheindexattributes.Incontrasttothein-memoryskiplistindexes,persistentindexesdonotneedtoberebuiltin-memorysotheydon'tinfluencetheloadingtimeofcollectionsasotherin-memoryindexesdo.
geoindex:thegeoindexprovidedbyArangoDBallowssearchingfordocumentswithinaradiusaroundatwo-dimensionalearthcoordinate(point),ortofinddocumentswithareclosesttoapoint.Documentcoordinatescaneitherbespecifiedintwodifferentdocumentattributesorinasingleattribute,e.g.
{"latitude":50.9406645,"longitude":6.9599115}
or
{"coords":[50.9406645,6.9599115]}
GeoindexeswillbeinvokedviaspecialfunctionsorAQLoptimization.TheoptimizationcanbetriggeredwhenacollectionwithgeoindexisenumeratedandaSORTorFILTERstatementisusedinconjunctionwiththedistancefunction.
fulltextindex:afulltextindexcanbeusedtoindexallwordscontainedinaspecificattributeofalldocumentsinacollection.Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.
Whichindextousewhen
100
Theindexsupportscompletematchqueries(fullwords)andprefixqueries.Fulltextindexeswillonlybeinvokedviaspecialfunctions.
Sparsevs.non-sparseindexes
Hashindexesandskiplistindexescanoptionallybecreatedsparse.Asparseindexdoesnotcontaindocumentsforwhichatleastoneoftheindexattributeisnotsetorcontainsavalueofnull.
Assuchdocumentsareexcludedfromsparseindexes,theymaycontainfewerdocumentsthantheirnon-sparsecounterparts.Thisenablesfasterindexingandcanleadtoreducedmemoryusageincasetheindexedattributedoesoccuronlyinsome,butnotalldocumentsofthecollection.Sparseindexeswillalsoreducethenumberofcollisionsinnon-uniquehashindexesincasenon-existingoroptionalattributesareindexed.
Inordertocreateasparseindex,anobjectwiththeattributesparsecanbeaddedtotheindexcreationcommands:
db.collection.ensureIndex({type:"hash",fields:["attributeName"],sparse:true});
db.collection.ensureIndex({type:"hash",fields:["attributeName1","attributeName2"],sparse:true});
db.collection.ensureIndex({type:"hash",fields:["attributeName"],unique:true,sparse:true});
db.collection.ensureIndex({type:"hash",fields:["attributeName1","attributeName2"],unique:true,sparse:true});
db.collection.ensureIndex({type:"skiplist",fields:["attributeName"],sparse:true});
db.collection.ensureIndex({type:"skiplist",fields:["attributeName1","attributeName2"],sparse:true});
db.collection.ensureIndex({type:"skiplist",fields:["attributeName"],unique:true,sparse:true});
db.collection.ensureIndex({type:"skiplist",fields:["attributeName1","attributeName2"],unique:true,sparse:true});
Whennotexplicitlyset,thesparseattributedefaultstofalsefornewindexes.Otherindexesthanhashandskiplistdonotsupportsparsity.
Assparseindexesmayexcludesomedocumentsfromthecollection,theycannotbeusedforalltypesofqueries.Sparsehashindexescannotbeusedtofinddocumentsforwhichatleastoneoftheindexedattributeshasavalueofnull.Forexample,thefollowingAQLquerycannotuseasparseindex,evenifonewascreatedonattributeattr:
FORdocIncollection
FILTERdoc.attr==null
RETURNdoc
Ifthelookupvalueisnon-constant,asparseindexmayormaynotbeused,dependingontheothertypesofconditionsinthequery.Iftheoptimizercansafelydeterminethatthelookupvaluecannotbenull,asparseindexmaybeused.Whenuncertain,theoptimizerwillnotmakeuseofasparseindexinaqueryinordertoproducecorrectresults.
Forexample,thefollowingqueriescannotuseasparseindexonattrbecausetheoptimizerwillnotknowbeforehandwhetherthevalueswhicharecomparedtodoc.attrwillincludenull:
FORdocIncollection
FILTERdoc.attr==SOME_FUNCTION(...)
RETURNdoc
FORotherINotherCollection
FORdocIncollection
FILTERdoc.attr==other.attr
RETURNdoc
Sparseskiplistindexescanbeusedforsortingiftheoptimizercansafelydetectthattheindexrangedoesnotincludenullforanyoftheindexattributes.
Notethatifyouintendtousejoinsitmaybeclevertousenon-sparsityandmaybeevenuniquenessforthatattribute,elseallitemscontainingthenullvaluewillmatchagainsteachotherandthusproducelargeresults.
Whichindextousewhen
101
IndexUtilizationInmostcasesArangoDBwilluseasingleindexpercollectioninagivenquery.AQLqueriescanusemorethanoneindexpercollectionwhenmultipleFILTERconditionsarecombinedwithalogicalORandthesecanbecoveredbyindexes.AQLquerieswilluseasingleindexpercollectionwhenFILTERconditionsarecombinedwithlogicalAND.
Creatingmultipleindexesondifferentattributesofthesamecollectionmaygivethequeryoptimizermorechoiceswhenpickinganindex.Creatingmultipleindexesondifferentattributescanalsohelpinspeedingupdifferentqueries,withFILTERconditionsondifferentattributes.
Itisoftenbeneficialtocreateanindexonmorethanjustoneattribute.Byaddingmoreattributestoanindex,anindexcanbecomemoreselectiveandthusreducethenumberofdocumentsthatqueriesneedtoprocess.
ArangoDB'sprimaryindexes,edgesindexesandhashindexeswillautomaticallyprovideselectivityestimates.Indexselectivityestimatesareprovidedinthewebinterface,thegetIndexes()returnvalueandintheexplain()outputforagivenquery.
Themoreselectiveanindexis,themoredocumentsitwillfilteronaverage.Theindexselectivityestimatesarethereforeusedbytheoptimizerwhencreatingqueryexecutionplanswhentherearemultipleindexestheoptimizercanchoosefrom.Theoptimizerwillthenselectacombinationofindexeswiththelowestestimatedtotalcost.Ingeneral,theoptimizerwillpicktheindexeswiththehighestestimatedselectivity.
Sparseindexesmayormaynotbepickedbytheoptimizerinaquery.Assparseindexesdonotcontainnullvalues,theywillnotbeusedforqueriesiftheoptimizercannotsafelydeterminewhetheraFILTERconditionincludesnullvaluesfortheindexattributes.Theoptimizerpolicyistoproducecorrectresults,regardlessofwhetherorwhichindexisusedtosatisfyFILTERconditions.Ifitisunsureaboutwhetherusinganindexwillviolatethepolicy,itwillnotmakeuseoftheindex.
Troubleshooting
WhenindoubtaboutwhetherandwhichindexeswillbeusedforexecutingagivenAQLquery,clicktheExplainbuttoninthewebinterfaceintheQueriesvieworusetheexplain()methodforthestatementasfollows(fromtheArangoShell):
varquery="FORdocINcollectionFILTERdoc.value>42RETURNdoc";
varstmt=db._createStatement(query);
stmt.explain();
Theexplain()commandwillreturnadetailedJSONrepresentationofthequery'sexecutionplan.TheJSONexplainoutputisintendedtobeusedbycode.Togetahuman-readableandmuchmorecompactexplanationofthequery,thereisanexplainertool:
varquery="FORdocINcollectionFILTERdoc.value>42RETURNdoc";
require("@arangodb/aql/explainer").explain(query);
Ifanyoftheexplainmethodsshowsthataqueryisnotusingindexes,thefollowingstepsmayhelp:
checkiftheattributenamesinthequeryarecorrectlyspelled.Inaschema-freedatabase,documentsinthesamecollectioncanhavevaryingstructures.Thereisnosuchthingasanon-existingattributeerror.Aquerythatreferstoattributenamesnotpresentinanyofthedocumentswillnotreturnanerror,andobviouslywillnotbenefitfromindexes.
checkthereturnvalueofthegetIndexes()methodforthecollectionsusedinthequeryandvalidatethatindexesareactuallypresentontheattributesusedinthequery'sfilterconditions.
ifindexesarepresentbutnotusedbythequery,theindexesmayhavethewrongtype.Forexample,ahashindexwillonlybeusedforequalitycomparisons(i.e.==)butnotforothercomparisontypessuchas<,<=,>,>=.Additionallyhashindexeswillonlybeusedifalloftheindexattributesareusedinthequery'sFILTERconditions.AskiplistindexwillonlybeusedifatleastitsfirstattributeisusedinaFILTERcondition.Ifadditionallyoftheskiplistindexattributesarespecifiedinthequery(fromleft-to-right),theymayalsobeusedandallowtofiltermoredocuments.
usingindexedattributesasfunctionparametersorinarbitraryexpressionswilllikelyleadtotheindexontheattributenotbeingused.Forexample,thefollowingquerieswillnotuseanindexonvalue:
IndexUtilization
102
FORdocINcollectionFILTERTO_NUMBER(doc.value)==42RETURNdoc
FORdocINcollectionFILTERdoc.value-1==42RETURNdoc
Inthesecasesthequeriesshouldberewrittensothatonlytheindexattributeispresentononesideoftheoperator,oradditionalfiltersandindexesshouldbeusedtorestricttheamountofdocumentsotherwise.
certainAQLfunctionssuchasWITHIN()orFULLTEXT()doutilizeindexesinternally,buttheiruseisnotmentionedinthequeryexplanationforfunctionsingeneral.Thesefunctionswillraisequeryerrors(atruntime)ifnosuitableindexispresentforthecollectioninquestion.
thequeryoptimizerwillingeneralpickoneindexpercollectioninaquery.ItcanpickmorethanoneindexpercollectioniftheFILTERconditioncontainsmultiplebranchescombinedwithlogicalOR.Forexample,thefollowingqueriescanuseindexes:
FORdocINcollectionFILTERdoc.value1==42||doc.value1==23RETURNdoc
FORdocINcollectionFILTERdoc.value1==42||doc.value2==23RETURNdoc
FORdocINcollectionFILTERdoc.value1<42||doc.value2>23RETURNdoc
ThetwoORsinthefirstquerywillbeconvertedtoanINlist,andifthereisasuitableindexonvalue1,itwillbeused.Thesecondqueryrequirestwoseparateindexesonvalue1andvalue2andwillusethemifpresent.Thethirdquerycanusetheindexesonvalue1andvalue2whentheyaresorted.
IndexUtilization
103
WorkingwithIndexes
IndexIdentifiersandHandles
Anindexhandleuniquelyidentifiesanindexinthedatabase.Itisastringandconsistsofthecollectionnameandanindexidentifierseparatedbya/.Theindexidentifierpartisanumericvaluethatisauto-generatedbyArangoDB.
Aspecificindexofacollectioncanbeaccessedusingitsindexhandleorindexidentifierasfollows:
db.collection.index("<index-handle>");
db.collection.index("<index-identifier>");
db._index("<index-handle>");
Forexample:Assumethattheindexhandle,whichisstoredinthe_idattributeoftheindex,isdemo/362549736andtheindexwascreatedinacollectionnameddemo.Thenthisindexcanbeaccessedas:
db.demo.index("demo/362549736");
Becausetheindexhandleisuniquewithinthedatabase,youcanleaveoutthecollectionandusetheshortcut:
db._index("demo/362549736");
CollectionMethods
Listingallindexesofacollection
returnsinformationabouttheindexesgetIndexes()
Returnsanarrayofallindexesdefinedforthecollection.
Notethat_keyimplicitlyhasanindexassignedtoit.
arangosh>db.test.ensureHashIndex("hashListAttribute",
........>"hashListSecondAttribute.subAttribute");
arangosh>db.test.getIndexes();
showexecutionresults
Creatinganindex
IndexesshouldbecreatedusingthegeneralmethodensureIndex.Thismethodobsoletesthespecializedindex-specificmethodsensureHashIndex,ensureSkiplist,ensureUniqueConstraintetc.
ensuresthatanindexexistscollection.ensureIndex(index-description)
Ensuresthatanindexaccordingtotheindex-descriptionexists.Anewindexwillbecreatedifnoneexistswiththegivendescription.
Theindex-descriptionmustcontainatleastatypeattribute.Otherattributesmaybenecessary,dependingontheindextype.
typecanbeoneofthefollowingvalues:
hash:hashindexskiplist:skiplistindexfulltext:fulltextindexgeo1:geoindex,withoneattributegeo2:geoindex,withtwoattributes
WorkingwithIndexes
104
sparsecanbetrueorfalse.
Forhash,andskiplistthesparsitycanbecontrolled,fulltextandgeoaresparsebydefinition.
uniquecanbetrueorfalseandissupportedbyhashorskiplist
Callingthismethodreturnsanindexobject.WhetherornottheindexobjectexistedbeforethecallisindicatedinthereturnattributeisNewlyCreated.
deduplicatecanbetrueorfalseandissupportedbyarrayindexesoftypehashorskiplist.Itcontrolswhetherinsertingduplicateindexvaluesfromthesamedocumentintoauniquearrayindexwillleadtoauniqueconstrainterrorornot.Thedefaultvalueistrue,soonlyasingleinstanceofeachnon-uniqueindexvaluewillbeinsertedintotheindexperdocument.Tryingtoinsertavalueintotheindexthatalreadyexistsintheindexwillalwaysfail,regardlessofthevalueofthisattribute.
Examples
arangosh>db.test.ensureIndex({type:"hash",fields:["a"],sparse:true});
arangosh>db.test.ensureIndex({type:"hash",fields:["a","b"],unique:true});
showexecutionresults
Droppinganindex
dropsanindexcollection.dropIndex(index)
Dropstheindex.Iftheindexdoesnotexist,thenfalseisreturned.Iftheindexexistedandwasdropped,thentrueisreturned.Notethatyoucannotdropsomespecialindexes(e.g.theprimaryindexofacollectionortheedgeindexofanedgecollection).
collection.dropIndex(index-handle)
Sameasabove.Insteadofanindexanindexhandlecanbegiven.
arangosh>db.example.ensureSkiplist("a","b");
arangosh>varindexInfo=db.example.getIndexes();
arangosh>indexInfo;
arangosh>db.example.dropIndex(indexInfo[0])
arangosh>db.example.dropIndex(indexInfo[1].id)
arangosh>indexInfo=db.example.getIndexes();
showexecutionresults
LoadIndexesintoMemory
LoadsallindexesofthiscollectionintoMemory.collection.loadIndexesIntoMemory()
Thisfunctiontriestocacheallindexentriesofthiscollectionintothemainmemory.Thereforeititeratesoverallindexesofthecollectionandstorestheindexedvalues,nottheentiredocumentdata,inmemory.Alllookupsthatcouldbefoundinthecachearemuchfasterthanlookupsnotstoredinthecachesoyougetaniceperformanceboost.Itisalsoguaranteedthatthecacheisconsistentwiththestoreddata.
ForthetimebeingthisfunctionisonlyusefulonRocksDBstorageengine,asinMMFilesengineallindexesareinmemoryanyways.
OnRocksDBthisfunctionhonorsallmemorylimits,iftheindexesyouwanttoloadaresmallerthanyourmemorylimitthisfunctionguaranteesthatmostindexvaluesarecached.Iftheindexislargerthanyourmemorylimitthisfunctionwillfillupvaluesuptothislimitandforthetimebeingthereisnowaytocontrolwhichindexesofthecollectionshouldhavepriorityoverothers.
arangosh>db.example.loadIndexesIntoMemory();
{
"result":true
}
WorkingwithIndexes
105
DatabaseMethods
Fetchinganindexbyhandle
findsanindexdb._index(index-handle)
Returnstheindexwithindex-handleornullifnosuchindexexists.
arangosh>db.example.ensureIndex({type:"skiplist",fields:["a","b"]});
arangosh>varindexInfo=db.example.getIndexes().map(function(x){returnx.id;});
arangosh>indexInfo;
arangosh>db._index(indexInfo[0])
arangosh>db._index(indexInfo[1])
showexecutionresults
Droppinganindex
dropsanindexdb._dropIndex(index)
Dropstheindex.Iftheindexdoesnotexist,thenfalseisreturned.Iftheindexexistedandwasdropped,thentrueisreturned.
db._dropIndex(index-handle)
Dropstheindexwithindex-handle.
arangosh>db.example.ensureIndex({type:"skiplist",fields:["a","b"]});
arangosh>varindexInfo=db.example.getIndexes();
arangosh>indexInfo;
arangosh>db._dropIndex(indexInfo[0])
arangosh>db._dropIndex(indexInfo[1].id)
arangosh>indexInfo=db.example.getIndexes();
showexecutionresults
Revalidatingwhetheranindexisused
findsanindex
Soyou'vecreatedanindex,andsinceitsmaintainanceisn'tforfree,youdefinitelywanttoknowwhetheryourquerycanutilizeit.
Youcanuseexplaintoverifywhetherskiplistsorhashindexesareused(ifyouomitcolors:falseyouwillgetnicecolorsinArangoShell):
arangosh>varexplain=require("@arangodb/aql/explainer").explain;
arangosh>db.example.ensureIndex({type:"skiplist",fields:["a","b"]});
arangosh>explain("FORdocINexampleFILTERdoc.a<23RETURNdoc",{colors:false});
showexecutionresults
WorkingwithIndexes
106
HashIndexes
IntroductiontoHashIndexes
Itispossibletodefineahashindexononeormoreattributes(orpaths)ofadocument.ThishashindexisthenusedinqueriestolocatedocumentsinO(1)operations.Ifthehashindexisunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.
Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.
AccessingHashIndexesfromtheShell
UniqueHashIndexes
Ensuresthatauniqueconstraintexists:collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],unique:true})
Createsauniquehashindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.
Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillwillfailiftheattributeuniquenessisviolated.
Tocreateasparseuniqueindex,setthesparseattributetotrue:
collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],unique:true,sparse:true})
Incasethattheindexwassuccessfullycreated,theindexidentifierisreturned.
Non-existingattributeswilldefaulttonull.Inasparseindexalldocumentswillbeexcludedfromtheindexforwhichallspecifiedindexattributesarenull.Suchdocumentswillnotbetakenintoaccountforuniquenesschecks.
Inanon-sparseindex,alldocumentsregardlessofnull-attributeswillbeindexedandwillbetakenintoaccountforuniquenesschecks.
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
arangosh>db.test.ensureIndex({type:"hash",fields:["a","b.c"],unique:true});
arangosh>db.test.save({a:1,b:{c:1}});
arangosh>db.test.save({a:1,b:{c:1}});
arangosh>db.test.save({a:1,b:{c:null}});
arangosh>db.test.save({a:1});
showexecutionresults
Non-uniqueHashIndexes
Ensuresthatanon-uniquehashindexexists:collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"]})
Createsanon-uniquehashindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.
Tocreateasparseuniqueindex,setthesparseattributetotrue:
collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],sparse:true})
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
arangosh>db.test.ensureIndex({type:"hash",fields:["a"]});
arangosh>db.test.save({a:1});
HashIndexes
107
arangosh>db.test.save({a:1});
arangosh>db.test.save({a:null});
showexecutionresults
HashArrayIndexes
Ensuresthatahasharrayindexexists(non-unique):collection.ensureIndex({type:"hash",fields:["field1[*]",...,"fieldn[*]"]})
Createsanon-uniquehasharrayindexfortheindividualelementsofthearrayattributesfield1[*],...fieldn[*]foundinthedocuments.Atleastoneattributepathhastobegiven.Theindexalwaystreatstheindexedarraysassparse.
Itispossibletocombinearrayindexingwithstandardindexing:collection.ensureIndex({type:"hash",fields:["field1[*]","field2"]})
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
arangosh>db.test.ensureIndex({type:"hash",fields:["a[*]"]});
arangosh>db.test.save({a:[1,2]});
arangosh>db.test.save({a:[1,3]});
arangosh>db.test.save({a:null});
showexecutionresults
Ensureuniquenessofrelationsinedgecollections
Itispossibletocreatesecondaryindexesusingtheedgeattributes_fromand_to,startingwithArangoDB3.0.Acombinedindexoverbothfieldstogetherwiththeuniqueoptionenabledcanbeusedtopreventduplicaterelationsfrombeingcreated.
Forexample,adocumentcollectionvertsmightcontainverticeswiththedocumenthandlesverts/A,verts/Bandverts/C.Relationsbetweenthesedocumentscanbestoredinanedgecollectionedgesforinstance.Now,youmaywanttomakesurethatthevertexverts/Aisneverlinkedtoverts/Bbyanedgemorethanonce.Thiscanbeachievedbyaddingaunique,non-sparsehashindexforthefields_fromand_to:
db.edges.ensureIndex({type:"hash",fields:["_from","_to"],unique:true});
Creatinganedge{_from:"verts/A",_to:"verts/B"}inedgeswillbeaccepted,butonlyonce.AnotherattempttostoreanedgewiththerelationA→Bwillberejectedbytheserverwithauniqueconstraintviolatederror.Thisincludesupdatestothe_fromand_tofields.
NotethataddingarelationB→Aisstillpossible,soisA→AandB→B,becausetheyarealldifferentrelationsinadirectedgraph.Eachonecanonlyoccuroncehowever.
HashIndexes
108
Skiplists
IntroductiontoSkiplistIndexes
ThisisanintroductiontoArangoDB'sskiplists.
Itispossibletodefineaskiplistindexononeormoreattributes(orpaths)ofdocuments.Thisskiplististhenusedinqueriestolocatedocumentswithinagivenrange.Iftheskiplistisdeclaredunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.
Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheskiplistindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.
AccessingSkiplistIndexesfromtheShell
UniqueSkiplistIndex
Ensuresthatauniqueskiplistindexexists:collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],unique:true})
Createsauniqueskiplistindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.
Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillfailiftheattributeuniquenessisviolated.
Tocreateasparseuniqueindex,setthesparseattributetotrue:
collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],unique:true,sparse:true})
Inasparseindexalldocumentswillbeexcludedfromtheindexthatdonotcontainatleastoneofthespecifiedindexattributesorthathaveavalueofnullinanyofthespecifiedindexattributes.Suchdocumentswillnotbeindexed,andnotbetakenintoaccountforuniquenesschecks.
Inanon-sparseindex,thesedocumentswillbeindexed(fornon-presentindexedattributes,avalueofnullwillbeused)andwillbetakenintoaccountforuniquenesschecks.
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
arangosh>db.ids.ensureIndex({type:"skiplist",fields:["myId"],unique:true});
arangosh>db.ids.save({"myId":123});
arangosh>db.ids.save({"myId":456});
arangosh>db.ids.save({"myId":789});
arangosh>db.ids.save({"myId":123});
showexecutionresults
arangosh>db.ids.ensureIndex({type:"skiplist",fields:["name.first","name.last"],
unique:true});
arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});
arangosh>db.ids.save({"name":{"first":"jens","last":"jensen"}});
arangosh>db.ids.save({"name":{"first":"hans","last":"jensen"}});
arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});
showexecutionresults
Non-uniqueSkiplistIndex
Skiplists
109
Ensuresthatanon-uniqueskiplistindexexists:collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"]})
Createsanon-uniqueskiplistindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.
Tocreateasparsenon-uniqueindex,setthesparseattributetotrue.
collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],sparse:true})
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
arangosh>db.names.ensureIndex({type:"skiplist",fields:["first"]});
arangosh>db.names.save({"first":"Tim"});
arangosh>db.names.save({"first":"Tom"});
arangosh>db.names.save({"first":"John"});
arangosh>db.names.save({"first":"Tim"});
arangosh>db.names.save({"first":"Tom"});
showexecutionresults
SkiplistArrayIndex
Ensuresthataskiplistarrayindexexists(non-unique):collection.ensureIndex({type:"skiplist",fields:["field1[*]",...,"fieldn[*]"]})
Createsanon-uniqueskiplistarrayindexfortheindividualelementsofthearrayattributesfield1[*],...fieldn[*]foundinthedocuments.Atleastoneattributepathhastobegiven.Theindexalwaystreatstheindexedarraysassparse.
Itispossibletocombinearrayindexingwithstandardindexing:collection.ensureIndex({type:"skiplist",fields:["field1[*]","field2"]})
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
arangosh>db.test.ensureIndex({type:"skiplist",fields:["a[*]"]});
arangosh>db.test.save({a:[1,2]});
arangosh>db.test.save({a:[1,3]});
arangosh>db.test.save({a:null});
showexecutionresults
Querybyexampleusingaskiplistindex
Constructsaquery-by-exampleusingaskiplistindex:collection.byExample(example)
Selectsalldocumentsfromthecollectionthatmatchthespecifiedexampleandreturnsacursor.Askiplistindexwillbeusedifpresent.
YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.
Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.Ifyouuse
{"a":{"c":1}}
asexample,thenyouwillfindalldocuments,suchthattheattributeacontainsadocumentoftheform{c:1}.Forexamplethedocument
{"a":{"c":1},"b":1}
willmatch,butthedocument
{"a":{"c":1,"b":1}}
Skiplists
110
willnot.
However,ifyouuse
{"a.c":1},
thenyouwillfindalldocuments,whichcontainasub-documentinathathasanattributecofvalue1.Boththefollowingdocuments
{"a":{"c":1},"b":1}
and
{"a":{"c":1,"b":1}}
willmatch.
Skiplists
111
Persistentindexes
IntroductiontoPersistentIndexes
ThisisanintroductiontoArangoDB'spersistentindexes.
Itispossibletodefineapersistentindexononeormoreattributes(orpaths)ofdocuments.Theindexisthenusedinqueriestolocatedocumentswithinagivenrange.Iftheindexisdeclaredunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.
Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.
AccessingPersistentIndexesfromtheShell
ensuresthatauniquepersistentindexexistscollection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"],unique:true})
Createsauniquepersistentindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.
Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillwillfailiftheattributeuniquenessisviolated.
Tocreateasparseuniqueindex,setthesparseattributetotrue:
collection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"],unique:true,sparse:true})
Inasparseindexalldocumentswillbeexcludedfromtheindexthatdonotcontainatleastoneofthespecifiedindexattributesorthathaveavalueofnullinanyofthespecifiedindexattributes.Suchdocumentswillnotbeindexed,andnotbetakenintoaccountforuniquenesschecks.
Inanon-sparseindex,thesedocumentswillbeindexed(fornon-presentindexedattributes,avalueofnullwillbeused)andwillbetakenintoaccountforuniquenesschecks.
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
arangosh>db.ids.ensureIndex({type:"persistent",fields:["myId"],unique:true});
arangosh>db.ids.save({"myId":123});
arangosh>db.ids.save({"myId":456});
arangosh>db.ids.save({"myId":789});
arangosh>db.ids.save({"myId":123});
showexecutionresults
arangosh>db.ids.ensureIndex({type:"persistent",fields:["name.first","name.last"],
unique:true});
arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});
arangosh>db.ids.save({"name":{"first":"jens","last":"jensen"}});
arangosh>db.ids.save({"name":{"first":"hans","last":"jensen"}});
arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});
showexecutionresultsensuresthatanon-uniquepersistentindexexistscollection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"]})
Createsanon-uniquepersistentindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.
Persistent
112
Tocreateasparseuniqueindex,setthesparseattributetotrue.
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
arangosh>db.names.ensureIndex({type:"persistent",fields:["first"]});
arangosh>db.names.save({"first":"Tim"});
arangosh>db.names.save({"first":"Tom"});
arangosh>db.names.save({"first":"John"});
arangosh>db.names.save({"first":"Tim"});
arangosh>db.names.save({"first":"Tom"});
showexecutionresults
Querybyexampleusingapersistentindex
constructsaquery-by-exampleusingapersistentindexcollection.byExample(example)
Selectsalldocumentsfromthecollectionthatmatchthespecifiedexampleandreturnsacursor.Apersistentindexwillbeusedifpresent.
YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.
Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.Ifyouuse
{"a":{"c":1}}
asexample,thenyouwillfindalldocuments,suchthattheattributeacontainsadocumentoftheform{c:1}.Forexamplethedocument
{"a":{"c":1},"b":1}
willmatch,butthedocument
{"a":{"c":1,"b":1}}
willnot.
However,ifyouuse
{"a.c":1},
thenyouwillfindalldocuments,whichcontainasub-documentinathathasanattributecofvalue1.Boththefollowingdocuments
{"a":{"c":1},"b":1}
and
{"a":{"c":1,"b":1}}
willmatch.
PersistentIndexesandServerLanguageTheorderofindexentriesinpersistentindexesadherestotheconfiguredserverlanguage.If,however,theserverisrestartedwithadifferentlanguagesettingaswhenthepersistentindexwascreated,notalldocumentsmaybereturnedanymoreandthesortorderofthosewhicharereturnedcanbewrong(wheneverthepersistentindexisconsulted).
Tofixpersistentindexesafteralanguagechange,deleteandre-createthem.Skiplistindexesarenotaffected,becausetheyarenotpersistedandautomaticallyrebuiltoneveryserverstart.
Persistent
113
FulltextindexesThisisanintroductiontoArangoDB'sfulltextindexes.
IntroductiontoFulltextIndexes
Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.
Afulltextindexcanbedefinedononeattributeonly,andwillincludeallwordscontainedindocumentsthathaveatextualvalueintheindexattribute.SinceArangoDB2.6theindexwillalsoincludewordsfromtheindexattributeiftheindexattributeisanarrayofstrings,oranobjectwithstringvaluemembers.
Forexample,givenafulltextindexonthetranslationsattributeandthefollowingdocuments,thensearchingforлисаusingthefulltextindexwouldreturnonlythefirstdocument.SearchingfortheindexfortheexactstringFoxwouldreturnthefirsttwodocuments,andsearchingforprefix:Foxwouldreturnallthreedocuments:
{translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}}
{translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"}
{translations:["ArangoDB","document","database","Foxx"]}
Notethatdeepernestedobjectsareignored.Forexample,afulltextindexontranslationswouldindexFuchs,butnotfox,giventhefollowingdocumentstructure:
{translations:{en:{US:"fox"},de:"Fuchs"}
Ifyouneedtosearchacrossmultiplefieldsand/ornestedobjects,youmaywriteallthestringsintoaspecialattribute,whichyouthencreatetheindexon(itmightbenecessarytocleanthestringsfirst,e.g.removelinebreaksandstripcertainwords).
Iftheindexattributeisneitherastring,anobjectoranarray,itscontentswillnotbeindexed.Whenindexingthecontentsofanarrayattribute,anarraymemberwillonlybeincludedintheindexifitisastring.Whenindexingthecontentsofanobjectattribute,anobjectmembervaluewillonlybeincludedintheindexifitisastring.Otherdatatypesareignoredandnotindexed.
Currently,fulltextindexesarenotyetsupportedwiththeRocksDBstorageengine.ThusthefunctionFULLTEXT()willbeunavailablewhenusingthisstorageengine.Tousefulltextindexes,pleaseusetheMMFilesstorageengineforthetimebeing.
AccessingFulltextIndexesfromtheShell
Ensuresthatafulltextindexexists:
collection.ensureIndex({type:"fulltext",fields:["field"],minLength:minLength})
Createsafulltextindexonalldocumentsonattributefield.
Fulltextindexesareimplicitlysparse:alldocumentswhichdonothavethespecifiedfieldattributeorthathaveanon-qualifyingvalueintheirfieldattributewillbeignoredforindexing.
Onlyasingleattributecanbeindexed.Specifyingmultipleattributesisunsupported.
TheminimumlengthofwordsthatareindexedcanbespecifiedviatheminLengthparameter.WordsshorterthanminLengthcharacterswillnotbeindexed.minLengthhasadefaultvalueof2,butthisvaluemightbechangedinfutureversionsofArangoDB.Itisthusrecommendedtoexplicitlyspecifythisvalue.
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetailsisreturned.
arangosh>db.example.ensureIndex({type:"fulltext",fields:["text"],minLength:3});
arangosh>db.example.save({text:"thequickbrown",b:{c:1}});
arangosh>db.example.save({text:"quickbrownfox",b:{c:2}});
arangosh>db.example.save({text:"brownfoxjums",b:{c:3}});
arangosh>db.example.save({text:"foxjumpsover",b:{c:4}});
FulltextIndexes
115
arangosh>db.example.save({text:"jumpsoverthe",b:{c:5}});
arangosh>db.example.save({text:"overthelazy",b:{c:6}});
arangosh>db.example.save({text:"thelazydog",b:{c:7}});
arangosh>db._query("FORdocumentINFULLTEXT(example,'text','the')RETURNdocument");
showexecutionresultsLooksupafulltextindex:
collection.lookupFulltextIndex(attribute,minLength)
Checkswhetherafulltextindexonthegivenattributeattributeexists.
FulltextAQLFunctions
FulltextAQLfunctionsaredetailedinFulltextfunctions.
FulltextIndexes
116
GeoIndexes
IntroductiontoGeoIndexes
ThisisanintroductiontoArangoDB'sgeoindexes.
AQL'sgeographicfeaturesaredescribedinGeofunctions.
ArangoDBusesHilbertcurvestoimplementgeo-spatialindexes.Seethisblogfordetails.
Ageo-spatialindexassumesthatthelatitudeisbetween-90and90degreeandthelongitudeisbetween-180and180degree.Ageoindexwillignorealldocumentswhichdonotfulfilltheserequirements.
AccessingGeoIndexesfromtheShell
ensuresthatageoindexexistscollection.ensureIndex({type:"geo",fields:["location"]})
Createsageo-spatialindexonalldocumentsusinglocationaspathtothecoordinates.Thevalueoftheattributehastobeanarraywithatleasttwonumericvalues.Thearraymustcontainthelatitude(firstvalue)andthelongitude(secondvalue).
Alldocuments,whichdonothavetheattributepathorhaveanon-conformingvalueinitareexcludedfromtheindex.
Ageoindexisimplicitlysparse,andthereisnowaytocontrolitssparsity.
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
Tocreateageoindexonanarrayattributethatcontainslongitudefirst,setthegeoJsonattributetotrue.ThiscorrespondstotheformatdescribedinRFC7946Position
collection.ensureIndex({type:"geo",fields:["location"],geoJson:true})
Tocreateageo-spatialindexonalldocumentsusinglatitudeandlongitudeasseparateattributepaths,twopathsneedtobespecifiedinthefieldsarray:
collection.ensureIndex({type:"geo",fields:["latitude","longitude"]})
Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.
Examples
Createageoindexforanarrayattribute:
arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});
arangosh>for(i=-90;i<=90;i+=10){
........>for(j=-180;j<=180;j+=10){
........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});
........>}
........>}
arangosh>db.geo.count();
arangosh>db.geo.near(0,0).limit(3).toArray();
arangosh>db.geo.near(0,0).count();
showexecutionresultsCreateageoindexforahasharrayattribute:
arangosh>db.geo2.ensureIndex({type:"geo",fields:["location.latitude",
"location.longitude"]});
arangosh>for(i=-90;i<=90;i+=10){
........>for(j=-180;j<=180;j+=10){
........>db.geo2.save({name:"Name/"+i+"/"+j,location:{latitude:i,
longitude:j}});
GeoIndexes
117
........>}
........>}
arangosh>db.geo2.near(0,0).limit(3).toArray();
showexecutionresultsUseGeoIndexwithAQLSORTstatement:
arangosh>db.geoSort.ensureIndex({type:"geo",fields:["latitude","longitude"]});
arangosh>for(i=-90;i<=90;i+=10){
........>for(j=-180;j<=180;j+=10){
........>db.geoSort.save({name:"Name/"+i+"/"+j,latitude:i,longitude
:j});
........>}
........>}
arangosh>varquery="FORdocingeoSortSORTDISTANCE(doc.latitude,doc.longitude,0,0)
LIMIT5RETURNdoc"
arangosh>db._explain(query,{},{colors:false});
arangosh>db._query(query);
showexecutionresultsUseGeoIndexwithAQLFILTERstatement:
arangosh>db.geoFilter.ensureIndex({type:"geo",fields:["latitude","longitude"]});
arangosh>for(i=-90;i<=90;i+=10){
........>for(j=-180;j<=180;j+=10){
........>db.geoFilter.save({name:"Name/"+i+"/"+j,latitude:i,
longitude:j});
........>}
........>}
arangosh>varquery="FORdocingeoFilterFILTERDISTANCE(doc.latitude,doc.longitude,
0,0)<2000RETURNdoc"
arangosh>db._explain(query,{},{colors:false});
arangosh>db._query(query);
showexecutionresultsconstructsageoindexselectioncollection.geo(location-attribute)
Looksupageoindexdefinedonattributelocation_attribute.
Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.
Thisisusefulforcollectionswithmultipledefinedgeoindexes.
collection.geo(location_attribute,true)
Looksupageoindexonacompoundattributelocation_attribute.
Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.
collection.geo(latitude_attribute,longitude_attribute)
Looksupageoindexdefinedonthetwoattributeslatitude_attributeandlongitude-attribute.
Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.
Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.
GeoIndexes
118
Note:thegeosimplequeryhelperfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforrunninggeoqueriesistousetheirAQLequivalents.
Examples
Assumeyouhavealocationstoredaslistintheattributehomeandadestinationstoredintheattributework.Thenyoucanusethegeooperatortoselectwhichgeo-spatialattributes(andthuswhichindex)touseinanearquery.
arangosh>for(i=-90;i<=90;i+=10){
........>for(j=-180;j<=180;j+=10){
........>db.complex.save({name:"Name/"+i+"/"+j,
........>home:[i,j],
........>work:[-i,-j]});
........>}
........>}
........>
arangosh>db.complex.near(0,170).limit(5);
arangosh>db.complex.ensureIndex({type:"geo",fields:["home"]});
arangosh>db.complex.near(0,170).limit(5).toArray();
arangosh>db.complex.geo("work").near(0,170).limit(5);
arangosh>db.complex.ensureIndex({type:"geo",fields:["work"]});
arangosh>db.complex.geo("work").near(0,170).limit(5).toArray();
showexecutionresultsconstructsanearqueryforacollectioncollection.near(latitude,longitude)
Thereturnedlistissortedaccordingtothedistance,withthenearestdocumenttothecoordinate(latitude,longitude)comingfirst.Ifthereareneardocumentsofequaldistance,documentsarechosenrandomlyfromthissetuntilthelimitisreached.Itispossibletochangethelimitusingthelimitoperator.
Inordertousethenearoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.
Note:neardoesnotsupportnegativeskips.//However,youcanstilluselimitfollowedtoskip.
collection.near(latitude,longitude).limit(limit)
Limitstheresulttolimitdocumentsinsteadofthedefault100.
Note:Unlikewithmultipleexplicitlimits,limitwillraisetheimplicitdefaultlimitimposedbywithin.
collection.near(latitude,longitude).distance()
Thiswilladdanattributedistancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.
collection.near(latitude,longitude).distance(name)
Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.
Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.
Note:thenearsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthenearoperatoristousetheAQLNEARfunctioninanAQLqueryasfollows:
FORdocINNEAR(@@collection,@latitude,@longitude,@limit)
RETURNdoc
Examples
Togetthenearesttwolocations:
GeoIndexes
119
arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});
arangosh>for(vari=-90;i<=90;i+=10){
........>for(varj=-180;j<=180;j+=10){
........>db.geo.save({
........>name:"Name/"+i+"/"+j,
........>loc:[i,j]});
........>}}
arangosh>db.geo.near(0,0).limit(2).toArray();
showexecutionresultsIfyouneedthedistanceaswell,thenyoucanusethedistanceoperator:
arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});
arangosh>for(vari=-90;i<=90;i+=10){
........>for(varj=-180;j<=180;j+=10){
........>db.geo.save({
........>name:"Name/"+i+"/"+j,
........>loc:[i,j]});
........>}}
arangosh>db.geo.near(0,0).distance().limit(2).toArray();
showexecutionresultsconstructsawithinqueryforacollectioncollection.within(latitude,longitude,radius)
Thiswillfindalldocumentswithinagivenradiusaroundthecoordinate(latitude,longitude).Thereturnedarrayissortedbydistance,beginningwiththenearestdocument.
Inordertousethewithinoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.
collection.within(latitude,longitude,radius).distance()
Thiswilladdanattribute_distancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.
collection.within(latitude,longitude,radius).distance(name)
Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.
Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.
Note:thewithinsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthewithinoperatoristousetheAQLWITHINfunctioninanAQLqueryasfollows:
FORdocINWITHIN(@@collection,@latitude,@longitude,@radius,@distanceAttributeName)
RETURNdoc
Examples
Tofindalldocumentswithinaradiusof2000kmuse:
arangosh>for(vari=-90;i<=90;i+=10){
........>for(varj=-180;j<=180;j+=10){
........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});}}
arangosh>db.geo.within(0,0,2000*1000).distance().toArray();
showexecutionresultsensuresthatageoindexexistscollection.ensureIndex({type:"geo",fields:["location"]})
GeoIndexes
120
SinceArangoDB2.5,thismethodisanaliasforensureGeoIndexsincegeoindexesarealwayssparse,meaningthatdocumentsthatdonotcontaintheindexattributesorhavenon-numericvaluesintheindexattributeswillnotbeindexed.ensureGeoConstraintisdeprecatedandensureGeoIndexshouldbeusedinstead.
Theindexdoesnotprovideauniqueoptionbecauseofitslimitedusability.Itwouldpreventidenticalcoordinatesfrombeinginsertedonly,butevenaslightlydifferentlocation(like1inchor1cmoff)wouldbeuniqueagainandnotconsideredaduplicate,althoughitprobablyshould.Thedesiredthresholdfordetectingduplicatesmayvaryforeveryproject(includinghowtocalculatethedistanceeven)andneedstobeimplementedontheapplicationlayerasneeded.YoucanwriteaFoxxserviceforthispurposeandmakeuseoftheAQLgeofunctionstofindnearbycoordinatessupportedbyageoindex.
GeoIndexes
121
VertexCentricIndexes
IntroductiontoVertexCentricIndexes
InArangoDBtherearespecialindicesdesignedtospeedupgraphoperations,especiallyifthegraphcontainssupernodes(verticesthathaveanexceptionallyhighamountofconnectededges).Theseindicesarecalledvertexcentricindexesandcanbeusedinadditiontotheexistingedgeindex.
Motivation
Theideaofthisindexistoindexacombinationofavertex,thedirectionandanyarbitrarysetofotherattributesontheedges.Totakeanexample,ifwehaveanattributecalledtypeontheedges,wecanuseanoutboundvertex-centricindexonthisattributetofindalledgesattachedtoavertexwithagiventype.Thefollowingqueryexamplecouldbenefitfromsuchanindex:
FORv,e,pIN3..5OUTBOUND@startGRAPH@graphName
FILTERp.edges[*].typeALL=="friend"
RETURNv
Usingthebuilt-inedge-indexArangoDBcanfindthelistofalledgesattachedtothevertexfast,butstillithastowalkthroughthislistandcheckifallofthemhavetheattributetype=="friend".Usingavertex-centricindexwouldallowArangoDBtofindalledgesforthevertexhavingtheattributetype=="friend"inthesametimeandcansavetheiterationtoverifythecondition.
IndexcreationAvertex-centriccanbeeitherofthefollowingtypes:
HashIndexSkiplistIndexPersistentIndex
Andiscreatedusingtheircreationoperations.Howeverinthelistoffieldsusedtocreatetheindexwehavetoincludeeither_fromor_to.Letusagainexplainthisbyanexample.Assumewewanttocreateanhash-basedoutboundvertex-centricindexontheattributetype.Thiscanbecreatedwiththefollowingway:
arangosh>db.collection.ensureIndex({type:"hash",fields:["_from","type"]})
showexecutionresultsAlloptionsthataresupportedbytherespectiveindexesaresupportedbythevertex-centricindexaswell.
Indexusage
TheAQLoptimizercandecidetouseavertex-centricwheneversuitable,howeveritisnotguaranteedthatthisindexisused,theoptimizermayestimatethatanotherindexisassumedtobebetter.Theoptimizerwillconsiderthistypeofindexesonexplicitfilteringof_fromrespectively_to:
FORedgeINcollection
FILTERedge._from=="vertices/123456"ANDedge.type=="friend"
RETURNedge
andduringpatternmatchingqueries:
FORv,e,pIN3..5OUTBOUND@startGRAPH@graphName
FILTERp.edges[*].typeALL=="friend"
RETURNv
VertexCentricIndexes
122
ArangoDBGraphs
FirstStepswithGraphs
AGraphconsistsofverticesandedges.Edgesarestoredasdocumentsinedgecollections.Avertexcanbeadocumentofadocumentcollectionorofanedgecollection(soedgescanbeusedasvertices).Whichcollectionsareusedwithinanamedgraphisdefinedviaedgedefinitions.Anamedgraphcancontainmorethanoneedgedefinition,atleastoneisneeded.Graphsallowyoutostructureyourmodelsinlinewithyourdomainandgroupthemlogicallyincollectionsandgivingyouthepowertoquerytheminthesamegraphqueries.
Newtographs?Takeourfreegraphcourseforfreshersandgetfromzeroknowledgetoadvancedquerytechniques.
Comingfromarelationalbackground-what'sagraph?
InSQLyoucommonlyhavetheconstructofarelationtabletostoren:mrelationsbetweentwodatatables.Anedgecollectionissomewhatsimilartotheserelationtables;vertexcollectionsresemblethedatatableswiththeobjectstoconnect.WhilesimplegraphquerieswithfixednumberofhopsviatherelationtablemaybedoableinSQLwithseveralnestedjoins,graphdatabasescanhandleanarbitrarynumberofthesehopsoveredgecollections-thisiscalledtraversal.Alsoedgesinoneedgecollectionmaypointtoseveralvertexcollections.Itscommontohaveattributesattachedtoedges,i.e.alabelnamingthisinterconnection.Edgeshaveadirection,withtheirrelations_fromand_topointingfromonedocumenttoanotherdocumentstoredinvertexcollections.Inqueriesyoucandefineinwhichdirectionstheedgerelationsmaybefollowed(OUTBOUND:_from→_to,INBOUND:_from←_to,ANY:_from↔_to).
NamedGraphs
NamedgraphsarecompletelymanagedbyArangoDB,andthusalsovisibleinthewebinterface.TheyusethefullspectrumofArangoDB'sgraphfeatures.Youmayaccessthemviaseveralinterfaces.
AQLGraphOperationswithseveralflavors:AQLTraversalsonbothnamedandanonymousgraphsAQLShortestPathonbothnamedandanonymousgraph
JavaScriptGeneralGraphimplementation,asyoumayuseitinFoxxServicesGraphManagement;creating&manipualatinggraphdefinitions;inserting,updatinganddeletingverticesandedgesintographsGraphFunctionsforworkingwithedgesandvertices,toanalyzethemandtheirrelations
JavaScriptSmartGraphimplementation,forscalablegraphsSmartGraphManagement;creating&manipualatingSmartGraphdefinitions;DifferencestoGeneralGraph
RESTfulGeneralGraphinterfaceusedtoimplementgraphmanagementinclientdrivers
Manipulatingcollectionsofnamedgraphswithregulardocumentfunctions
Theunderlyingcollectionsofthenamedgraphsarestillaccessibleusingthestandardmethodsforcollections.Howeverthegraphmoduleaddsanadditionallayerontopofthesecollectionsgivingyouthefollowingguarantees:
AllmodificationsareexecutedtransactionalIfyoudeleteavertexalledgeswillbedeleted,youwillneverhavelooseendsIfyouinsertanedgeitischeckediftheedgematchestheedgedefinitions,youredgecollectionswillonlycontainvalidedges
TheseguaranteesarelostifyouaccessthecollectionsinanyotherwaythanthegraphmoduleorAQL,soifyoudeletedocumentsfromyourvertexcollectionsdirectly,theedgespointingtothemwillberemaininplace.
Anonymousgraphs
Sometimesyoumaynotneedallthepowersofnamedgraphs,butsomeofitsbitsmaybevaluabletoyou.YoumayuseanonymousgraphsinthetraversalsandintheWorkingwithEdgeschapter.Anonymousgraphsdon'thaveedgedefinitionsdescribingwhichvertexcollectionisconnectedbywhichedgecollection.Thegraphmodelhastobemaintainedintheclientsidecode.Thisgivesyoumorefreedomthanthestrictnamedgraphs.
Graphs
124
AQLGraphOperationsareavailableforboth,namedandanonymousgraphs:AQLTraversalsAQLShortestPath
Whentochooseanonymousornamedgraphs?
Asnotedabove,namedgraphsensuregraphintegrity,bothwheninsertingorremovingedgesorvertices.Soyouwon'tencounterdanglingedges,evenifyouusethesamevertexcollectioninseveralnamedgraphs.Thisinvolvesmoreoperationsinsidethedatabasewhichcomeatacost.Thereforeanonymousgraphsmaybefasterinmanyoperations.Sothisquestionmaybenarroweddownto:'CanIaffordtheadditionaleffortordoIneedthewarrantyforintegrity?'.
Multipleedgecollectionsvs.FILTERsonedgedocumentattributes
Ifyouwanttoonlytraverseedgesofaspecifictype,therearetwowaystoachievethis.Thefirstwouldbeanattributeintheedgedocument-i.e.type,whereyouspecifyadifferentiatorfortheedge-i.e."friends","family","married"or"workmates",soyoucanlaterFILTERe.type="friends"ifyouonlywanttofollowthefriendedges.
Anotherway,whichmaybemoreefficientinsomecases,istousedifferentedgecollectionsfordifferenttypesofedges,soyouhavefriend_edges,family_edges,married_edgesandworkmate_edgesascollectionnames.Youcanthenconfigureseveralnamedgraphsincludingasubsetoftheavailableedgeandvertexcollections-oryouuseanonymousgraphqueries,whereyouspecifyalistofedgecollectionstotakeintoaccountinthatquery.Toonlyfollowfriendedges,youwouldspecifyfriend_edgesassoleedgecollection.
Bothapproacheshaveadvantagesanddisadvantages.FILTERoperationsonedgeattributeswilldocomparisonsoneachtraversededge,whichmaybecomeCPU-intense.Whennotfindingtheedgesinthefirstplacebecauseofthecollectioncontainingthemisnottraversedatall,therewillneverbeareasontoactualycheckfortheirtypeattributewithFILTER.
Themultipleedgecollectionsapproachislimitedbythenumberofcollectionsthatcanbeusedsimultaneouslyinonequery.EverycollectionusedinaqueryrequiressomeresourcesinsideofArangoDBandthenumberisthereforelimitedtocaptheresourcerequirements.Youmayalsohaveconstraintsonotheredgeattributes,suchasahashindexwithauniqueconstraint,whichrequiresthedocumentstobeinasinglecollectionfortheuniquenessguarantee,anditmaythusnotbepossibletostorethedifferenttypesofedgesinmultipleedgecollections.
So,ifyouredgeshaveaboutadozendifferenttypes,it'sokaytochoosethecollectionapproach,otherwisetheFILTERapproachispreferred.YoucanstilluseFILTERoperationsonedgesofcourse.YoucangetridofaFILTERonthetypewiththeformerapproach,everythingelsecanstaythesame.
WhichpartofmydataisanEdgeandwhichaVertex?
Themainobjectsinyourdatamodel,suchasusers,groupsorarticles,areusuallyconsideredtobevertices.Foreachtypeofobject,adocumentcollection(alsocalledvertexcollection)shouldstoretheindividualentities.Entitiescanbeconnectedbyedgestoexpressandclassifyrelationsbetweenvertices.Itoftenmakessensetohaveanedgecollectionperrelationtype.
ArangoDBdoesnotrequireyoutostoreyourdataingraphstructureswithedgesandvertices,youcanalsodecidetoembedattributessuchaswhichgroupsauserispartof,or_idsofdocumentsinanotherdocumentinsteadofconnectingthedocumentswithedges.Itcanbeameaningfulperformanceoptimizationfor1:nrelationships,ifyourdataisnotfocusedonrelationsandyoudon'tneedgraphtraversalwithvaryingdepth.Itusuallymeanstointroducesomeredundancyandpossiblyinconsistenciesifyouembeddata,butitcanbeanacceptabletradeoff.
Vertices
Let'ssaywehavetwovertexcollections,UsersandGroups.DocumentsintheGroupscollectioncontaintheattributesoftheGroup,i.e.whenitwasfounded,itssubject,aniconURLandsoon.Usersdocumentscontainthedataspecifictoauser-likeallnames,birthdays,AvatarURLs,hobbies...
Edges
Wecanuseanedgecollectiontostorerelationsbetweenusersandgroups.Sincemultipleusersmaybeinanarbitrarynumberofgroups,thisisanm:nrelation.TheedgecollectioncanbecalledUsersInGroupswithi.e.oneedgewith_frompointingtoUsers/Johnand_topointingtoGroups/BowlingGroupHappyPin.ThismakestheuserJohnamemberofthegroupBowlingGroupHappyPin.
Graphs
125
Attributesofthisrelationmaycontainqualifierstothisrelation,likethepermissionsofJohninthisgroup,thedatewhenhejoinedthegroupetc.
Soroughlyput,ifyouusedocumentsandtheirattributesinasentence,nounswouldtypicallybevertices,verbsbecometheedges.Youcanseethisintheknowsgraphbelow:
AliceknowsBob,whointermknowsCharlie.
Advantagesofthisapproach
Graphsgiveyoutheadvantageofnotjustbeingabletohaveafixednumberofm:nrelationsinarow,butanarbitrarynumber.Edgescanbetraversedinbothdirections,soit'seasytodetermineallgroupsauserisin,butalsotofindoutwhichmembersacertaingrouphas.Userscouldalsobeinterconnectedtocreateasocialnetwork.
Usingthegraphdatamodel,dealingwithdatathathaslotsofrelationsstaysmanageableandcanbequeriedinveryflexibleways,whereasitwouldcauseheadachetohandleitinarelationaldatabasesystem.
Backupandrestore
Forsureyouwanttohavebackupsofyourgraphdata,youcanuseArangodumptocreatethebackup,andArangorestoretorestoreabackupintoanewArangoDB.Youshouldhowevernotethat:
youneedthesystemcollection_graphsifyoubackupnamedgraphs.youneedtobackupthecompletesetofalledgeandvertexcollectionsyourgraphconsistsof.Partialdump/restoremaynotwork.
Managinggraphs
Bydefaultyoushouldusetheinterfaceyourdriverprovidestomanagegraphs.
Thisisi.e.documentedinGraphs-SectionoftheArangoDBJavadriver.
ExampleGraphs
ArangoDBcomeswithasetofeasilygraspablegraphsthatareusedtodemonstratetheAPIs.Youcanusetheaddsamplestabinthecreategraphwindowinthewebinterface,orloadthemodule@arangodb/graph-examples/example-graphinarangoshanduseittocreateinstancesofthesegraphsinyourArangoDB.Onceyou'vecreatedthem,youcaninspecttheminthewebinterface-whichwasusedtocreatethepicturesbelow.
Youcaneasilylookintotheinnardsofthisscriptforreferenceabouthowtomanagegraphsprogramatically.
TheKnows_Graph
Graphs
126
Asetofpersonsknowingeachother:
Theknowsgraphconsistsofonevertexcollectionpersonsconnectedviaoneedgecollectionknows.ItwillcontainfivepersonsAlice,Bob,Charlie,DaveandEve.Wewillhavethefollowingdirectedrelations:
AliceknowsBobBobknowsCharlieBobknowsDaveEveknowsAliceEveknowsBob
Thisishowwecreateit,inspectitsverticesandedges,anddropitagain:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("knows_graph");
arangosh>db.persons.toArray()
arangosh>db.knows.toArray();
arangosh>examples.dropGraph("knows_graph");
showexecutionresults
TheSocialGraph
Asetofpersonsandtheirrelations:
Graphs
127
Thisexamplehasfemaleandmalepersonsasverticesintwovertexcollections-femaleandmale.Theedgesaretheirconnectionsintherelationedgecollection.Thisishowwecreateit,inspectitsverticesandedges,anddropitagain:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>db.female.toArray()
arangosh>db.male.toArray()
arangosh>db.relation.toArray()
arangosh>examples.dropGraph("social");
showexecutionresults
TheCityGraph
Asetofeuropeancities,andtheirfictionaltravelingdistancesasconnections:
Graphs
128
Theexamplehasthecitiesasverticesinseveralvertexcollections-germanCityandfrenchCity.Theedgesaretheirinterconnectionsinseveraledgecollectionsfrench/german/internationalHighway.Thisishowwecreateit,inspectitsedgesandvertices,anddropitagain:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("routeplanner");
arangosh>db.frenchCity.toArray();
arangosh>db.germanCity.toArray();
arangosh>db.germanHighway.toArray();
arangosh>db.frenchHighway.toArray();
arangosh>db.internationalHighway.toArray();
arangosh>examples.dropGraph("routeplanner");
showexecutionresults
TheTraversalGraph
Thisgraphwasdesignedtodemonstratefiltersintraversals.Ithassomelabelstofilteronit.
Graphs
129
Theexamplehasallitsverticesinthecirclescollection,andanedgesedgecollectiontoconnectthem.Circleshaveuniquenumericlabels.Edgeshavetwobooleanattributes(theFalsealwaysbeingfalse,theTruthalwaysbeingtrue)andalabelsortingB-Dtotheleftside,G-Ktotherightside.LeftandrightsidesplitintoPaths-atBandGwhichareeachdirectneighboursoftheroot-nodeA.StartingfromAthegraphhasadepthof3onallitspaths.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("traversalGraph");
arangosh>db.circles.toArray();
arangosh>db.edges.toArray();
arangosh>examples.dropGraph("traversalGraph");
showexecutionresults
TheWorldGraph
Theworldcountrygraphstructuresitsnodeslikethat:world→continent→country→capital.Insomecasesedgedirectionsaren'tforward(thereforeitwillbedisplayeddisjunctinthegraphviewer).Ithastwowaysofcreatingit.Oneusingthenamedgraphutilities(worldCountry),onewithout(worldCountryUnManaged).Itisusedtodemonstraterawtraversaloperations.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("worldCountry");
arangosh>db.worldVertices.toArray();
arangosh>db.worldEdges.toArray();
arangosh>examples.dropGraph("worldCountry");
arangosh>varg=examples.loadGraph("worldCountryUnManaged");
arangosh>examples.dropGraph("worldCountryUnManaged");
showexecutionresults
Graphs
130
Cookbookexamples
TheabovereferencedchaptersdescribethevariousAPIsofArangoDBsgraphenginewithsmallexamples.Ourcookbookhassomemorereallifeexamples:
TraversingagraphinfulldepthUsinganexamplevertexwiththejavadriverRetrievingdocumentsfromArangoDBwithoutknowingthestructureUsingacustomvisitorfromnode.jsAQLExampleQueriesonanActorsandMoviesDatabase
Highervolumegraphexamples
Alloftheaboveexamplesarerathersmallsotheyareeasiertocomprehendandcandemonstratethewaythefunctionalityworks.Therearehoweverseveraldatasetsfreelyavailableonthewebthatarealotbigger.Wecollectedsomeofthemwithimportscriptssoyoumayplayaroundwiththem.AnotherhugegraphisthePokecsocialnetworkfromSlovakiathatweusedforperformancetestingonseveraldatabases;Youwillfindimportingscriptsetc.inthisblogpost.
Graphs
131
GraphsThischapterdescribesthegeneral-graphmodule.Itallowsyoutodefineagraphthatisspreadacrossseveraledgeanddocumentcollections.Thisallowsyoutostructureyourmodelsinlinewithyourdomainandgroupthemlogicallyincollectionsgivingyouthepowertoquerytheminthesamegraphqueries.Thereisnoneedtoincludethereferencedcollectionswithinthequery,thismodulewillhandleitforyou.
ThreeStepstocreateagraph
Createagraph
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>vargraph=graph_module._create("myGraph");
arangosh>graph;
{[Graph]
}
Addsomevertexcollections
arangosh>graph._addVertexCollection("shop");
arangosh>graph._addVertexCollection("customer");
arangosh>graph._addVertexCollection("pet");
arangosh>graph;
showexecutionresultsDefinerelationsontheGraph
arangosh>varrel=graph_module._relation("isCustomer",["shop"],["customer"]);
arangosh>graph._extendEdgeDefinitions(rel);
arangosh>graph;
showexecutionresults
GeneralGraphs
132
GraphManagementThischapterdescribesthejavascriptinterfaceforcreatingandmodifyingnamedgraphs.Inordertocreateanonemptygraphthefunctionalitytocreateedgedefinitionshastobeintroducedfirst:
EdgeDefinitions
Anedgedefinitionisalwaysadirectedrelationofagraph.Eachgraphcanhavearbitrarymanyrelationsdefinedwithintheedgedefinitionsarray.
Initializethelist
Createalistofedgedefinitionstoconstructagraph.
graph_module._edgeDefinitions(relation1,relation2,...,relationN)
Thelistofedgedefinitionsofagraphcanbemanagedbythegraphmoduleitself.Thisfunctionistheentrypointforthemanagementandwillreturnthecorrectlist.
Parameters
relationX(optional)Anobjectrepresentingadefinitionofonerelationinthegraph
Examples
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>directed_relation=graph_module._relation("lives_in","user","city");
arangosh>undirected_relation=graph_module._relation("knows","user","user");
arangosh>edgedefinitions=graph_module._edgeDefinitions(directed_relation,
undirected_relation);
showexecutionresults
Extendthelist
Extendthelistofedgedefinitionstoconstructagraph.
graph_module._extendEdgeDefinitions(edgeDefinitions,relation1,relation2,...,relationN)
Inordertoaddmoreedgedefinitionstothegraphbeforecreatingthisfunctioncanbeusedtoaddmoredefinitionstotheinitiallist.
Parameters
edgeDefinitions(required)Alistofrelationdefinitionobjects.relationX(required)Anobjectrepresentingadefinitionofonerelationinthegraph
Examples
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>directed_relation=graph_module._relation("lives_in","user","city");
arangosh>undirected_relation=graph_module._relation("knows","user","user");
arangosh>edgedefinitions=graph_module._edgeDefinitions(directed_relation);
arangosh>edgedefinitions=graph_module._extendEdgeDefinitions(undirected_relation);
showexecutionresults
Relation
Defineadirectedrelation.
GraphManagement
133
graph_module._relation(relationName,fromVertexCollections,toVertexCollections)
TherelationNamedefinesthenameofthisrelationandreferencestotheunderlyingedgecollection.ThefromVertexCollectionsisanArrayofdocumentcollectionsholdingthestartvertices.ThetoVertexCollectionsisanArrayofdocumentcollectionsholdingthetargetvertices.RelationsareonlyallowedinthedirectionfromanycollectioninfromVertexCollectionstoanycollectionintoVertexCollections.
Parameters
relationName(required)Thenameoftheedgecollectionwheretheedgesshouldbestored.Willbecreatedifitdoesnotyetexist.fromVertexCollections(required)Oneoralistofcollectionnames.Sourceverticesfortheedgeshavetobestoredinthesecollections.Collectionswillbecreatediftheydonotexist.toVertexCollections(required)Oneoralistofcollectionnames.Targetverticesfortheedgeshavetobestoredinthesecollections.Collectionswillbecreatediftheydonotexist.
Examples
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>graph_module._relation("has_bought",["Customer","Company"],["Groceries",
"Electronics"]);
showexecutionresults
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>graph_module._relation("has_bought","Customer","Product");
showexecutionresults
CreateagraphAfterhavingintroducededgedefinitionsagraphcanbecreated.
Createagraph
graph_module._create(graphName,edgeDefinitions,orphanCollections)
Thecreationofagraphrequiresthenameofthegraphandadefinitionofitsedges.
Foreverytypeofedgedefinitionaconveniencemethodexiststhatcanbeusedtocreateagraph.Optionallyalistofvertexcollectionscanbeadded,whicharenotusedinanyedgedefinition.Thesecollectionsarereferredtoasorphancollectionswithinthischapter.Allcollectionsusedwithinthecreationprocessarecreatediftheydonotexist.
Parameters
graphName(required)UniqueidentifierofthegraphedgeDefinitions(optional)ListofrelationdefinitionobjectsorphanCollections(optional)Listofadditionalvertexcollectionnames
Examples
Createanemptygraph,edgedefinitionscanbeaddedatruntime:
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>graph=graph_module._create("myGraph");
{[Graph]
}
Createagraphusinganedgecollectionedgesandasinglevertexcollectionvertices
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>varedgeDefinitions=[{collection:"edges","from":["vertices"],"to":[
"vertices"]}];
GraphManagement
134
arangosh>graph=graph_module._create("myGraph",edgeDefinitions);
{[Graph]
"edges":[ArangoCollection16644,"edges"(typeedge,statusloaded)],
"vertices":[ArangoCollection16639,"vertices"(typedocument,statusloaded)]
}
Createagraphwithedgedefinitionsandorphancollections:
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>graph=graph_module._create("myGraph",
........>[graph_module._relation("myRelation",["male","female"],["male","female"])],
["sessions"]);
showexecutionresults
CompleteExampletocreateagraph
ExampleCall:
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>varedgeDefinitions=graph_module._edgeDefinitions();
arangosh>graph_module._extendEdgeDefinitions(edgeDefinitions,
graph_module._relation("friend_of","Customer","Customer"));
arangosh>graph_module._extendEdgeDefinitions(
........>edgeDefinitions,graph_module._relation(
........>"has_bought",["Customer","Company"],["Groceries","Electronics"]));
arangosh>graph_module._create("myStore",edgeDefinitions);
showexecutionresultsalternativecall:
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>varedgeDefinitions=graph_module._edgeDefinitions(
........>graph_module._relation("friend_of",["Customer"],["Customer"]),
graph_module._relation(
........>"has_bought",["Customer","Company"],["Groceries","Electronics"]));
arangosh>graph_module._create("myStore",edgeDefinitions);
showexecutionresults
Listavailablegraphs
Listallgraphs.
graph_module._list()
Listsallgraphnamesstoredinthisdatabase.
Examples
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>graph_module._list();
[]
Loadagraph
GraphManagement
135
Getagraph
graph_module._graph(graphName)
Agraphcanberetrievedbyitsname.
Parameters
graphName(required)Uniqueidentifierofthegraph
Examples
Getagraph:
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>graph=graph_module._graph("social");
showexecutionresults
Removeagraph
Removeagraph
graph_module._drop(graphName,dropCollections)
Agraphcanbedroppedbyitsname.Thiscandropallcollectionscontainedinthegraphaslongastheyarenotusedwithinothergraphs.Todropthecollectionsonlybelongingtothisgraph,theoptionalparameterdrop-collectionshastobesettotrue.
Parameters
graphName(required)UniqueidentifierofthegraphdropCollections(optional)Defineifcollectionsshouldbedropped(default:false)
Examples
Dropagraphandkeepcollections:
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>graph_module._drop("social");
true
arangosh>db._collection("female");
[ArangoCollection16731,"female"(typedocument,statusloaded)]
arangosh>db._collection("male");
[ArangoCollection16734,"male"(typedocument,statusloaded)]
arangosh>db._collection("relation");
[ArangoCollection16737,"relation"(typeedge,statusloaded)]
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>graph_module._drop("social",true);
true
arangosh>db._collection("female");
null
arangosh>db._collection("male");
null
arangosh>db._collection("relation");
null
ModifyagraphdefinitionduringruntimeAfteryouhavecreatedangraphitsdefinitionisnotimmutable.Youcanstilladd,deleteormodifyedgedefinitionsandvertexcollections.
GraphManagement
136
Extendtheedgedefinitions
Addanotheredgedefinitiontothegraph
graph._extendEdgeDefinitions(edgeDefinition)
Extendstheedgedefinitionsofagraph.Ifanorphancollectionisusedinthisedgedefinition,itwillberemovedfromtheorphanage.Iftheedgecollectionoftheedgedefinitiontoaddisalreadyusedinthegraphorusedinadifferentgraphwithdifferentfromand/ortocollectionsanerroristhrown.
Parameters
edgeDefinition(required)Therelationdefinitiontoextendthegraph
Examples
arangosh>vargraph_module=require("@arangodb/general-graph")
arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);
arangosh>vared2=graph_module._relation("myEC2",["myVC1"],["myVC3"]);
arangosh>vargraph=graph_module._create("myGraph",[ed1]);
arangosh>graph._extendEdgeDefinitions(ed2);
Modifyanedgedefinition
Modifyanrelationdefinition
graph_module._editEdgeDefinition(edgeDefinition)
Editsonerelationdefinitionofagraph.Theedgedefinitionusedasargumentwillreplacetheexistingedgedefinitionofthegraphwhichhasthesamecollection.VertexCollectionsofthereplacededgedefinitionthatarenotusedinthenewdefinitionwilltransformtoanorphan.Orphansthatareusedinthisnewedgedefinitionwillbedeletedfromthelistoforphans.Othergraphswiththesameedgedefinitionwillbemodified,too.
Parameters
edgeDefinition(required)Theedgedefinitiontoreplacetheexistingedgedefinitionwiththesameattributecollection.
Examples
arangosh>vargraph_module=require("@arangodb/general-graph")
arangosh>varoriginal=graph_module._relation("myEC1",["myVC1"],["myVC2"]);
arangosh>varmodified=graph_module._relation("myEC1",["myVC2"],["myVC3"]);
arangosh>vargraph=graph_module._create("myGraph",[original]);
arangosh>graph._editEdgeDefinitions(modified);
Deleteanedgedefinition
Deleteonerelationdefinition
graph_module._deleteEdgeDefinition(edgeCollectionName,dropCollection)
Deletesarelationdefinitiondefinedbytheedgecollectionofagraph.Ifthecollectionsdefinedintheedgedefinition(collection,from,to)arenotusedinanotheredgedefinitionofthegraph,theywillbemovedtotheorphanage.
Parameters
edgeCollectionName(required)Nameofedgecollectionintherelationdefinition.dropCollection(optional)Defineiftheedgecollectionshouldbedropped.Defaultfalse.
Examples
Removeanedgedefinitionbutkeeptheedgecollection:
arangosh>vargraph_module=require("@arangodb/general-graph")
GraphManagement
137
arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);
arangosh>vared2=graph_module._relation("myEC2",["myVC1"],["myVC3"]);
arangosh>vargraph=graph_module._create("myGraph",[ed1,ed2]);
arangosh>graph._deleteEdgeDefinition("myEC1");
arangosh>db._collection("myEC1");
[ArangoCollection22374,"myEC1"(typeedge,statusloaded)]
Removeanedgedefinitionanddroptheedgecollection:
arangosh>vargraph_module=require("@arangodb/general-graph")
arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);
arangosh>vared2=graph_module._relation("myEC2",["myVC1"],["myVC3"]);
arangosh>vargraph=graph_module._create("myGraph",[ed1,ed2]);
arangosh>graph._deleteEdgeDefinition("myEC1",true);
arangosh>db._collection("myEC1");
null
ExtendvertexCollections
Eachgraphcanhaveanarbitraryamountofvertexcollections,whicharenotpartofanyedgedefinitionofthegraph.Thesecollectionsarecalledorphancollections.Ifthegraphisextendedwithanedgedefinitionusingoneoftheorphans,itwillberemovedfromthesetoforphancollectionautomatically.
Addavertexcollection
Addavertexcollectiontothegraph
graph._addVertexCollection(vertexCollectionName,createCollection)
Addsavertexcollectiontothesetoforphancollectionsofthegraph.Ifthecollectiondoesnotexist,itwillbecreated.Ifitisalreadyusedbyanyedgedefinitionofthegraph,anerrorwillbethrown.
Parameters
vertexCollectionName(required)Nameofvertexcollection.createCollection(optional)Iftruethecollectionwillbecreatedifitdoesnotexist.Default:true.
Examples
arangosh>vargraph_module=require("@arangodb/general-graph");
arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);
arangosh>vargraph=graph_module._create("myGraph",[ed1]);
arangosh>graph._addVertexCollection("myVC3",true);
Gettheorphanedcollections
Getallorphancollections
graph._orphanCollections()
Returnsallvertexcollectionsofthegraphthatarenotusedinanyedgedefinition.
Examples
arangosh>vargraph_module=require("@arangodb/general-graph")
arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);
arangosh>vargraph=graph_module._create("myGraph",[ed1]);
arangosh>graph._addVertexCollection("myVC3",true);
arangosh>graph._orphanCollections();
GraphManagement
138
[
"myVC3"
]
Removeavertexcollection
Removeavertexcollectionfromthegraph
graph._removeVertexCollection(vertexCollectionName,dropCollection)
Removesavertexcollectionfromthegraph.Onlycollectionsnotusedinanyrelationdefinitioncanberemoved.Optionallythecollectioncanbedeleted,ifitisnotusedinanyothergraph.
Parameters
vertexCollectionName(required)Nameofvertexcollection.dropCollection(optional)Iftruethecollectionwillbedroppedifitisnotusedinanyothergraph.Default:false.
Examples
arangosh>vargraph_module=require("@arangodb/general-graph")
arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);
arangosh>vargraph=graph_module._create("myGraph",[ed1]);
arangosh>graph._addVertexCollection("myVC3",true);
arangosh>graph._addVertexCollection("myVC4",true);
arangosh>graph._orphanCollections();
arangosh>graph._removeVertexCollection("myVC3");
arangosh>graph._orphanCollections();
showexecutionresults
ManiuplatingVertices
Saveavertex
CreateanewvertexinvertexCollectionName
graph.vertexCollectionName.save(data)
Parameters
data(required)JSONdataofvertex.
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.male.save({name:"Floyd",_key:"floyd"});
showexecutionresults
Replaceavertex
ReplacesthedataofavertexincollectionvertexCollectionName
graph.vertexCollectionName.replace(vertexId,data,options)
Parameters
vertexId(required)_idattributeofthevertexdata(required)JSONdataofvertex.
GraphManagement
139
options(optional)Seecollectiondocumentation
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.male.save({neym:"Jon",_key:"john"});
arangosh>graph.male.replace("male/john",{name:"John"});
showexecutionresults
Updateavertex
UpdatesthedataofavertexincollectionvertexCollectionName
graph.vertexCollectionName.update(vertexId,data,options)
Parameters
vertexId(required)_idattributeofthevertexdata(required)JSONdataofvertex.options(optional)Seecollectiondocumentation
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.female.save({name:"Lynda",_key:"linda"});
arangosh>graph.female.update("female/linda",{name:"Linda",_key:"linda"});
showexecutionresults
Removeavertex
RemovesavertexincollectionvertexCollectionName
graph.vertexCollectionName.remove(vertexId,options)
Additionallyremovesallingoingandoutgoingedgesofthevertexrecursively(seeedgeremove).
Parameters
vertexId(required)_idattributeofthevertexoptions(optional)Seecollectiondocumentation
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.male.save({name:"Kermit",_key:"kermit"});
arangosh>db._exists("male/kermit")
arangosh>graph.male.remove("male/kermit")
arangosh>db._exists("male/kermit")
showexecutionresults
ManipulatingEdges
Saveanewedge
GraphManagement
140
CreatesanedgefromvertexfromtovertextoincollectionedgeCollectionName
graph.edgeCollectionName.save(from,to,data,options)
Parameters
from(required)_idattributeofthesourcevertexto(required)_idattributeofthetargetvertexdata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.relation.save("male/bob","female/alice",{type:"married",_key:
"bobAndAlice"});
showexecutionresultsIfthecollectionsoffromandtoarenotdefinedinanedgedefinitionofthegraph,theedgewillnotbestored.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.relation.save(
........>"relation/aliceAndBob",
........>"female/alice",
........>{type:"married",_key:"bobAndAlice"});
[ArangoError1906:invalidedgebetweenrelation/aliceAndBobandfemale/alice.Doesn't
conformtoanyedgedefinition]
Replaceanedge
ReplacesthedataofanedgeincollectionedgeCollectionName.Notethat_fromand_toaremandatory.
graph.edgeCollectionName.replace(edgeId,data,options)
Parameters
edgeId(required)_idattributeoftheedgedata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.relation.save("female/alice","female/diana",{typo:"nose",_key:
"aliceAndDiana"});
arangosh>graph.relation.replace("relation/aliceAndDiana",{type:"knows",_from:
"female/alice",_to:"female/diana"});
showexecutionresults
Updateanedge
UpdatesthedataofanedgeincollectionedgeCollectionName
graph.edgeCollectionName.update(edgeId,data,options)
Parameters
GraphManagement
141
edgeId(required)_idattributeoftheedgedata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.relation.save("female/alice","female/diana",{type:"knows",_key:
"aliceAndDiana"});
arangosh>graph.relation.update("relation/aliceAndDiana",{type:"quarreled",_key:
"aliceAndDiana"});
showexecutionresults
Removeanedge
RemovesanedgeincollectionedgeCollectionName
graph.edgeCollectionName.remove(edgeId,options)
Ifthisedgeisusedasavertexbyanotheredge,theotheredgewillberemoved(recursively).
Parameters
edgeId(required)_idattributeoftheedgeoptions(optional)Seecollectiondocumentation
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>graph.relation.save("female/alice","female/diana",{_key:"aliceAndDiana"});
arangosh>db._exists("relation/aliceAndDiana")
arangosh>graph.relation.remove("relation/aliceAndDiana")
arangosh>db._exists("relation/aliceAndDiana")
showexecutionresults
Connectedges
Getallconnectingedgesbetween2groupsofverticesdefinedbytheexamples
graph._getConnectingEdges(vertexExample,vertexExample2,options)
Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.
Parameters
vertexExample1(optional)SeeDefinitionofexamplesvertexExample2(optional)SeeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:
edgeExamples:Filtertheedges,seeDefinitionofexamplesedgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.vertex1CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.vertex2CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.
Examples
Arouteplannerexample,allconnectingedgesbetweencapitals.
GraphManagement
142
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._getConnectingEdges({isCapital:true},{isCapital:true});
[]
GraphManagement
143
GraphFunctionsThischapterdescribesvariousfunctionsonagraph.Alotoftheseacceptavertex(oredge)exampleasparameterasdefinedinthenextsection.
ExampleswillexplaintheAPIonthethecitygraph:
DefinitionofexamplesFormanyofthefollowingfunctionsexamplescanbepassedinasaparameter.Examplesareusedtofiltertheresultsetforobjectsthatmatchtheconditions.Theseexamplescanhavethefollowingvalues:
null,thereisnomatchingexecutedallfoundresultsarevalid.Astring,onlyresultsarereturned,which_idequalthevalueofthestringAnexampleobject,definingasetofattributes.Onlyresultshavingtheseattributesarematched.Alistcontainingexampleobjectsand/orstrings.Allresultsmatchingatleastoneoftheelementsinthelistarereturned.
Getverticesfromedges.
Getvertexfromofanedge
Getthesourcevertexofanedge
graph._fromVertex(edgeId)
Returnsthevertexdefinedwiththeattribute_fromoftheedgewithedgeIdasits_id.
Parameters
edgeId(required)_idattributeoftheedge
Examples
GraphFunctions
144
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>varany=require("@arangodb").db.relation.any();
arangosh>graph._fromVertex("relation/"+any._key);
showexecutionresults
Getvertextoofanedge
Getthetargetvertexofanedge
graph._toVertex(edgeId)
Returnsthevertexdefinedwiththeattribute_tooftheedgewithedgeIdasits_id.
Parameters
edgeId(required)_idattributeoftheedge
Examples
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("social");
arangosh>varany=require("@arangodb").db.relation.any();
arangosh>graph._toVertex("relation/"+any._key);
showexecutionresults
_neighbors
Getallneighborsoftheverticesdefinedbytheexample
graph._neighbors(vertexExample,options)
Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.ThecomplexityofthismethodisO(n*m^x)withnbeingtheverticesdefinedbytheparametervertexExamplex,mtheaverageamountofneighborsandxthemaximaldepths.HencethedefaultcallwouldhaveacomplexityofO(n*m);
Parameters
vertexExample(optional)SeeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:
direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeExamples:Filtertheedges,seeDefinitionofexamplesneighborExamples:Filtertheneighborvertices,seeDefinitionofexamplesedgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.vertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.minDepth:Definestheminimalnumberofintermediatestepstoneighbors(defaultis1).maxDepth:Definesthemaximalnumberofintermediatestepstoneighbors(defaultis1).
Examples
Arouteplannerexample,allneighborsofcapitals.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._neighbors({isCapital:true});
showexecutionresultsArouteplannerexample,alloutboundneighborsofHamburg.
GraphFunctions
145
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._neighbors('germanCity/Hamburg',{direction:'outbound',maxDepth:2});
showexecutionresults
_commonNeighbors
Getallcommonneighborsoftheverticesdefinedbytheexamples.
graph._commonNeighbors(vertex1Example,vertex2Examples,optionsVertex1,optionsVertex2)
Thisfunctionreturnstheintersectionofgraph_module._neighbors(vertex1Example,optionsVertex1)andgraph_module._neighbors(vertex2Example,optionsVertex2).Forparameterdocumentationsee_neighbors.
ThecomplexityofthismethodisO(n*m^x)withnbeingthemaximalamountofverticesdefinedbytheparametersvertexExamples,mtheaverageamountofneighborsandxthemaximaldepths.HencethedefaultcallwouldhaveacomplexityofO(n*m);
Examples
Arouteplannerexample,allcommonneighborsofcapitals.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._commonNeighbors({isCapital:true},{isCapital:true});
showexecutionresultsArouteplannerexample,allcommonoutboundneighborsofHamburgwithanyotherlocationwhichhaveamaximaldepthof2:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._commonNeighbors(
........>'germanCity/Hamburg',
........>{},
........>{direction:'outbound',maxDepth:2},
........>{direction:'outbound',maxDepth:2});
showexecutionresults
_countCommonNeighborsGettheamountofcommonneighborsoftheverticesdefinedbytheexamples.
graph._countCommonNeighbors(vertex1Example,vertex2Examples,optionsVertex1,optionsVertex2)
Similarto_commonNeighborsbutreturnscountinsteadoftheelements.
Examples
Arouteplannerexample,allcommonneighborsofcapitals.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>varexample={isCapital:true};
arangosh>varoptions={includeData:true};
arangosh>graph._countCommonNeighbors(example,example,options,options);
showexecutionresults
GraphFunctions
146
Arouteplannerexample,allcommonoutboundneighborsofHamburgwithanyotherlocationwhichhaveamaximaldepthof2:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>varoptions={direction:'outbound',maxDepth:2,includeData:true};
arangosh>graph._countCommonNeighbors('germanCity/Hamburg',{},options,options);
showexecutionresults
_commonPropertiesGettheverticesofthegraphthatsharecommonproperties.
graph._commonProperties(vertex1Example,vertex2Examples,options)
Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertex1Exampleandvertex2Example.
ThecomplexityofthismethodisO(n)withnbeingthemaximalamountofverticesdefinedbytheparametersvertexExamples.
Parameters
vertex1Examples(optional)Filterthesetofsourcevertices,seeDefinitionofexamples
vertex2Examples(optional)Filterthesetofverticescomparedto,seeDefinitionofexamples
options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:vertex1CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbesearchedforsourcevertices.vertex2CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbesearchedforcomparevertices.ignoreProperties:Oneoralistofattributenamesofadocumentthatshouldbeignored.
Examples
Arouteplannerexample,alllocationswiththesameproperties:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._commonProperties({},{});
showexecutionresultsArouteplannerexample,allcitieswhichsharesamepropertiesexceptforpopulation.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._commonProperties({},{},{ignoreProperties:'population'});
showexecutionresults
_countCommonPropertiesGettheamountofverticesofthegraphthatsharecommonproperties.
graph._countCommonProperties(vertex1Example,vertex2Examples,options)
Similarto_commonPropertiesbutreturnscountinsteadoftheobjects.
Examples
Arouteplannerexample,alllocationswiththesameproperties:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
GraphFunctions
147
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._countCommonProperties({},{});
showexecutionresultsArouteplannerexample,allGermancitieswhichsharesamepropertiesexceptforpopulation.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._countCommonProperties({},{},{vertex1CollectionRestriction:
'germanCity',
........>vertex2CollectionRestriction:'germanCity',ignoreProperties:'population'});
showexecutionresults
_paths
The_pathsfunctionreturnsallpathsofagraph.
graph._paths(options)
Thisfunctiondeterminesallavailablepathsinagraph.
ThecomplexityofthismethodisO(n*n*m)withnbeingtheamountofverticesinthegraphandmtheaverageamountofconnectededges;
Parameters
options(optional)Anobjectcontainingoptions,seebelow:direction:Thedirectionoftheedges.Possiblevaluesareany,inboundandoutbound(default).followCycles(optional):Ifsettotruethequeryfollowscyclesinthegraph,defaultisfalse.minLength(optional):Definestheminimallengthapathmusthavetobereturned(defaultis0).maxLength(optional):Definesthemaximallengthapathmusthavetobereturned(defaultis10).
Examples
Returnallpathsofthegraph"social":
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("social");
arangosh>g._paths();
showexecutionresultsReturnallinboundpathsofthegraph"social"withamaximallengthof1andaminimallengthof2:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("social");
arangosh>g._paths({direction:'inbound',minLength:1,maxLength:2});
showexecutionresults
_shortestPath
The_shortestPathfunctionreturnsallshortestpathsofagraph.
graph._shortestPath(startVertexExample,endVertexExample,options)
GraphFunctions
148
Thisfunctiondeterminesallshortestpathsinagraph.Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforstartandendvertex.Thelengthofapathisbydefaulttheamountofedgesfromonestartvertextoanendvertex.Theoptionweightallowstheusertodefineanedgeattributerepresentingthelength.
Parameters
startVertexExample(optional)AnexampleforthedesiredstartVertices(seeDefinitionofexamples).endVertexExample(optional)AnexampleforthedesiredendVertices(seeDefinitionofexamples).options(optional)Anobjectcontainingoptions,seebelow:
direction:Thedirectionoftheedgesasastring.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneormultipleedgecollectionnames.Onlyedgesfromthesecollectionswillbeconsideredforthepath.startVertexCollectionRestriction:Oneormultiplevertexcollectionnames.Onlyverticesfromthesecollectionswillbeconsideredasstartvertexofapath.endVertexCollectionRestriction:Oneormultiplevertexcollectionnames.Onlyverticesfromthesecollectionswillbeconsideredasendvertexofapath.weight:Thenameoftheattributeoftheedgescontainingthelengthasastring.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedaslength.IfnodefaultissuppliedthedefaultwouldbepositiveInfinitysothepathcouldnotbecalculated.
Examples
Arouteplannerexample,shortestpathfromallgermantoallfrenchcities:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("routeplanner");
arangosh>g._shortestPath({},{},{weight:'distance',endVertexCollectionRestriction:
'frenchCity',
........>startVertexCollectionRestriction:'germanCity'});
showexecutionresultsArouteplannerexample,shortestpathfromHamburgandColognetoLyon:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("routeplanner");
arangosh>g._shortestPath([{_id:'germanCity/Cologne'},{_id:'germanCity/Munich'}],
'frenchCity/Lyon',
........>{weight:'distance'});
showexecutionresults
_distanceTo
The_distanceTofunctionreturnsallpathsandtheredistancewithinagraph.
graph._distanceTo(startVertexExample,endVertexExample,options)
Thisfunctionisawrapperofgraph._shortestPath.Itdoesnotreturntheactualpathbutonlythedistancebetweentwovertices.
Examples
Arouteplannerexample,shortestdistancefromallgermantoallfrenchcities:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("routeplanner");
arangosh>g._distanceTo({},{},{weight:'distance',endVertexCollectionRestriction:
'frenchCity',
........>startVertexCollectionRestriction:'germanCity'});
GraphFunctions
149
showexecutionresultsArouteplannerexample,shortestdistancefromHamburgandColognetoLyon:
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>varg=examples.loadGraph("routeplanner");
arangosh>g._distanceTo([{_id:'germanCity/Cologne'},{_id:'germanCity/Munich'}],
'frenchCity/Lyon',
........>{weight:'distance'});
showexecutionresults
_absoluteEccentricityGettheeccentricityoftheverticesdefinedbytheexamples.
graph._absoluteEccentricity(vertexExample,options)
Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.
Parameters
vertexExample(optional)Filterthevertices,seeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:
direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.startVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredforsourcevertices.endVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredfortargetvertices.weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheeccentricitycannotbecalculated.
Examples
Arouteplannerexample,theabsoluteeccentricityofalllocations.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._absoluteEccentricity({});
showexecutionresultsArouteplannerexample,theabsoluteeccentricityofalllocations.Thisconsiderstheactualdistances.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._absoluteEccentricity({},{weight:'distance'});
showexecutionresultsArouteplannerexample,theabsoluteeccentricityofallcitiesregardingonlyoutboundpaths.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._absoluteEccentricity({},{startVertexCollectionRestriction:
'germanCity',
........>direction:'outbound',weight:'distance'});
showexecutionresults
GraphFunctions
150
_eccentricity
Getthenormalizedeccentricityoftheverticesdefinedbytheexamples.
graph._eccentricity(vertexExample,options)
Similarto_absoluteEccentricitybutreturnsanormalizedresult.
Examples
Arouteplannerexample,theeccentricityofalllocations.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._eccentricity();
showexecutionresultsArouteplannerexample,theweightedeccentricity.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._eccentricity({weight:'distance'});
showexecutionresults
_absoluteCloseness
Gettheclosenessoftheverticesdefinedbytheexamples.
graph._absoluteCloseness(vertexExample,options)
Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.
Parameters
vertexExample(optional)Filterthevertices,seeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:
direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.startVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredforsourcevertices.endVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredfortargetvertices.weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheclosenesscannotbecalculated.
Examples
Arouteplannerexample,theabsoluteclosenessofalllocations.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._absoluteCloseness({});
showexecutionresultsArouteplannerexample,theabsoluteclosenessofalllocations.Thisconsiderstheactualdistances.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
GraphFunctions
151
arangosh>graph._absoluteCloseness({},{weight:'distance'});
showexecutionresultsArouteplannerexample,theabsoluteclosenessofallGermanCitiesregardingonlyoutboundpaths.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._absoluteCloseness({},{startVertexCollectionRestriction:'germanCity',
........>direction:'outbound',weight:'distance'});
showexecutionresults
_closeness
Getthenormalizedclosenessofgraphsvertices.
graph._closeness(options)
Similarto_absoluteClosenessbutreturnsanormalizedvalue.
Examples
Arouteplannerexample,thenormalizedclosenessofalllocations.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._closeness();
showexecutionresultsArouteplannerexample,theclosenessofalllocations.Thisconsiderstheactualdistances.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._closeness({weight:'distance'});
showexecutionresultsArouteplannerexample,theclosenessofallcitiesregardingonlyoutboundpaths.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._closeness({direction:'outbound',weight:'distance'});
showexecutionresults
_absoluteBetweennessGetthebetweennessofallverticesinthegraph.
graph._absoluteBetweenness(vertexExample,options)
Parameters
vertexExample(optional)Filterthevertices,seeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:
direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthis
GraphFunctions
152
defaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencethebetweenesscannotbecalculated.
Examples
Arouteplannerexample,theabsolutebetweennessofalllocations.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._absoluteBetweenness({});
showexecutionresultsArouteplannerexample,theabsolutebetweennessofalllocations.Thisconsiderstheactualdistances.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._absoluteBetweenness({weight:'distance'});
{
}
Arouteplannerexample,theabsolutebetweennessofallcitiesregardingonlyoutboundpaths.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._absoluteBetweenness({direction:'outbound',weight:'distance'});
{
}
_betweenness
Getthenormalizedbetweennessofgraphsvertices.
graph_module._betweenness(options)
Similarto_absoluteBetweenessbutreturnsnormalizedvalues.
Examples
Arouteplannerexample,thebetweennessofalllocations.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._betweenness();
showexecutionresultsArouteplannerexample,thebetweennessofalllocations.Thisconsiderstheactualdistances.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._betweenness({weight:'distance'});
showexecutionresultsArouteplannerexample,thebetweennessofallcitiesregardingonlyoutboundpaths.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._betweenness({direction:'outbound',weight:'distance'});
GraphFunctions
153
showexecutionresults
_radiusGettheradiusofagraph.
`
Parameters
options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheradiuscannotbecalculated.
Examples
Arouteplannerexample,theradiusofthegraph.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._radius();
1
Arouteplannerexample,theradiusofthegraph.Thisconsiderstheactualdistances.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._radius({weight:'distance'});
1
Arouteplannerexample,theradiusofthegraphregardingonlyoutboundpaths.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._radius({direction:'outbound',weight:'distance'});
1
_diameter
Getthediameterofagraph.
graph._diameter(graphName,options)
Parameters
options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheradiuscannotbecalculated.
Examples
GraphFunctions
154
Arouteplannerexample,thediameterofthegraph.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._diameter();
1
Arouteplannerexample,thediameterofthegraph.Thisconsiderstheactualdistances.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._diameter({weight:'distance'});
1
Arouteplannerexample,thediameterofthegraphregardingonlyoutboundpaths.
arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");
arangosh>vargraph=examples.loadGraph("routeplanner");
arangosh>graph._diameter({direction:'outbound',weight:'distance'});
1
GraphFunctions
155
SmartGraphsThisfeatureisonlyavailableintheEnterpriseEdition.
Thischapterdescribesthesmart-graphmodule.Itenablesyoutomanagegraphsatscale,itwillgiveavastperformancebenefitforallgraphsshardedinanArangoDBCluster.Onasingleserverthisfeatureispointless,henceitisonlyavailableinaclustermode.IntermsofqueryingthereisnodifferencebetweensmartandGeneralGraphs.Theformerareatransparentreplacementforthelatter.SoforqueryingthegraphpleaserefertoAQLGraphOperationsandGraphFunctionssections.TheoptimizeriscleverenoughtoidentifyifweareonaSmartGraphornot.
Thedifferenceisonlyinthemanagementsection:creatingandmodifyingtheunderlyingcollectionsofthegraph.ForadetailedAPIreferencepleaserefertoSmartGraphManagement.
Whatmakesagraphsmart?
Mostgraphshaveonefeaturethatdividestheentiregraphintoseveralsmallersubgraphs.Thesesubgraphshavealargeamountofedgesthatonlyconnectverticesinthesamesubgraphandonlyhavefewedgesconnectingverticesfromothersubgraphs.Examplesforthesegraphsare:
SocialNetworks
Typicallythefeaturehereistheregion/countryuserslivein.Everyusertypicalliyhasmorecontactsinthesameregion/countrythenshehasinotherregions/countries
TransportSystems
Forthosealsothefeatureistheregion/country.Youhavemanylocaltransportionbutonlyfewaccrosscountries.
E-Commerce
Inthiscaseprobablythecategoryofproductsisagoodfeature.Oftenproductsofthesamecategoryareboughttogether.
Ifthisfeatureisknown,SmartGraphscanmakeuseifit.WhencreatingaSmartGraphyouhavetodefineasmartAttribute,whichisthenameofanattributestoredineveryvertex.Thegraphwillthanbeautomaticallyshardedinsuchawaythatallverticeswiththesamevaluearestoredonthesamephysicalmachine,alledgesconnectingverticeswithidenticalsmartAttributevaluesarestoredonthismachineaswell.Duringquerytimethequeryoptimizerandthequeryexecutorbothknowforeverydocumentexactlywhereitisstoredandcantherebyminimizenetworkoverhead.Everythingthatcanbecomputedlocallywillbecomputedlocally.
BenefitsofSmartGraphs
Becauseoftheabovedescribedguaranteedsharding,theperformanceofqueriesthatonlycoveronesubgraphhaveaperformancealmostequaltoanonlylocalcomputation.Queriesthatcovermorethanonesubgraphrequiresomenetworkoverhead.Themoresubgraphsaretouchedthemorenetworkcostwillapply.HowevertheoverallperformanceisneverworsethanthesamequeryonaGeneralGraph.
Gettingstarted
FirstofallSmartGraphscannotuseexistingcollections,whenswitchingtoSmartGraphfromanexistingdatasetyouhavetoimportthedataintoafreshSmartGraph.Thisswitchcanbeeasilyachievedwitharangodumpandarangorestore.TheonlythingyouhavetochangeinthispipelineisthatyoucreatethenewcollectionswiththeSmartGraphbeforestartingarangorestore.
Createagraph
IncomparisontoGeneralGraphwehavetoaddmoreoptionswhencreatingthegraph.ThetwooptionssmartGraphAttributeandnumberOfShardsarerequiredandcannotbemodifedlater.
arangosh>vargraph_module=require("@arangodb/smart-graph");
arangosh>vargraph=graph_module._create("myGraph",[],[],{smartGraphAttribute:"region",numberOfShards:9});
arangosh>graph;
[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:[]]
Addsomevertexcollections
SmartGraphs
156
ThisisagainidenticaltoGeneralGraph.Themodulewillsetupcorrectshardingforallthesecollections.Note:Thecollectionshavetobenew.
arangosh>graph._addVertexCollection("shop");
arangosh>graph._addVertexCollection("customer");
arangosh>graph._addVertexCollection("pet");
arangosh>graph;
[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:["shop","customer","pet"]]
DefinerelationsontheGraph
arangosh>varrel=graph_module._relation("isCustomer",["shop"],["customer"]);
arangosh>graph._extendEdgeDefinitions(rel);
arangosh>graph;
[SmartGraphmyGraphEdgeDefinitions:["isCustomer:[shop]->[customer]"]VertexCollections:["pet"]]
SmartGraphs
157
SmartGraphManagementThischapterdescribestheJavaScriptinterfaceforcreatingandmodifyingSmartGraphs.AtfirstyouhavetonotethateverySmartGraphisaspecializedversionofaGeneralGraph,whichmeansalloftheGeneralGraphfunctionalityisavailableonaSmartGraphaswell.Themajordifferenceofbothmodulesishandlingoftheunderlyingcollections,theGeneralGraphdoesnotenforceormaintainanyshardingofthecollectionsandcanthereforcombinearbitrarysetsofexistingcollections.SmartGraphsenforceandrelyonaspecialshardingoftheunderlyingcollectionsandhencecanonlyworkwithcollectionsthatarecreatedthroughtheSmartGraphitself.ThisalsomeansthatSmartGraphscannotbeoverlapping,acollectioncaneitherbeshardedforoneSmartGraphorfortheother.IfyouneedtomakesurethatallqueriescanbeexecutedwithSmartGraphperformance,justcreateonelargeSmartGraphcoveringeverythingandqueryitstatingthesubsetofedgecollectionsexplicitly.TogenerallyunderstandtheconceptofthismodulepleasereadthechapteraboutGeneralGraphManagementfirst.Inthefollowingwewillonlydescribetheoverloadedfunctionality.Everythingelseworksidenticalinbothmodules.
Createagraph
AlsoSmartGraphsrequireedgerelationstobecreated,theformatoftherelationsisidentical.TheonlydifferenceisthatallcollectionsusedwithintherelationstocreateanewSmartGraphcannotexistyet.TheyhavetobecreatedbytheGraphinordertoenforcethecorrectsharding.
Createagraph
graph_module._create(graphName,edgeDefinitions,orphanCollections,smartOptions)
ThecreationofagraphrequiresthenameandsomeSmartGraphoptions.DuetotheAPIedgeDefinitionsandorphanCollectionshavetobegiven,butbothcanbeemptyarraysandcanbecreatedlater.TheedgeDefinitionscanbecreatedusingtheconveniencemethod_relationknownfromthegeneral-graphmodule,whichisalsoavailablehere.orphanCollectionsagainisjustalistofadditionalvertexcollectionswhicharenotyetconnectedviaedgesbutshouldfollowthesameshardingtobeconnectedlateron.Allcollectionsusedwithinthecreationprocessarenewlycreated.Theprocesswillfailifoneofthemalreadyexists.Allnewlycreatedcollectionswillimmediatelybedroppedagaininthefailedcase.
Parameters
graphName(required)UniqueidentifierofthegraphedgeDefinitions(required)Listofrelationdefinitionobjects,maybeemptyorphanCollections(required)Listofadditionalvertexcollectionnames,maybeemptysmartOptions(required)AJSONobjecthavingthefollowingkeys:
numberOfShards(required)Thenumberofshardsthatwillbecreatedforeachcollection.Tomaintainthecorrectshardingallcollectionsneedanidenticalnumberofshards.Thiscannotbemodifiedaftercreationofthegraph.smartGraphAttribute(required)Theattributethatwillbeusedforsharding.Allverticesarerequiredtohavethisattributesetandithastobeastring.Edgesderivetheattributefromtheirconnectedvertices.
Examples
Createanemptygraph,edgedefinitionscanbeaddedatruntime:
arangosh>vargraph_module=require("@arangodb/smart-graph");
arangosh>vargraph=graph_module._create("myGraph",[],[],{smartGraphAttribute:"region",numberOfShards:9});
[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:[]]
Createagraphusinganedgecollectionedgesandasinglevertexcollectionvertices
arangosh>vargraph_module=require("@arangodb/smart-graph");
arangosh>varedgeDefinitions=[graph_module._relation("edges","vertices","vertices")];
arangosh>vargraph=graph_module._create("myGraph",edgeDefinitions,[],{smartGraphAttribute:"region",numberOfShards:9});
[SmartGraphmyGraphEdgeDefinitions:["edges:[vertices]->[vertices]"]VertexCollections:[]]
Createagraphwithedgedefinitionsandorphancollections:
arangosh>vargraph_module=require("@arangodb/smart-graph");
arangosh>varedgeDefinitions=[graph_module._relation("myRelation",["male","female"],["male","female"])];
SmartGraphManagement
158
arangosh>vargraph=graph_module._create("myGraph",edgeDefinitions,["sessions"],{smartGraphAttribute:"region",numberOfSh
ards:9});
[GraphmyGraphEdgeDefinitions:[
"myRelation:[female,male]->[female,male]"
]VertexCollections:[
"sessions"
]]
ModifyagraphdefinitionduringruntimeAfteryouhavecreatedaSmartGraphitsdefinitionisalsonotimmutable.Youcanstilladdorremoverelations.ThisisagainidenticaltoGeneralGraphs.Howeverthereisoneimportantdifference:Youcanonlyaddcollectionsthateitherdonotexist,orthathavebeencreatedbythisgraphearlier.Thelatercanbeachievedifyouforexampleremoveanorphancollectionfromthisgraph,withoutdroppingthecollectionitself.Thanaftersometimeyoudecidetoadditagain,itcanbeused.Thisisbecausetheenforcedshardingisstillappliedtothisvertexcollection,henceitissuitabletobeaddedagain.
Removeavertexcollection
Removeavertexcollectionfromthegraph
graph._removeVertexCollection(vertexCollectionName,dropCollection)
InmostcasesthisfunctionworksidenticallytotheGeneralGraphone.Butthereisonespecialcase:Thefirstvertexcollectionaddedtothegraph(eitherorphanorwithinarelation)definestheshardingforallcollectionswithinthegraph.Thiscollectioncanneverberemovedfromthegraph.
Parameters
vertexCollectionName(required)Nameofvertexcollection.dropCollection(optional)Iftruethecollectionwillbedroppedifitisnotusedinanyothergraph.Default:false.
Examples
Thefollowingexampleshowsthatyoucannotdroptheinitialcollection.Youhavetodropthecompletegraph.Ifyoujustwanttogetridofthedatatruncateit.
arangosh>vargraph_module=require("@arangodb/smart-graph")
arangosh>varrelation=graph_module._relation("edges","vertices","vertices");
arangosh>vargraph=graph_module._create("myGraph",[relation],["other"],{smartGraphAttribute:"region",numberOfShards:
9});
arangosh>graph._orphanCollections();
[
"other"
]
arangosh>graph._deleteEdgeDefinition("edges");
arangosh>graph._orphanCollections();
[
"vertices",
"other"
]
arangosh>graph._removeVertexCollection("other");
arangosh>graph._orphanCollections();
[
"vertices"
]
arangosh>graph._removeVertexCollection("vertices");
ArangoError4002:cannotdropthissmartcollection
SmartGraphManagement
159
TraversalsArangoDBprovidesseveralwaystoquerygraphdata.Verysimpleoperationscanbecomposedwiththelow-leveledgemethodsedges,inEdges,andoutEdgesforedgecollections.Theseworkonnamedandanonymousgraphs.Formorecomplexoperations,ArangoDBprovidespredefinedtraversalobjects.
AlsoTraversalshavebeenaddedtoAQL.PleasereadthechapteraboutAQLtraversersalsbeforeyoucontinuereadinghere.MostofthetraversalcasesarecoveredbyAQLandwillbeexecutedinanoptimizedway.OnlyifthelogicforyouristoocomplextobedefinedusingAQLfiltersyoucanusethetraversalobjectdefinedherewhichgivesyoucompleteprogrammaticaccesstothedata.
Foranyofthefollowingexamples,we'llbeusingtheexamplecollectionsvande,populatedwithcontinents,countriesandcapitalsdatalistedbelow(seeExampleData).
StartingfromScratchArangoDBprovidestheedges,inEdges,andoutEdgesmethodsforedgecollections.Thesemethodscanbeusedtoquicklydetermineifavertexisconnectedtoothervertices,andwhich.ThisfunctionalitycanbeexploitedtowriteverysimplegraphqueriesinJavaScript.
Forexample,todeterminewhichedgesarelinkedtotheworldvertex,wecanuseinEdges:
db.e.inEdges('v/world').forEach(function(edge){
require("@arangodb").print(edge._from,"->",edge.type,"->",edge._to);
});
inEdgeswillgiveusallingoingedgesforthespecifiedvertexv/world.TheresultisaJavaScriptarraythatwecaniterateoverandprinttheresults:
v/continent-africa->is-in->v/world
v/continent-south-america->is-in->v/world
v/continent-asia->is-in->v/world
v/continent-australia->is-in->v/world
v/continent-europe->is-in->v/world
v/continent-north-america->is-in->v/world
Note:edges,inEdges,andoutEdgesreturnanarrayofedges.Ifwewanttoretrievethelinkedvertices,wecanuseeachedges'_fromand_toattributesasfollows:
db.e.inEdges('v/world').forEach(function(edge){
require("@arangodb").print(db._document(edge._from).name,"->",edge.type,"->",db._document(edge._to).name);
});
Weareusingthedocumentmethodfromthedbobjecttoretrievetheconnectedverticesnow.
Whilethismaybesufficientforone-levelgraphoperations,writingatraversalbyhandmaybecometoocomplexformulti-leveltraversals.
Traversals
160
GettingstartedTouseatraversalobject,wefirstneedtorequirethetraversalmodule:
vartraversal=require("@arangodb/graph/traversal");
varexamples=require("@arangodb/graph-examples/example-graph.js");
examples.loadGraph("worldCountry");
Wethenneedtosetupaconfigurationforthetraversalanddetermineatwhichvertextostartthetraversal:
varconfig={
datasource:traversal.generalGraphDatasourceFactory("worldCountry"),
strategy:"depthfirst",
order:"preorder",
filter:traversal.visitAllFilter,
expander:traversal.inboundExpander,
maxDepth:1
};
varstartVertex=db._document("v/world");
Note:ThestartVertexneedstobeadocument,notonlyadocumentid.
Wecanthencreateatraverserandstartthetraversalbycallingitstraversemethod.Notethattraverseneedsaresultobject,whichitcanmodifyinplace:
varresult={
visited:{
vertices:[],
paths:[]
}
};
vartraverser=newtraversal.Traverser(config);
traverser.traverse(result,startVertex);
Finally,wecanprintthecontentsoftheresultsobject,limitedtothevisitedvertices.Wewillonlyprintthenameandtypeofeachvisitedvertexforbrevity:
require("@arangodb").print(result.visited.vertices.map(function(vertex){
returnvertex.name+"("+vertex.type+")";
}));
Thefullscript,whichincludesallstepscarriedoutsofaristhus:
vartraversal=require("@arangodb/graph/traversal");
varconfig={
datasource:traversal.generalGraphDatasourceFactory("worldCountry"),
strategy:"depthfirst",
order:"preorder",
filter:traversal.visitAllFilter,
expander:traversal.inboundExpander,
maxDepth:1
};
varstartVertex=db._document("v/world");
varresult={
visited:{
vertices:[],
paths:[]
}
};
vartraverser=newtraversal.Traverser(config);
traverser.traverse(result,startVertex);
UsingTraversalObjects
161
require("@arangodb").print(result.visited.vertices.map(function(vertex){
returnvertex.name+"("+vertex.type+")";
}));
Theresultisanarrayofverticesthatwerevisitedduringthetraversal,startingatthestartvertex(i.e.v/worldinourexample):
[
"World(root)",
"Africa(continent)",
"Asia(continent)",
"Australia(continent)",
"Europe(continent)",
"NorthAmerica(continent)",
"SouthAmerica(continent)"
]
Note:Theresultislimitedtoverticesdirectlyconnectedtothestartvertex.WeachievedthisbysettingthemaxDepthattributeto1.Notsettingitwouldreturnthefullarrayofvertices.
TraversalDirection
Fortheexamplescontainedinthismanual,we'llbestartingthetraversalsatvertexv/world.Verticesinourgraphareconnectedlikethis:
v/world<-is-in<-continent(Africa)<-is-in<-country(Algeria)<-is-in<-capital(Algiers)
Togetanymeaningfulresults,wemusttraversethegraphininboundorder.Thismeans,we'llbefollowingallincomingedgesoftoavertex.Inthetraversalconfiguration,wehavespecifiedthisviatheexpanderattribute:
varconfig={
...
expander:traversal.inboundExpander
};
Forothergraphs,wemightwanttotraverseviatheoutgoingedges.Forthis,wecanusetheoutboundExpander.ThereisalsoananyExpander,whichwillfollowbothoutgoingandincomingedges.Thisshouldbeusedwithcareandthetraversalshouldalwaysbelimitedtoamaximumnumberofiterations(e.g.usingthemaxIterationsattribute)inordertoterminateatsomepoint.
Toinvokethedefaultoutboundexpanderforagraph,simplyusethepredefinedfunction:
varconfig={
...
expander:traversal.outboundExpander
};
Pleasenotetheoutboundexpanderwillnotproduceanyoutputfortheexamplesifwestillstartthetraversalatthev/worldvertex.
Still,wecanusetheoutboundexpanderifwestartsomewhereelseinthegraph,e.g.
vartraversal=require("@arangodb/graph/traversal");
varconfig={
datasource:traversal.generalGraphDatasourceFactory("world_graph"),
strategy:"depthfirst",
order:"preorder",
filter:traversal.visitAllFilter,
expander:traversal.outboundExpander
};
varstartVertex=db._document("v/capital-algiers");
varresult={
visited:{
vertices:[],
paths:[]
}
};
UsingTraversalObjects
162
vartraverser=newtraversal.Traverser(config);
traverser.traverse(result,startVertex);
require("@arangodb").print(result.visited.vertices.map(function(vertex){
returnvertex.name+"("+vertex.type+")";
}));
Theresultis:
[
"Algiers(capital)",
"Algeria(country)",
"Africa(continent)",
"World(root)"
]
whichconfirmsthatnowwe'regoingoutbound.
TraversalStrategy
Depth-firsttraversals
Thevisitationorderofverticesisdeterminedbythestrategyandorderattributessetintheconfiguration.Wechosedepthfirstandpreorder,meaningthetraverserwillvisiteachvertexbeforehandlingconnectededges(pre-order),anddescendintoanyconnectededgesbeforeprocessingotherverticesonthesamelevel(depth-first).
Let'sremovethemaxDepthattributenow.We'llnowbegettingallvertices(directlyandindirectlyconnectedtothestartvertex):
varconfig={
datasource:traversal.generalGraphDatasourceFactory("world_graph"),
strategy:"depthfirst",
order:"preorder",
filter:traversal.visitAllFilter,
expander:traversal.inboundExpander
};
varresult={
visited:{
vertices:[],
paths:[]
}
};
vartraverser=newtraversal.Traverser(config);
traverser.traverse(result,startVertex);
require("@arangodb").print(result.visited.vertices.map(function(vertex){
returnvertex.name+"("+vertex.type+")";
}));
Theresultwillbealongerarray,assembledindepth-first,pre-orderorder.Foreachcontinentfound,thetraverserwilldescendintolinkedcountries,andthenintothelinkedcapital:
[
"World(root)",
"Africa(continent)",
"Algeria(country)",
"Algiers(capital)",
"Angola(country)",
"Luanda(capital)",
"Botswana(country)",
"Gaborone(capital)",
"BurkinaFaso(country)",
"Ouagadougou(capital)",
...
]
UsingTraversalObjects
163
Let'sswitchtheorderattributefrompreordertopostorder.Thiswillmakethetraverservisitverticesafterallconnectedverticeswerevisited(i.e.mostdistantverticeswillbeemittedfirst):
[
"Algiers(capital)",
"Algeria(country)",
"Luanda(capital)",
"Angola(country)",
"Gaborone(capital)",
"Botswana(country)",
"Ouagadougou(capital)",
"BurkinaFaso(country)",
"Bujumbura(capital)",
"Burundi(country)",
"Yaounde(capital)",
"Cameroon(country)",
"N'Djamena(capital)",
"Chad(country)",
"Yamoussoukro(capital)",
"Coted'Ivoire(country)",
"Cairo(capital)",
"Egypt(country)",
"Asmara(capital)",
"Eritrea(country)",
"Africa(continent)",
...
]
Breadth-firsttraversals
Ifwegobacktopreorder,butchangethestrategytobreadth-firstandre-runthetraversal,we'llseethatthereturnorderchanges,anditemsonthesamelevelwillbereturnedadjacently:
[
"World(root)",
"Africa(continent)",
"Asia(continent)",
"Australia(continent)",
"Europe(continent)",
"NorthAmerica(continent)",
"SouthAmerica(continent)",
"BurkinaFaso(country)",
"Burundi(country)",
"Cameroon(country)",
"Chad(country)",
"Algeria(country)",
"Angola(country)",
...
]
Note:Theorderofitemsreturnedforthesamelevelisundefined.Thisisbecausethereisnonaturalorderofedgesforavertexwithmultipleconnectededges.Toexplicitlysettheorderforedgesonthesamelevel,youcanspecifyanedgecomparatorfunctionwiththesortattribute:
varconfig={
...
sort:function(l,r){returnl._key<r._key?1:-1;}
...
};
Theargumentslandrareedgedocuments.Thiswilltraverseedgesofthesamevertexinbackward_keyorder:
[
"World(root)",
"SouthAmerica(continent)",
"NorthAmerica(continent)",
"Europe(continent)",
"Australia(continent)",
"Asia(continent)",
UsingTraversalObjects
164
"Africa(continent)",
"Ecuador(country)",
"Colombia(country)",
"Chile(country)",
"Brazil(country)",
"Bolivia(country)",
"Argentina(country)",
...
]
Note:Thisattributeonlyworksfortheusualexpanderstraversal.inboundExpander,traversal.outboundExpander,traversal.anyExpanderandtheircorresponding"WithLabels"variants.Ifyouareusingcustomexpandersyouhavetoorganizethesortingwithinthespecifiedexpander.
WritingCustomVisitors
Sofarwehaveusedmuchofthetraverser'sdefaultfunctions.Thetraverserisveryconfigurableandmanyofthedefaultfunctionscanbeoverriddenwithcustomfunctionality.
Forexample,wehavebeenusingthedefaultvisitorfunction(whichisalwaysusediftheconfigurationdoesnotcontainthevisitorattribute).Thedefaultvisitorfunctioniscalledforeachvertexinatraversal,andwillpushitintotheresult.Thisisthereasonwhytheresultvariablelookeddifferentafterthetraversal,andneededtobeinitializedbeforethetraversalwasstarted.
Notethatthedefaultvisitor(namedtrackingVisitor)willaddeveryvisitedvertexintotheresult,includingthefullpathsfromthestartvertex.Thisisusefulforlearninganddebuggingpurposes,butshouldbeavoidedinproductionbecauseitmightproduce(andcopy)hugeamountsofdata.Instead,onlythosedatashouldbecopiedintotheresultthatareactuallynecessary.
Thetraversercomeswiththefollowingpredefinedvisitors:
trackingVisitor:thisisthedefaultvisitor.Itwillcopyalldataofeachvisitedvertexplusthefullpathinformationintotheresult.Thiscanbeslowiftheresultsetishugeorverticescontainalotofdata.countingVisitor:thisisaverylightweightvisitor:allitdoesisincreaseacounterintheresultforeachvertexvisited.Vertexdataandpathswillnotbecopiedintotheresult.doNothingVisitor:ifnoactionshallbecarriedoutwhenavertexisvisited,thisvisitorcanbeemployed.Itwillnotdoanythingandwillthusbefast.Itcanbeusedforperformancecomparisonswithothervisitors.
Wecanalsowriteourownvisitorfunctionifwewantto.Thegeneralfunctionsignatureforvisitorfunctionsisasfollows:
varconfig={
...
visitor:function(config,result,vertex,path,connected){...}
};
Note:theconnectedparametervaluewillonlybesetifthetraversalorderissettopreorder-expander.Otherwise,thisparameterwon'tbesetbythetraverser.
Visitorfunctionsarenotexpectedtoreturnanyvalues.Instead,theycanmodifytheresultvariable(e.g.bypushingthecurrentvertexintoit),ordoanythingelse.Forexample,wecancreateasimplevisitorfunctionthatonlyprintsinformationaboutthecurrentvertexaswetraverse:
varconfig={
datasource:traversal.generalGraphDatasourceFactory("world_graph"),
strategy:"depthfirst",
order:"preorder",
filter:traversal.visitAllFilter,
expander:traversal.inboundExpander,
visitor:function(config,result,vertex,path){
require("@arangodb").print("visitingvertex",vertex.name);
}
};
vartraverser=newtraversal.Traverser(config);
traverser.traverse(undefined,startVertex);
Towriteavisitorthatincrementsacountereachtimeavertexisvisited,wecouldwritethefollowingcustomvisitor:
UsingTraversalObjects
165
config.visitor=function(config,result,vertex,path,connected){
if(!result){
result={};
}
if(!result.hasOwnProperty('count')){
result.count=0;
}
++result.count;
}
Notethatsuchvisitorisalreadypredefined(it'sthecountingVisitordescribedabove).Itcanbeusedasfollows:
config.visitor=traversal.countingVisitor;
Anotherexampleofavisitorisonethatcollectsthe_idvaluesofallverticesvisited:
config.visitor=function(config,result,vertex,path,connected){
if(!result){
result={};
}
if(!result.hasOwnProperty("visited")){
result.visited={vertices:[]};
}
result.visited.vertices.push(vertex._id);
}
Whenthetraversalorderissettopreorder-expander,thetraverserwillpassafifthparametervalueintothevisitorfunction.Thisparametercontainstheconnectededgesofthevisitedvertexasanarray.Thiscanbehandybecauseinthiscasethevisitorwillgetallinformationaboutthevertexandtheconnectededgestogether.
Forexample,thefollowingvisitorcanbeusedtoprintonlyleafnodes(thatdonothaveanyfurtherconnectededges):
config.visitor=function(config,result,vertex,path,connected){
if(connected&&connected.length===0){
require("@arangodb").print("foundaleaf-node:",vertex);
}
}
Notethatforthisvisitortowork,thetraversalorderattributeneedstobesettothevaluepreorder-expander.
FilteringVerticesandEdges
FilteringVertices
Sofarwehaveprintedorreturnedallverticesthatwerevisitedduringthetraversal.Thisisnotalwaysrequired.Iftheresultshallberestricttojustspecificvertices,wecanuseafilterfunctionforvertices.Itcanbedefinedbysettingthefilterattributeofatraversalconfiguration,e.g.:
varconfig={
filter:function(config,vertex,path){
if(vertex.type!=='capital'){
return'exclude';
}
}
}
Theabovefilterfunctionwillexcludeallverticesthatdonothaveatypevalueofcapital.Thefilterfunctionwillbecalledforeachvertexfoundduringthetraversal.Itwillreceivethetraversalconfiguration,thecurrentvertex,andthefullpathfromthetraversalstartvertextothecurrentvertex.Thepathconsistsofanarrayofedges,andanarrayofvertices.Wecouldalsofiltereverythingbutcapitalsby
UsingTraversalObjects
166
checkingthelengthofthepathfromthestartvertextothecurrentvertex.Capitalswillhaveadistanceof3fromthev/worldstartvertex(capital→is-in→country→is-in→continent→is-in→world):
varconfig={
...
filter:function(config,vertex,path){
if(path.edges.length<3){
return'exclude';
}
}
}
Note:Ifafilterfunctionreturnsnothing(orundefined),thecurrentvertexwillbeincluded,andallconnectededgeswillbefollowed.Ifafilterfunctionreturnsexcludethecurrentvertexwillbeexcludedfromtheresult,andallstillallconnectededgeswillbefollowed.Ifafilterfunctionreturnsprune,thecurrentvertexwillbeincluded,butnoconnectededgeswillbefollowed.
Forexample,thefollowingfilterfunctionwillnotdescendintoconnectededgesofcontinents,limitingthedepthofthetraversal.Still,continentverticeswillbeincludedintheresult:
varconfig={
...
filter:function(config,vertex,path){
if(vertex.type==='continent'){
return'prune';
}
}
}
Itisalsopossibletocombineexcludeandprunebyreturninganarraywithbothvalues:
return['exclude','prune'];
FilteringEdges
Itispossibletoexcludecertainedgesfromthetraversal.Tofilteronedges,afilterfunctioncanbedefinedviatheexpandFilterattribute.TheexpandFilterisafunctionwhichiscalledforeachedgeduringatraversal.
Itwillreceivethecurrentedge(edgevariable)andthevertexwhichtheedgeconnectsto(inthedirectionofthetraversal).Italsoreceivesthecurrentpathfromthestartvertexuptothecurrentvertex(excludingthecurrentedgeandthevertextheedgepointsto).
Ifthefunctionreturnstrue,theedgewillbefollowed.Ifthefunctionreturnsfalse,theedgewillnotbefollowed.Hereisaverysimplecustomedgefilterfunctionimplementation,whichsimplyincludesedgesifthe(edges)pathlengthislessthan1,andwillexcludeanyotheredges.Thiswilleffectivelyterminatethetraversalafterthefirstlevelofedges:
varconfig={
...
expandFilter:function(config,vertex,edge,path){
return(path.edges.length<1);
}
};
WritingCustomExpanders
Theedgesconnectedtoavertexaredeterminedbytheexpander.Sofarwehaveusedadefaultexpander(thedefaultinboundexpandertobeprecise).Thedefaultinboundexpandersimplyenumeratesallconnectedingoingedgesforavertex,basedontheedgecollectionspecifiedinthetraversalconfiguration.
Thereisalsoadefaultoutboundexpander,whichwillenumerateallconnectedoutgoingedges.Finally,thereisananyexpander,whichwillfollowbothingoingandoutgoingedges.
Ifconnectededgesmustbedeterminedinsomedifferentfashionforwhateverreason,acustomexpandercanbewrittenandregisteredbysettingtheexpanderattributeoftheconfiguration.Theexpanderfunctionsignatureisasfollows:
UsingTraversalObjects
167
varconfig={
...
expander:function(config,vertex,path){...}
}
Itistheexpander'sresponsibilitytoreturnalledgesandverticesdirectlyconnectedtothecurrentvertex(whichispassedviathevertexvariable).Thefullpathfromthestartvertexuptothecurrentvertexisalsosuppliedviathepathvariable.Anexpanderisexpectedtoreturnanarrayofobjects,whichneedtohaveanedgeandavertexattributeeach.
Note:Ifyouwanttorelyonaparticularorderinwhichtheedgesaretraversed,youhavetosorttheedgesreturnedbyyourexpanderwithinthecodeoftheexpander.Thefunctionstogetoutbound,inboundoranyedgesfromavertexdonotguaranteeanyparticularorder!
Acustomimplementationofaninboundexpandercouldlooklikethis(thisisanon-deterministicexpander,whichrandomlydecideswhetherornottoincludeconnectededges):
varconfig={
...
expander:function(config,vertex,path){
varconnected=[];
vardatasource=config.datasource;
datasource.getInEdges(vertex._id).forEach(function(edge){
if(Math.random()>=0.5){
connected.push({edge:edge,vertex:(edge._from)});
}
});
returnconnected;
}
};
Acustomexpandercanalsobeusedasanedgefilterbecauseithasfullcontroloverwhichedgeswillbereturned.
Followingaretwoexamplesofcustomexpandersthatpickedgesbasedonattributesoftheedgesandtheconnectedvertices.
Findingtheconnectededges/verticesbasedonanattributewhenintheconnectedvertices.Thegoalistofollowtheedgethatleadstothevertexwiththehighestvalueinthewhenattribute:
varconfig={
...
expander:function(config,vertex,path){
vardatasource=config.datasource;
//determinealloutgoingedges
varoutEdges=datasource.getOutEdges(vertex);
if(outEdges.length===0){
return[];
}
vardata=[];
outEdges.forEach(function(edge){
data.push({edge:edge,vertex:datasource.getInVertex(edge)});
});
//sortoutgoingverticesaccordingto"when"attributevalue
data.sort(function(l,r){
if(l.vertex.when===r.vertex.when){
return0;
}
return(l.vertex.when<r.vertex.when?1:-1);
});
//pickfirstvertexfound(withhighest"when"attributevalue)
return[data[0]];
}
...
};
Findingtheconnectededges/verticesbasedonanattributewhenintheedgeitself.Thegoalistopicktheoneedge(outofpotentiallymany)thathasthehighestwhenattributevalue:
UsingTraversalObjects
168
varconfig={
...
expander:function(config,vertex,path){
vardatasource=config.datasource;
//determinealloutgoingedges
varoutEdges=datasource.getOutEdges(vertex);
if(outEdges.length===0){
return[];//returnanemptyarray
}
//sortalloutgoingedgesaccordingto"when"attribute
outEdges.sort(function(l,r){
if(l.when===r.when){
return0;
}
return(l.when<r.when?-1:1);
});
//returnfirstedge(theonewithhighest"when"value)
varedge=outEdges[0];
try{
varv=datasource.getInVertex(edge);
return[{edge:edge,vertex:v}];
}
catch(e){}
return[];
}
...
};
HandlingUniqueness
Graphsmaycontaincycles.Tobeontopofwhathappenswhenatraversalencountersavertexoranedgeithasalreadyvisited,thereareconfigurationoptions.
Thedefaultconfigurationistovisiteveryvertex,regardlessofwhetheritwasalreadyvisitedinthesametraversal.However,edgeswillbydefaultonlybefollowediftheyarenotalreadypresentinthecurrentpath.
Imaginethefollowinggraphwhichcontainsacycle:
A->B->C->A
WhenthetraversalfindstheedgefromCtoA,itwillbydefaultfollowit.Thisisbecausewehavenotseenthisedgeyet.ItwillalsovisitvertexAagain.Thisisbecausebydefaultallverticeswillbevisited,regardlessofwhetheralreadyvisitedornot.
However,thetraversalwillnotagainfollowingtheoutgoingedgefromAtoB.ThisisbecausewealreadyhavetheedgefromAtoBinourcurrentpath.
Thesedefaultsettingswillpreventinfinitetraversals.
Toadjusttheuniquenessforvisitingvertices,therearethefollowingoptionsforuniqueness.vertices:
"none":alwaysvisitavertices,regardlessofwhetheritwasalreadyvisitedornot"global":visitavertexonlyifitwasnotvisitedinthetraversal"path":visitavertexifitisnotincludedinthecurrentpath
Toadjusttheuniquenessforfollowingedges,therearethefollowingoptionsforuniqueness.edges:
"none":alwaysfollowanedge,regardlessofwhetheritwasfollowedbefore"global":followanedgeonlyifitwasn'tfollowedinthetraversal"path":followanedgeifitisnotincludedinthecurrentpath
Notethatuniquenesscheckingwillhavesomeeffectonbothruntimeandmemoryusage.Forexample,whenuniquenesschecksaresetto"global" ,arraysofvisitedverticesandedgesmustbekeptinmemorywhilethetraversalisexecuted.Globaluniquenessshouldthusonlybeusedwhenatraversalisexpectedtovisitfewnodes.
UsingTraversalObjects
169
Intermsofruntime,turningoffuniquenesschecks(bysettingbothoptionsto"none")isthebestchoice,butitisonlysafeforgraphsthatdonotcontaincycles.Whenuniquenesschecksaredeactivatedinagraphwithcycles,thetraversalmightnotabortinasensibleamountoftime.
Optimizations
Thereareafewoptionsformakingatraversalrunfaster.
Thebestoptionistomaketheamountofvisitedverticesandfollowededgesassmallaspossible.Thiscanbeachievedbywritingcustomfilterandexpanderfunctions.Suchfunctionsshouldonlyincludeverticesofinterest,andonlyfollowedgesthatmightbeinteresting.
TraversaldepthcanalsobeboundedwiththeminDepthandmaxDepthoptions.
Anotherwaytospeeduptraversalsistowriteacustomvisitorfunction.Thedefaultvisitorfunction(trackingVisitor)willcopyeveryvisitedvertexintotheresult.Ifverticescontainlotsofdata,thismightbeexpensive.Itisthereforerecommendedtoonlycopysuchdataintotheresultthatisactuallyneeded.Thedefaultvisitorfunctionwillalsocopythefullpathtothevisiteddocumentintotheresult.Thisisevenmoreexpensiveandshouldbeavoidedifpossible.
Ifthegoalofatraversalistoonlycountthenumberofvisitedvertices,theprefabcountingVisitorwillbemuchmoreefficientthanthedefaultvisitor.
Forgraphsthatareknowntonotcontainanycycles,uniquenesschecksshouldbeturnedoff.Thiscanachievedviatheuniquenessconfigurationoptions.Notethatuniquenesschecksshouldnotbeturnedoffforgraphsthatareknowncontaincyclesorifthereisnoinformationaboutthegraph'sstructure.
Bydefault,atraversalwillonlyprocessalimitednumberofvertices.Thisisprotecttheuserfromunintentionallyrunanever-endingtraversalonagraphwithcyclicdata.HowmanyverticeswillbeprocessedatmostisdeterminedbythemaxIterationsconfigurationoption.IfatraversalhitsthecapspecifiedbymaxIterations,itwillabortandthrowatoomanyiterationsexception.Ifthiserrorisencountered,themaxIterationsvalueshouldbeincreasedifitismadesurethattheothertraversalconfigurationparametersaresaneandthetraversalwillabortnaturallyatsomepoint.
Finally,thebuildVerticesconfigurationoptioncanbesettofalsetoavoidlookingupandfullyconstructingvertexdata.Ifallthat'sneededfromverticesarethe_idor_keyattributes,thebuildverticesoptioncanbesettofalse.Ifvisitor,filterorexpandFilterfunctionsneedtoaccessothervertexattributes,theoptionshouldnotbechanged.
ConfigurationOverview
Thissectionsummarizestheconfigurationattributesforthetraversalobject.Theconfigurationcanconsistofthefollowingattributes:
visitor:visitorfunctionforvertices.Itwillbecalledforallnon-excludedvertices.Thegeneralvisitorfunctionsignatureisfunction(config,result,vertex,path).Ifthetraversalorderispreorder-expander,theconnectingedgesofthevisitedvertexwillbepassedasthefifthparameter,extendingthefunctionsignatureto:function(config,result,vertex,path,edges).
Visitorfunctionsarenotexpectedtoreturnvalues,buttheymaymodifytheresultvariableasneeded(e.g.bypushingvertexdataintotheresult).
expander:expanderfunctionthatisresponsibleforreturningedgesandverticesdirectlyconnectedtoavertex.Thefunctionsignatureisfunction(config,vertex,path).Theexpanderfunctionisrequiredtoreturnanarrayofconnectionobjects,consistingofanedgeandvertexattributeeach.Iftherearenoconnectingedges,theexpanderisexpectedtoreturnanemptyarray.filter:vertexfilterfunction.Thefunctionsignatureisfunction(config,vertex,path).Itmayreturnoneofthefollowingvalues:
undefined:vertexwillbeincludedintheresultandconnectededgeswillbetraversed"exclude":vertexwillnotbeincludedintheresultandconnectededgeswillbetraversed"prune":vertexwillbeincludedintheresultbutconnectededgeswillnotbetraversed["prune" ,"exclude" ]:vertexwillnotbeincludedintheresultandconnectededgeswillnotbereturned
expandFilter:filterfunctionappliedoneachedge/vertexcombinationdeterminedbytheexpander.Thefunctionsignatureisfunction(config,vertex,edge,path).Thefunctionshouldreturntrueiftheedge/vertexcombinationshouldbeprocessed,andfalseifitshouldbeignored.sort:afilterfunctiontodeterminetheorderinwhichconnectededgesareprocessed.Thefunctionsignatureisfunction(l,r).Thefunctionisrequiredtoreturnoneofthefollowingvalues:
-1iflshouldhaveasortvaluelessthanr
UsingTraversalObjects
170
1iflshouldhaveahighersortvaluethanr0iflandrhavethesamesortvalue
strategy:determinesthevisitationstrategy.Possiblevaluesaredepthfirstandbreadthfirst.order:determinesthevisitationorder.Possiblevaluesarepreorder,postorder,andpreorder-expander.preorder-expanderisthesameaspreorder,exceptthatthesignatureofthevisitorfunctionwillchangeasdescribedabove.itemOrder:determinestheorderinwhichconnectionsreturnedbytheexpanderwillbeprocessed.Possiblevaluesareforwardandbackward.maxDepth:ifsettoavaluegreaterthan0,thiswilllimitthetraversaltothismaximumdepth.minDepth:ifsettoavaluegreaterthan0,allverticesfoundonalevelbelowtheminDepthlevelwillnotbeincludedintheresult.maxIterations:themaximumnumberofiterationsthatthetraversalisallowedtoperform.Itissensibletosetthisnumbersounboundedtraversalswillterminateatsomepoint.uniqueness:anobjectthatdefineshowrepeatedvisitationsofverticesshouldbehandled.Theuniquenessobjectcanhaveasub-attributevertices,andasub-attributeedges.Eachsub-attributecanhaveoneofthefollowingvalues:
"none":nouniquenessconstraints"path":elementisexcludedifitisalreadycontainedinthecurrentpath.Thissettingmaybesensibleforgraphsthatcontaincycles(e.g.A→B→C→A)."global":elementisexcludedifitwasalreadyfound/visitedatanypointduringthetraversal.
buildVertices:thisattributecontrolswhetherverticesencounteredduringthetraversalwillbelookedupinthedatabaseandwillbemadeavailabletovisitor,filter,andexpandFilterfunctions.Bydefault,verticeswillbelookedupandmadeavailable.However,therearesomespecialusecaseswhenfullyconstructingvertexobjectsisnotnecessaryandcanbeavoided.Forexample,ifatraversalismeanttoonlycountthenumberofvisitedverticesbutdonotreadanydatafromvertices,thisoptionmightbesettotrue.
UsingTraversalObjects
171
ExampleDataThefollowingexamplesalluseavertexcollectionvandanedgecollectione.Thevertexcollectionvcontainscontinents,countries,andcapitals.Theedgecollectionecontainsconnectionsbetweencontinentsandcountries,andbetweencountriesandcapitals.
Tosetupthecollectionsandpopulatethemwithinitialdata,thefollowingscriptwasused:
db._create("v");
db._createEdgeCollection("e");
//vertices:rootnode
db.v.save({_key:"world",name:"World",type:"root"});
//vertices:continents
db.v.save({_key:"continent-africa",name:"Africa",type:"continent"});
db.v.save({_key:"continent-asia",name:"Asia",type:"continent"});
db.v.save({_key:"continent-australia",name:"Australia",type:"continent"});
db.v.save({_key:"continent-europe",name:"Europe",type:"continent"});
db.v.save({_key:"continent-north-america",name:"NorthAmerica",type:"continent"});
db.v.save({_key:"continent-south-america",name:"SouthAmerica",type:"continent"});
//vertices:countries
db.v.save({_key:"country-afghanistan",name:"Afghanistan",type:"country",code:"AFG"});
db.v.save({_key:"country-albania",name:"Albania",type:"country",code:"ALB"});
db.v.save({_key:"country-algeria",name:"Algeria",type:"country",code:"DZA"});
db.v.save({_key:"country-andorra",name:"Andorra",type:"country",code:"AND"});
db.v.save({_key:"country-angola",name:"Angola",type:"country",code:"AGO"});
db.v.save({_key:"country-antigua-and-barbuda",name:"AntiguaandBarbuda",type:"country",code:"ATG"});
db.v.save({_key:"country-argentina",name:"Argentina",type:"country",code:"ARG"});
db.v.save({_key:"country-australia",name:"Australia",type:"country",code:"AUS"});
db.v.save({_key:"country-austria",name:"Austria",type:"country",code:"AUT"});
db.v.save({_key:"country-bahamas",name:"Bahamas",type:"country",code:"BHS"});
db.v.save({_key:"country-bahrain",name:"Bahrain",type:"country",code:"BHR"});
db.v.save({_key:"country-bangladesh",name:"Bangladesh",type:"country",code:"BGD"});
db.v.save({_key:"country-barbados",name:"Barbados",type:"country",code:"BRB"});
db.v.save({_key:"country-belgium",name:"Belgium",type:"country",code:"BEL"});
db.v.save({_key:"country-bhutan",name:"Bhutan",type:"country",code:"BTN"});
db.v.save({_key:"country-bolivia",name:"Bolivia",type:"country",code:"BOL"});
db.v.save({_key:"country-bosnia-and-herzegovina",name:"BosniaandHerzegovina",type:"country",code:"BIH"});
db.v.save({_key:"country-botswana",name:"Botswana",type:"country",code:"BWA"});
db.v.save({_key:"country-brazil",name:"Brazil",type:"country",code:"BRA"});
db.v.save({_key:"country-brunei",name:"Brunei",type:"country",code:"BRN"});
db.v.save({_key:"country-bulgaria",name:"Bulgaria",type:"country",code:"BGR"});
db.v.save({_key:"country-burkina-faso",name:"BurkinaFaso",type:"country",code:"BFA"});
db.v.save({_key:"country-burundi",name:"Burundi",type:"country",code:"BDI"});
db.v.save({_key:"country-cambodia",name:"Cambodia",type:"country",code:"KHM"});
db.v.save({_key:"country-cameroon",name:"Cameroon",type:"country",code:"CMR"});
db.v.save({_key:"country-canada",name:"Canada",type:"country",code:"CAN"});
db.v.save({_key:"country-chad",name:"Chad",type:"country",code:"TCD"});
db.v.save({_key:"country-chile",name:"Chile",type:"country",code:"CHL"});
db.v.save({_key:"country-colombia",name:"Colombia",type:"country",code:"COL"});
db.v.save({_key:"country-cote-d-ivoire",name:"Coted'Ivoire",type:"country",code:"CIV"});
db.v.save({_key:"country-croatia",name:"Croatia",type:"country",code:"HRV"});
db.v.save({_key:"country-czech-republic",name:"CzechRepublic",type:"country",code:"CZE"});
db.v.save({_key:"country-denmark",name:"Denmark",type:"country",code:"DNK"});
db.v.save({_key:"country-ecuador",name:"Ecuador",type:"country",code:"ECU"});
db.v.save({_key:"country-egypt",name:"Egypt",type:"country",code:"EGY"});
db.v.save({_key:"country-eritrea",name:"Eritrea",type:"country",code:"ERI"});
db.v.save({_key:"country-finland",name:"Finland",type:"country",code:"FIN"});
db.v.save({_key:"country-france",name:"France",type:"country",code:"FRA"});
db.v.save({_key:"country-germany",name:"Germany",type:"country",code:"DEU"});
db.v.save({_key:"country-people-s-republic-of-china",name:"People'sRepublicofChina",type:"country",code:"CHN"});
//vertices:capitals
db.v.save({_key:"capital-algiers",name:"Algiers",type:"capital"});
db.v.save({_key:"capital-andorra-la-vella",name:"AndorralaVella",type:"capital"});
db.v.save({_key:"capital-asmara",name:"Asmara",type:"capital"});
db.v.save({_key:"capital-bandar-seri-begawan",name:"BandarSeriBegawan",type:"capital"});
db.v.save({_key:"capital-beijing",name:"Beijing",type:"capital"});
db.v.save({_key:"capital-berlin",name:"Berlin",type:"capital"});
db.v.save({_key:"capital-bogota",name:"Bogota",type:"capital"});
db.v.save({_key:"capital-brasilia",name:"Brasilia",type:"capital"});
ExampleData
172
db.v.save({_key:"capital-bridgetown",name:"Bridgetown",type:"capital"});
db.v.save({_key:"capital-brussels",name:"Brussels",type:"capital"});
db.v.save({_key:"capital-buenos-aires",name:"BuenosAires",type:"capital"});
db.v.save({_key:"capital-bujumbura",name:"Bujumbura",type:"capital"});
db.v.save({_key:"capital-cairo",name:"Cairo",type:"capital"});
db.v.save({_key:"capital-canberra",name:"Canberra",type:"capital"});
db.v.save({_key:"capital-copenhagen",name:"Copenhagen",type:"capital"});
db.v.save({_key:"capital-dhaka",name:"Dhaka",type:"capital"});
db.v.save({_key:"capital-gaborone",name:"Gaborone",type:"capital"});
db.v.save({_key:"capital-helsinki",name:"Helsinki",type:"capital"});
db.v.save({_key:"capital-kabul",name:"Kabul",type:"capital"});
db.v.save({_key:"capital-la-paz",name:"LaPaz",type:"capital"});
db.v.save({_key:"capital-luanda",name:"Luanda",type:"capital"});
db.v.save({_key:"capital-manama",name:"Manama",type:"capital"});
db.v.save({_key:"capital-nassau",name:"Nassau",type:"capital"});
db.v.save({_key:"capital-n-djamena",name:"N'Djamena",type:"capital"});
db.v.save({_key:"capital-ottawa",name:"Ottawa",type:"capital"});
db.v.save({_key:"capital-ouagadougou",name:"Ouagadougou",type:"capital"});
db.v.save({_key:"capital-paris",name:"Paris",type:"capital"});
db.v.save({_key:"capital-phnom-penh",name:"PhnomPenh",type:"capital"});
db.v.save({_key:"capital-prague",name:"Prague",type:"capital"});
db.v.save({_key:"capital-quito",name:"Quito",type:"capital"});
db.v.save({_key:"capital-saint-john-s",name:"SaintJohn's",type:"capital"});
db.v.save({_key:"capital-santiago",name:"Santiago",type:"capital"});
db.v.save({_key:"capital-sarajevo",name:"Sarajevo",type:"capital"});
db.v.save({_key:"capital-sofia",name:"Sofia",type:"capital"});
db.v.save({_key:"capital-thimphu",name:"Thimphu",type:"capital"});
db.v.save({_key:"capital-tirana",name:"Tirana",type:"capital"});
db.v.save({_key:"capital-vienna",name:"Vienna",type:"capital"});
db.v.save({_key:"capital-yamoussoukro",name:"Yamoussoukro",type:"capital"});
db.v.save({_key:"capital-yaounde",name:"Yaounde",type:"capital"});
db.v.save({_key:"capital-zagreb",name:"Zagreb",type:"capital"});
//edges:continent->world
db.e.save("v/continent-africa","v/world",{type:"is-in"});
db.e.save("v/continent-asia","v/world",{type:"is-in"});
db.e.save("v/continent-australia","v/world",{type:"is-in"});
db.e.save("v/continent-europe","v/world",{type:"is-in"});
db.e.save("v/continent-north-america","v/world",{type:"is-in"});
db.e.save("v/continent-south-america","v/world",{type:"is-in"});
//edges:country->continent
db.e.save("v/country-afghanistan","v/continent-asia",{type:"is-in"});
db.e.save("v/country-albania","v/continent-europe",{type:"is-in"});
db.e.save("v/country-algeria","v/continent-africa",{type:"is-in"});
db.e.save("v/country-andorra","v/continent-europe",{type:"is-in"});
db.e.save("v/country-angola","v/continent-africa",{type:"is-in"});
db.e.save("v/country-antigua-and-barbuda","v/continent-north-america",{type:"is-in"});
db.e.save("v/country-argentina","v/continent-south-america",{type:"is-in"});
db.e.save("v/country-australia","v/continent-australia",{type:"is-in"});
db.e.save("v/country-austria","v/continent-europe",{type:"is-in"});
db.e.save("v/country-bahamas","v/continent-north-america",{type:"is-in"});
db.e.save("v/country-bahrain","v/continent-asia",{type:"is-in"});
db.e.save("v/country-bangladesh","v/continent-asia",{type:"is-in"});
db.e.save("v/country-barbados","v/continent-north-america",{type:"is-in"});
db.e.save("v/country-belgium","v/continent-europe",{type:"is-in"});
db.e.save("v/country-bhutan","v/continent-asia",{type:"is-in"});
db.e.save("v/country-bolivia","v/continent-south-america",{type:"is-in"});
db.e.save("v/country-bosnia-and-herzegovina","v/continent-europe",{type:"is-in"});
db.e.save("v/country-botswana","v/continent-africa",{type:"is-in"});
db.e.save("v/country-brazil","v/continent-south-america",{type:"is-in"});
db.e.save("v/country-brunei","v/continent-asia",{type:"is-in"});
db.e.save("v/country-bulgaria","v/continent-europe",{type:"is-in"});
db.e.save("v/country-burkina-faso","v/continent-africa",{type:"is-in"});
db.e.save("v/country-burundi","v/continent-africa",{type:"is-in"});
db.e.save("v/country-cambodia","v/continent-asia",{type:"is-in"});
db.e.save("v/country-cameroon","v/continent-africa",{type:"is-in"});
db.e.save("v/country-canada","v/continent-north-america",{type:"is-in"});
db.e.save("v/country-chad","v/continent-africa",{type:"is-in"});
db.e.save("v/country-chile","v/continent-south-america",{type:"is-in"});
db.e.save("v/country-colombia","v/continent-south-america",{type:"is-in"});
db.e.save("v/country-cote-d-ivoire","v/continent-africa",{type:"is-in"});
db.e.save("v/country-croatia","v/continent-europe",{type:"is-in"});
db.e.save("v/country-czech-republic","v/continent-europe",{type:"is-in"});
db.e.save("v/country-denmark","v/continent-europe",{type:"is-in"});
db.e.save("v/country-ecuador","v/continent-south-america",{type:"is-in"});
ExampleData
173
db.e.save("v/country-egypt","v/continent-africa",{type:"is-in"});
db.e.save("v/country-eritrea","v/continent-africa",{type:"is-in"});
db.e.save("v/country-finland","v/continent-europe",{type:"is-in"});
db.e.save("v/country-france","v/continent-europe",{type:"is-in"});
db.e.save("v/country-germany","v/continent-europe",{type:"is-in"});
db.e.save("v/country-people-s-republic-of-china","v/continent-asia",{type:"is-in"});
//edges:capital->country
db.e.save("v/capital-algiers","v/country-algeria",{type:"is-in"});
db.e.save("v/capital-andorra-la-vella","v/country-andorra",{type:"is-in"});
db.e.save("v/capital-asmara","v/country-eritrea",{type:"is-in"});
db.e.save("v/capital-bandar-seri-begawan","v/country-brunei",{type:"is-in"});
db.e.save("v/capital-beijing","v/country-people-s-republic-of-china",{type:"is-in"});
db.e.save("v/capital-berlin","v/country-germany",{type:"is-in"});
db.e.save("v/capital-bogota","v/country-colombia",{type:"is-in"});
db.e.save("v/capital-brasilia","v/country-brazil",{type:"is-in"});
db.e.save("v/capital-bridgetown","v/country-barbados",{type:"is-in"});
db.e.save("v/capital-brussels","v/country-belgium",{type:"is-in"});
db.e.save("v/capital-buenos-aires","v/country-argentina",{type:"is-in"});
db.e.save("v/capital-bujumbura","v/country-burundi",{type:"is-in"});
db.e.save("v/capital-cairo","v/country-egypt",{type:"is-in"});
db.e.save("v/capital-canberra","v/country-australia",{type:"is-in"});
db.e.save("v/capital-copenhagen","v/country-denmark",{type:"is-in"});
db.e.save("v/capital-dhaka","v/country-bangladesh",{type:"is-in"});
db.e.save("v/capital-gaborone","v/country-botswana",{type:"is-in"});
db.e.save("v/capital-helsinki","v/country-finland",{type:"is-in"});
db.e.save("v/capital-kabul","v/country-afghanistan",{type:"is-in"});
db.e.save("v/capital-la-paz","v/country-bolivia",{type:"is-in"});
db.e.save("v/capital-luanda","v/country-angola",{type:"is-in"});
db.e.save("v/capital-manama","v/country-bahrain",{type:"is-in"});
db.e.save("v/capital-nassau","v/country-bahamas",{type:"is-in"});
db.e.save("v/capital-n-djamena","v/country-chad",{type:"is-in"});
db.e.save("v/capital-ottawa","v/country-canada",{type:"is-in"});
db.e.save("v/capital-ouagadougou","v/country-burkina-faso",{type:"is-in"});
db.e.save("v/capital-paris","v/country-france",{type:"is-in"});
db.e.save("v/capital-phnom-penh","v/country-cambodia",{type:"is-in"});
db.e.save("v/capital-prague","v/country-czech-republic",{type:"is-in"});
db.e.save("v/capital-quito","v/country-ecuador",{type:"is-in"});
db.e.save("v/capital-saint-john-s","v/country-antigua-and-barbuda",{type:"is-in"});
db.e.save("v/capital-santiago","v/country-chile",{type:"is-in"});
db.e.save("v/capital-sarajevo","v/country-bosnia-and-herzegovina",{type:"is-in"});
db.e.save("v/capital-sofia","v/country-bulgaria",{type:"is-in"});
db.e.save("v/capital-thimphu","v/country-bhutan",{type:"is-in"});
db.e.save("v/capital-tirana","v/country-albania",{type:"is-in"});
db.e.save("v/capital-vienna","v/country-austria",{type:"is-in"});
db.e.save("v/capital-yamoussoukro","v/country-cote-d-ivoire",{type:"is-in"});
db.e.save("v/capital-yaounde","v/country-cameroon",{type:"is-in"});
db.e.save("v/capital-zagreb","v/country-croatia",{type:"is-in"});
ExampleData
174
Edges,Identifiers,HandlesThisisanintroductiontoArangoDB'sinterfaceforedges.Edgesmaybeusedingraphs.HereweworkwithedgesfromtheJavaScriptshellarangosh.ForotherlanguagesseethecorrespondinglanguageAPI.
Agraphdatamodelalwaysconsistsofatleasttwocollections:therelationsbetweenthenodesinthegraphsarestoredinan"edgescollection",thenodesinthegrapharestoredindocumentsinregularcollections.
EdgesinArangoDBarespecialdocuments.Inadditiontothesystemattributes_key,_idand_rev,theyhavetheattributes_fromand_to,whichcontaindocumenthandles,namelythestart-pointandtheend-pointoftheedge.
Example:
the"edge"collectionstorestheinformationthatacompany'sreceptionissub-unittotheservicesunitandtheservicesunitissub-unittotheCEO.Youwouldexpressthisrelationshipwiththe_fromand_toattributesthe"normal"collectionstoresallthepropertiesaboutthereception,e.g.that20peopleareworkingthereandtheroomnumberetc_fromisthedocumenthandleofthelinkedvertex(incomingrelation)_toisthedocumenthandleofthelinkedvertex(outgoingrelation)
Edgecollectionsarespecialcollectionsthatstoreedgedocuments.Edgedocumentsareconnectiondocumentsthatreferenceotherdocuments.Thetypeofacollectionmustbespecifiedwhenacollectioniscreatedandcannotbechangedafterwards.
Tochangeedgeendpointsyouwouldneedtoremoveolddocument/edgeandinsertnewone.Otherfieldscanbeupdatedasindefaultcollection.
WorkingwithEdges
Edgesarenormaldocumentsthatalwayscontaina_fromanda_toattribute.
WorkingwithEdges
175
DistributedIterativeGraphProcessing(Pregel)Distributedgraphprocessingenablesyoutodoonlineanalyticalprocessingdirectlyongraphsstoredintoarangodb.Thisisintendedtohelpyougainanalyticalinsightsonyourdata,withouthavingtouseexternalprocessingsytems.ExamplesofalgorithmstoexecutearePageRank,VertexCentrality,VertexCloseness,ConnectedComponents,CommunityDetection.Thissystemisnotusefulfortypicalonlinequeries,whereyoujustdoworkonasmallsetofvertices.ThesekindoftasksarebettersuitedforAQL.
TheprocessingsysteminsideArangoDBisbasedon:Pregel:ASystemforLarge-ScaleGraphProcessing–Malewiczetal.(Google)2010Thisconceptenablesustoperformdistributedgraphprocessing,withouttheneedfordistributedgloballocking.
PrerequisitesIfyouarerunningasingleArangoDBinstanceinsingle-servermode,therearenorequirementsregardingthemodelingofyourdata.Allyouneedisatleastonevertexcollectionandoneedgecollection.Notethattheperformancemaybebetter,ifthenumberofyourshards/collectionsmatchesthenumberofCPUcores.
WhenyouuseArangoDBCommunityeditioninclustermode,youmightneedtomodelyourcollectionsinacertainwaytoensurecorrectresults.Formoreinformationseethenextsection.
RequirementsforCollectionsinaCluster(NonSmartGraph)Toenableiterativegraphprocessingforyourdata,youwillneedtoensurethatyourvertexandedgecollectionsareshardedinaspecificway.
ThepregelcomputingmodelrequiresalledgestobepresentontheDBServerwherethevertexdocumentidentifiedbythe_fromvalueislocated.Thismeansthevertexcollectionsneedtobeshardedby'_key'andtheedgecollectionwillneedtobeshardedafteranattributewhichalwayscontainsthe'_key'ofthevertex.
Ourimplementationcurrentlyrequireseveryedgecollectiontobeshardedaftera"vertex"attributes,additionallyyouwillneedtospecifythekeydistributeShardsLikeandanequalnumberofshardsoneverycollection.OnlyiftheserequirementsaremetcanArangoDBplacetheedgesandverticescorrectly.
Forexampleyoumightcreateyourcollectionslikethis:
//Createmainvertexcollection:
db._create("vertices",{
shardKeys:['_key'],
numberOfShards:8
});
//Optionallycreatearbitraryadditionalvertexcollections
db._create("additonal",{
distributeShardsLike:"vertices",
numberOfShards:8
});
//Create(oneormore)edge-collections:
db._createEdgeCollection("edges",{
shardKeys:['vertex'],
distributeShardsLike:"vertices",
numberOfShards:8
});
Youwillneedtoensurethatedgedocumentscontainthepropervaluesintheirshardingattribute.Foravertexdocumentwiththefollowingcontent{_key:"A",value:0}thecorrespondingedgedocumentswouldhavelooklikethis:
{_from:"vertices/A",_to:"vertices/B",vertex:"A"}
{_from:"vertices/A",_to:"vertices/C",vertex:"A"}
{_from:"vertices/A",_to:"vertices/D",vertex:"A"}
...
Pregel
176
ThiswillensurethatoutgoingedgedocumentswillbeplacedonthesameDBServerasthevertex.Withoutthecorrectplacementoftheedges,thepregelgraphprocessingsystemwillnotworkcorrectly,becauseedgeswillnotloadcorrectly.
ArangoshAPI
StartinganAlgorithmExecution
ThepregelAPIisaccessiblethroughthe@arangodb/pregelpackage.Tostartanexecutionyouneedtospecifythealgorithmnameandthevertexandedgecollections.Alternativelyyoucanspecifyanamedgraph.Additionallyyoucanspecifycustomparameterswhichvaryforeachalgorithm.ThestartmethodwillalwaysauniqueIDwhichcanbeusedtointeractwiththealgorithmandlateron.
Thebelowversionofthestartmethodcanbeusedfornamedgraphs:
varpregel=require("@arangodb/pregel");
varparams={};
varexecution=pregel.start("<algorithm>","<yourgraph>",params);
Paramsneedstobeanobject,thevalidkeysarementionedbelowinthesectionAlgorithms
Alternativelyyoumightwanttospecifythevertexandedgecollectionsdirectly.Thecall-syntaxofthestart``methodchangesinthiscase.ThesecondargumentmustbeanobjectwiththekeysvertexCollectionsandedgeCollections`.
varpregel=require("@arangodb/pregel");
varparams={};
varexecution=pregel.start("<algorithm>",{vertexCollections:["vertices"],edgeCollections:["edges"]},{});
Thelastargumentisstilltheparameterobject.Seebelowforalistofalgorithmsandparameters.
StatusofanAlgorithmExecution
Thecodereturnedbythepregel.start(...)methodcanbeusedtotrackthestatusofyouralgorithm.
varexecution=pregel.start("sssp","demograph",{source:"vertices/V"});
varstatus=pregel.status(execution);
Theresultwilltellyouthecurrentstatusofthealgorithmexecution.Itwilltellyouthecurrentstateoftheexecution,thecurrentglobalsuperstep,theruntime,theglobalaggregatorvaluesaswellasthenumberofsendandreceivedmessages.
Validvaluesforthestatefieldinclude:
"running"algorithmisstillrunning"done":Theexecutionisdone,theresultmightnotbewrittenbackintothecollectionyet."canceled":Theexecutionwaspermanentlycanceled,eitherbytheuserorbyanerror."inerror":Theexeuctionisinanerrorstate.ThiscanbecausedbyprimaryDBServersbeingnotreachableorbeingnonresponsive.Theexecutionmightrecoverlater,orswitchto"canceled"ifitwasnotabletorecoversuccessfuly"recovering":Theexecutionisactivelyrecovering,willswitchbackto"running"iftherecoverywassuccessful
Theobjectreturnedbythestatusmethodmightforexamplelooksomethinglikethis:
{
"state":"running",
"gss":12,
"totalRuntime":123.23,
"aggregators":{
"converged":false,
"max":true,
"phase":2
},
"sendCount":3240364978,
"receivedCount":3240364975
}
Pregel
177
CancelinganExecution/Discardingresults
Tocancelanexecutionwhichisstillrunnning,anddiscardanyintermediareresultsyoucanusethecancelmethod.Thiswillimmediatlyfreeallmemorytakenupbytheexecution,andwillmakeyouloseallintermediarydata.
Youmightgetinconsistentresultsifyoucancelanexecutionwhileitisalreadyinit'sdonestate.Thedataiswrittenmulti-threadedintoallcollectionshardsatonce,thismeanstherearemultipletransactionssimultaniously.Atransactionmightalreadybecommitedwhenyoucanceltheexecutionjob,thereforeyoumightseetheresultinyourcollection.Thisdoesnotapplyifyouconfiguredtheexecutiontonotwritedataintothecollection.
//startasinglesourceshortestpathjob
varexecution=pregel.start("sssp","demograph",{source:"vertices/V"});
pregel.cancel(execution);
AQLintegrationArangoDBsupportsretrievingtemporarypregelresultsthroughtheArangoDBquerylanguage(AQL).Whenourgraphprocessingsubsystemfinishesexecutinganalgorithm,theresultcaneitherbewrittenbackintothedatabaseorkeptinmemory.InbothcasestheresultcanbequeriedviaAQL.Ifthedatawasnotwrittentothedatabasestoreitisonlyheldtemporarily,untiltheusercallsthecancelmethodForexampleausermightwanttoqueryonlynodeswiththemostrankfromtheresultsetofaPageRankexecution.
FORvINPREGEL_RESULT(<handle>)
FILTERv.value>=0.01
RETURNv._key
AvailableAlgorithmsThereareanumberofgeneralparameterswhichapplytoalmostallalgorithms:
store:Isperdefaulttrue,thepregelenginewillwriteresultsbacktothedatabase.ifthevalueisfalsethenyoucanquerytheresultsviaAQLk,seeAQLintegration.maxGSS:Maximumnumberofglobaliterationsforthisalgorithmparallelism:Numberofparellelthreadstouseperworker.Doesnotinfluencethenumberofthreadsusedtoload
orstoredatafromthedatabase(thisdependsonthenumberofshards).
async:Algorithmswichsupportasyncmode,willrunwithoutsynchronizedglobaliterations,mightleadtoperformanceincreasesifyouhaveloadimbalances.resultField:Mostalgorithmswillwritetheresultintothisfield
PageRank
PageRankisawellknownalgorithmtorankdocumentsinagraph.Thealgorithmwillrununtiltheexecutionconverges.Specifyacustomthresholdwiththeparameterthreshold,torunforafixednumberofiterationsusethemaxGSSparameter.
varpregel=require("@arangodb/pregel");
pregel.start("pagerank","graphname",{maxGSS:100,threshold:0.00000001})
Single-SourceShortestPath
Calculatesthedistanceofeachvertextoacertainshortestpath.Thealgorithmwillrununtilitconverges,theiterationsareboundbythediameter(thelongestshortestpath)ofyourgraph.
varpregel=require("@arangodb/pregel");
pregel.start("sssp","graphname",{source:"vertices/1337"})
ConnectedComponents
Pregel
178
Therearetwoalgorithmstofindconnectedcomponentsinagraph.Tofindweaklyconnectedcomponents(WCC)youcanusethealgorithmnamed"connectedcomponents",tofindstronglyconnectedcomponents(SCC)youcanusethealgorithmnamed"scc".BothalgorithmwillassignacomponentIDtoeachvertex.
Aweaklyconnectedcomponentsmeansthatthereexistapathfromeveryvertexpairinthatcomponent.WCCisaverysimpleandfastalgorithm,whichwillonlyworkcorrectlyonundirectedgraphs.Yourresultsondirectedgraphsmayvary,dependingonhowconnectedyourcomponentsare.
InthecaseofSCCacomponentmeanseveryvertexisreachablefromanyothervertexinthesamecomponent.ThealgorithmismorecomplexthantheWCCalgorithmandrequiresmoreRAM,becauseeachvertexneedstostoremuchmorestate.ConsiderusingWCCifyouthinkyourdatamaybesuitableforit.
varpregel=require("@arangodb/pregel");
//weaklyconnectedcomponents
pregel.start("connectedcomponents","graphname")
//stronglyconnectedcomponents
pregel.start("scc","graphname")
Hyperlink-InducedTopicSearch(HITS)
HITSisalinkanalysisalgorithmthatratesWebpages,developedbyJonKleinberg(Thealgorithmisalsoknownashubsandauthorities).
TheideabehindHubsandAuthoritiescomesfromthetypicalstructureoftheweb:Certainwebsitesknownashubs,serveaslargedirectoriesthatarenotactuallyauthoritativeontheinformationthattheyhold.Thesehubsareusedascompilationsofabroadcatalogofinformationthatleadsusersdirecttootherauthoritativewebpages.Thealgorithmassignseachvertextwoscores:Theauthority-scoreandthehub-score.Theauthorityscorerateshowmanygoodhubspointtoaparticularvertex(orwebpage),thehubscorerateshowgood(authoritative)theverticespointedtoare.Formoreseehttps://en.wikipedia.org/wiki/HITS_algorithm
Ourversionofthealgorithmconvergesafteracertainamountoftime.Theparameterthresholdcanbeusedtosetalimitfortheconvergence(measuredasmaximumabsolutedifferenceofthehubandauthorityscoresbetweenthecurrentandlastiteration)Whenyouspecifytheresultfieldname,thehubscorewillbestoredin"_hub"andtheauthorityscorein"_auth".Thealgorithmcanbeexecutedlikethis:
varpregel=require("@arangodb/pregel");
varhandle=pregel.start("hits","yourgraph",{threshold:0.00001,resultField:"score"});
VertexCentrality
Centralitymeasureshelpidentifythemostimportantverticesinagraph.Theycanbeusedinawiderangeofapplications:Forexampletheycanbeusedtoidentifyinfluencersinsocialnetworks,ormiddle-meninterroristnetworks.Therearevariousdefinitionsforcentrality,thesimplestonebeingthevertexdegree.Thesedefinitionswerenotdesignedwithscalabilityinmind.Itisprobablyimpossibletodiscoveranefficientalgorithmwhichcomputestheminadistributedway.Fortunatelytherearescalablesubstitutionsavailable,whichshouldbeequallyusableformostusecases.
Pregel
179
EffectiveCloseness
Acommondefinitionsofcentralityistheclosenesscentrality(orcloseness).Theclosenessofavertexinagraphistheinverseaveragelengthoftheshortestpathbetweenthevertexandallothervertices.Forverticesx,yandshortestdistanced(y,x)itisdefinedas
EffectiveClosenessapproximatestheclosenessmeasure.Thealgorithmworksbyiterativelyestimatingthenumberofshortestpathspassingthrougheachvertex.Thescorewillapproximatesthetherealclosenessscore,sinceitisnotpossibletoactuallycountallshortestpathsduetothehorrendousO(n^2d)memoryrequirements.ThealgorithmisfromthepaperCentralitiesinLargeNetworks:AlgorithmsandObservations(UKanget.al.2011)*
ArangoDBsimplementationapproximatesthenumberofshortestpathineachiterationbyusingaHyperLogLogcounterwith64buckets.Thisshouldworkwellonlargegraphsandonsmalleronesaswell.ThememoryrequirementsshouldbeO(n*d)wherenisthenumberofverticesanddthediameterofyourgraph.Eachvertexwillstoreacounterforeachiterationofthealgorithm.Thealgorithmcanbeusedlikethis
constpregel=require("@arangodb/pregel");
consthandle=pregel.start("effectivecloseness","yourgraph",{resultField:"closeness"});
LineRank
Anothercommonmeasureisthebetweenness*centrality:Itmeasuresthenumberoftimesavertexispartofshortestpathsbetweenanypairsofvertices.Foravertexvbetweennessisdefinedas
Wheretheσrepresentsthenumberofshortestpathsbetweenxandy,andσ(v)representsthenumberofpathsalsopassingthroughavertexv.Byintuitionavertexwithhigherbetweenesscentralitywillhavemoreinformationpassingthroughit.
LineRankapproximatestherandomwalkbetweennessofeveryvertexinagraph.Thisistheprobabilitythatsomeonestartingonanarbitaryvertex,willvisitthisnodewhenherandomlychoosesedgestovisit.Thealgoruthmessentiallybuildsalinegraphoutofyourgraph(switchestheverticesandedges),andthencomputesascoresimilartoPageRank.Thiscanbeconsideredascalableequivalenttovertexbetweeness,whichcanbeexecuteddistributedlyinArangoDB.ThealgorithmisfromthepaperCentralitiesinLargeNetworks:AlgorithmsandObservations(UKanget.al.2011)
Pregel
180
constpregel=require("@arangodb/pregel");
consthandle=pregel.start("linerank","yourgraph",{"resultField":"rank"});
CommunityDetection
Graphsbasedonrealworldnetworksoftenhaveacommunitystructure.Thismeansitispossibletofindgroupsofverticessuchthateacheachvertexgroupisinternallymoredenselyconnectedthanoutsidethegroup.Thishasmanyapplicationswhenyouwanttoanalyzeyournetworks,forexampleSocialnetworksincludecommunitygroups(theoriginoftheterm,infact)basedoncommonlocation,interests,occupation,etc.
LabelPropagation
LabelPropagationcanbeusedtoimplementcommunitydetectiononlargegraphs.Theideaisthateachvertexshouldbeinthecommunitythatmostofhisneighboursarein.WeiterativelydeteminethisbyfirstassigningrandomCommunityID's.Theneachitertation,avertexwillsendit'scurrentcommunityIDtoallhisneighborvertices.TheneachvertexadoptsthecommunityIDhereceivedmostfrequentlyduringtheiteration.
Thealgorithmrunsuntilitconverges,whichlikelyneverreallyhappensonlargegraphs.Thereforeyouneedtospecifyamaximumiterationboundwhichsuitsyou.Thedefaultboundis500iterations,whichislikelytoolargeforyourapplication.Shouldworkbestonundirectedgraphs,resultsondirectedgraphsmightvarydependingonthedensityofyourgraph.
constpregel=require("@arangodb/pregel");
consthandle=pregel.start("labelpropagation","yourgraph",{maxGSS:100,resultField:"community"});
Speaker-ListenerLabelPropagation
TheSpeaker-listenerLabelPropagation(SLPA)canbeusedtoimplementcommunitydetection.Itworkssimilartothelabelpropagationalgorithm,butnoweverynodeadditionallyaccumulatesamemoryofobservedlabels(insteadofforgettingallbutonelabel).
Beforethealgorithmrun,everyvertexisinitializedwithanuniqueID(theinitialcommunitylabel).Duringtherunthreestepsareexecutedforeachvertex:
1. Currentvertexisthelistenerallotherverticesarespeakers2. Eachspeakersendsoutalabelfrommemory,wesendoutarandomlabelwithaprobabilityproportionaltothenumberoftimesthe
vertexobservedthelabel3. Thelistenerremembersoneofthelabels,wealwayschoosethemostfrequentlyobservedlabel
constpregel=require("@arangodb/pregel");
consthandle=pregel.start("slpa","yourgraph",{maxGSS:100,resultField:"community"});
YoucanalsoexecuteSLPAwiththemaxCommunitiesparametertolimitthenumberofouputcommunities.Internallythealgorithmwillstillkeepthememoryofalllabels,buttheoutputisreducedtojusthenmostfrequentlyobservedlabels.
constpregel=require("@arangodb/pregel");
consthandle=pregel.start("slpa","yourgraph",{maxGSS:100,resultField:"community",maxCommunities:1});
//checkthestatusperiodicallyforcompletion
pregel.status(handle);
Pregel
181
FoxxTraditionally,server-sideprojectshavebeendevelopedasstandaloneapplicationsthatguidethecommunicationbetweentheclient-sidefrontendandthedatabasebackend.Thishasledtoapplicationsthatwereeitherdevelopedassinglemonolithsorthatduplicateddataaccessanddomainlogicacrossallservicesthathadtoaccessthedatabase.Additionally,toolstoabstractawaytheunderlyingdatabasecallscouldincuralotofnetworkoverheadwhenusingremotedatabaseswithoutcarefuloptimization.
ArangoDBallowsapplicationdeveloperstowritetheirdataaccessanddomainlogicasmicroservicesrunningdirectlywithinthedatabasewithnativeaccesstoin-memorydata.TheFoxxmicroserviceframeworkmakesiteasytoextendArangoDB'sownRESTAPIwithcustomHTTPendpointsusingmodernJavaScriptrunningonthesameV8engineyouknowfromNode.jsandtheGoogleChromewebbrowser.
Unliketraditionalapproachestostoringlogicinthedatabase(likestoredprocedures),thesemicroservicescanbewrittenasregularstructuredJavaScriptapplicationsthatcanbeeasilydistributedandversioncontrolled.Dependingonyourproject'sneedsFoxxcanbeusedtobuildanythingfromoptimizedRESTendpointsperformingcomplexdataaccesstoentirestandaloneapplicationsrunningdirectlyinsidethedatabase.
FoxxMicroservices
182
FoxxataglanceEachFoxxserviceisdefinedbyaJSONmanifestspecifyingtheentrypoint,anyscriptsdefinedbytheservice,possibleconfigurationoptionsandFoxxdependencies,aswellasothermetadata.Withinaservice,theseoptionsareexposedastheservicecontext.
AttheheartoftheFoxxframeworkliestheFoxxRouterwhichisusedtodefineHTTPendpoints.AservicecanaccessthedatabaseeitherdirectlyfromitscontextusingprefixedcollectionsortheArangoDBdatabaseAPI.
WhileFoxxisprimarilydesignedtobeusedtoaccessthedatabaseitself,ArangoDBalsoprovidesanAPItomakeHTTPrequeststoexternalservices.
Scriptscanbeusedtoperformone-offtasks,whichcanalsobescheduledtobeperformedasynchronouslyusingthebuilt-injobqueue.
Finally,FoxxservicescanbeinstalledandmanagedovertheWeb-UIorthroughArangoDBsHTTPAPI.
HowdoesitworkFoxxservicesconsistofJavaScriptcoderunningintheV8JavaScriptruntimeembeddedinsideArangoDB.EachserviceismountedineachavailableV8context(thenumberofcontextscanbeadjustedintheArangoDBconfiguration).Incomingrequestsaredistributedaccrossthesecontextsautomatically.
Ifyou'recomingfromanotherJavaScriptenvironmentlikeNode.jsthisissimilartorunningmultipleNode.jsprocessesbehindaloadbalancer:youshouldnotrelyonserver-sidestate(otherthanthedatabaseitself)betweendifferentrequestsasthereisnowayofmakingsureconsecutiverequestswillbehandledinthesamecontext.
BecausetheJavaScriptcodeisrunninginsidethedatabaseanotherdifferenceisthatallFoxxandArangoDBAPIsarepurelysynchronousandshouldbeconsideredblocking.Thisisespeciallyimportantfortransactions,whichinArangoDBcanexecutearbitrarycodebutmayhavetolockentirecollections(effectivelypreventinganydatatobewritten)untilthecodehascompleted.
Forinformationonhowthisaffectsinteroperabilitywiththird-partyJavaScriptmoduleswrittenforotherJavaScriptenvironmentsseethechapterondependencies.
DevelopmentmodeDevelopmentmodeallowsyoutomakechangestodeployedservicesin-placedirectlyonthedatabaseserver'sfilesystemwithoutdownloadingandre-uploadingtheservicebundle.Additionallyerrormessageswillcontainstacktraces.
YoucantoggledevelopmentmodeonandoffintheservicesettingstabofthewebinterfaceorusingtheHTTPAPI.Onceactivatedtheservice'sfilesystempathwillbeshownintheinfotab.
Onceenabledtheservice'ssourcefilesandmanifestwillbere-evaluated,andthesetupscript(ifpresent)re-executed,everytimearouteoftheserviceisaccessed,effectivelyre-deployingtheserviceoneveryrequest.Asthenameindicatesthisisintendedtobeusedstrictlyduringdevelopmentandismostdefinitelyabadideaonproductionservers.Theadditionalinformationexposedduringdevelopmentmodemayincludefilesystempathsandpartsoftheservice'ssourcecode.
Alsonotethatifyouareservingstaticfilesaspartofyourservice,accessingthesefilesfromabrowsermayalsotriggerare-deploymentoftheservice.Finally,makingHTTPrequeststoaservicerunningindevelopmentmodefromwithintheservice(i.e.usingthe@arangodb/requestmoduletoaccesstheserviceitself)isprobablynotagoodideaeither.
Bewareofdeletingthedatabasetheserviceisdeployedon:itwillerasethesourcefilesoftheservicealongwiththecollections.Youshouldbackupthecodeyouworkedonindevelopmentbeforedoingthattoavoidlosingyourprogress.
FoxxstoreTheFoxxstoreprovidesaccesstoanumberofready-to-useofficialandcommunity-maintainedFoxxservicesyoucaninstallwithasingleclick,includingexampleservicesandwrappersforexternalSaaStoolsliketransactionale-mailservices,bugloggersoranalyticstrackers.
YoucanfindtheFoxxstoreinthewebinterfacebyusingtheAddServicebuttonintheservicelist.
Ataglance
183
Cluster-Foxx
WhenrunningArangoDBinaclustertheFoxxserviceswillrunoneachcoordinator.Installing,upgradinganduninstallingservicesonanycoordinatorwillautomaticallydistributetheservicetotheothercoordinators,makingdeploymentsaseasyasinsingle-servermode.However,thismeanstherearesomelimitations:
Youshouldavoidanykindoffilesystemstatebeyondthedeployedservicebundleitself.Don'twritedatatothefilesystemorencodeanyexpectationsofthefilesystemstateotherthanthefilesintheservicefolderthatwereinstalledaspartoftheservice(e.g.fileuploadsorcustomlogfiles).
Additionally,thedevelopmentmodewillleadtoaninconsistentstateoftheclusteruntilitisdisabled.WhileaserviceisrunningindevelopmentmodeyoucanmakechangestotheserviceonthefilesystemofanycoordinatorandseethemreflectedinrealtimejustlikewhenrunningArangoDBasasingleserver.Howeverthechangesmadeononecoordinatorwillnotbereflectedacrossothercoordinatorsuntilthedevelopmentmodeisdisabled.Whendisablingthedevelopmentmodeforaservice,thecoordinatorwillcreateanewbundleanddistributeitacrosstheservicelikeamanualupgradeoftheservice.
Forthesereasonsitisstronglyrecommendednottousedevelopmentmodeinaclusterwithmultiplecoordinatorsunlessyouaresurethatnorequestsorchangeswillbemadetoothercoordinatorswhileyouaremodifyingtheservice.Usingdevelopmentmodeinaproductionclusterisextremelyunsafeandhighlydiscouraged.
Ataglance
184
GettingStartedWe'regoingtostartwithanemptyfolder.Thiswillbetherootfolderofourservices.Youcannameitsomethingcleverbutforthecourseofthisguidewe'llassumeit'scalledthenameofyourservice:getting-started.
Firstweneedtocreateamanifest.Createanewfilecalledmanifest.jsonandaddthefollowingcontent:
{
"engines":{
"arangodb":"^3.0.0"
}
}
ThisjusttellsArangoDBtheserviceiscompatiblewithversions3.0.0andlater(allthewayuptobutnotincluding4.0.0),allowingolderversionsofArangoDBtounderstandthatthisservicelikelywon'tworkforthemandnewerversionswhatbehaviortoemulateshouldtheystillsupportit.
Thelittlehattotheleftoftheversionnumberisnotatypo,it'scalleda"caret"andindicatestheversionrange.Foxxusessemanticversioning(alsocalled"semver")formostofitsversionhandling.Youcanfindoutmoreabouthowsemverworksattheofficialsemverwebsite.
Nextwe'llneedtospecifyanentrypointtoourservice.ThisistheJavaScriptfilethatwillbeexecutedtodefineourservice'sHTTPendpoints.Wecandothisbyaddinga"main"fieldtoourmanifest:
{
"engines":{
"arangodb":"^3.0.0"
},
"main":"index.js"
}
That'sallweneedinourmanifestfornow,solet'snextcreatetheindex.jsfile:
'usestrict';
constcreateRouter=require('@arangodb/foxx/router');
constrouter=createRouter();
module.context.use(router);
Thefirstlinecausesourfiletobeinterpretedusingstrictmode.AllexamplesintheArangoDBdocumentationassumestrictmode,soyoumightwanttofamiliarizeyourselfwithitifyouhaven'tencountereditbefore.
Thesecondlineimportsthe@arangodb/foxx/routermodulewhichprovidesafunctionforcreatingnewFoxxrouters.We'reusingthisfunctiontocreateanewrouterobjectwhichwe'llbeusingforourservice.
Themodule.contextistheso-calledFoxxcontextorservicecontext.ThisvariableisavailableinallfilesthatarepartofyourFoxxserviceandprovidesaccesstoFoxxAPIsspecifictothecurrentservice,liketheusemethod,whichtellsFoxxtomounttherouterinthisservice(andtoexposeitsroutestoHTTP).
Nextlet'sdefinearoutethatprintsagenericgreeting:
//continued
router.get('/hello-world',function(req,res){
res.send('HelloWorld!');
})
.response(['text/plain'],'Agenericgreeting.')
.summary('Genericgreeting')
.description('Printsagenericgreeting.');
Therouterprovidesthemethodsget,post,etccorrespondingtoeachHTTPverbaswellasthecatch-allall.ThesemethodsindicatethatthegivenrouteshouldbeusedtohandleincomingrequestswiththegivenHTTPverb(oranymethodwhenusingall).
Gettingstarted
185
Thesemethodstakeanoptionalpath(ifomitted,itdefaultsto"/")aswellasarequesthandler,whichisafunctiontakingthereq(request)andres(response)objectstohandletheincomingrequestandgeneratetheoutgoingresponse.IfyouhaveusedtheexpressframeworkinNode.js,youmayalreadybefamiliarwithhowthisworks,otherwisecheckoutthechapteronroutes.
Theobjectreturnedbytherouter'smethodsprovidesadditionalmethodstoattachmetadataandvalidationtotheroute.We'reusingsummaryanddescriptiontodocumentwhattheroutedoes--thesearen'tstrictlynecessarybutgiveussomeniceauto-generateddocumentation.Theresponsemethodletsusadditionallydocumenttheresponsecontenttypeandwhattheresponsebodywillrepresent.
Tryitout
AtthispointyoucanuploadtheservicefolderasaziparchivefromthewebinterfaceusingtheServicestab.
ClickAddServicethenpicktheZipoptioninthedialog.Youwillneedtoprovideamountpath,whichistheURLprefixatwhichtheservicewillbemounted(e.g./getting-started).
Onceyouhavepickedtheziparchiveusingthefilepicker,theuploadshouldbeginimmediatelyandyourserviceshouldbeinstalled.OtherwisepresstheInstallbuttonandwaitforthedialogtodisappearandtheservicetoshowupintheservicelist.
Clickanywhereonthecardwithyourmountpathonthelabeltoopentheservice'sdetails.
IntheAPIdocumentationyoushouldseetheroutewedefinedearlier(/hello-world)withthewordGETnexttoitindicatingtheHTTPmethoditsupportsandthesummaryweprovidedontheright.Byclickingontheroute'spathyoucanopenthedocumentationfortheroute.
Notethatthedescriptionweprovidedappearsinthegenerateddocumentationaswellasthedescriptionweaddedtotheresponse(whichshouldcorrectlyindicatethecontenttypetext/plain,i.e.plaintext).
ClicktheTryitout!buttontosendarequesttotherouteandyoushouldseeanexamplerequestwiththeservice'sresponse:"HelloWorld!".
Congratulations!Youhavejustcreated,installedandusedyourfirstFoxxservice.
Parametervalidation
Let'saddanotherroutethatprovidesamorepersonalizedgreeting:
//continued
constjoi=require('joi');
router.get('/hello/:name',function(req,res){
res.send(`Hello${req.pathParams.name}`);
})
.pathParam('name',joi.string().required(),'Nametogreet.')
.response(['text/plain'],'Apersonalizedgreeting.')
.summary('Personalizedgreeting')
.description('Printsapersonalizedgreeting.');
ThefirstlineimportsthejoimodulefromnpmwhichcomesbundledwithArangoDB.JoiisavalidationlibrarythatisusedthroughoutFoxxtodefineschemasandparametertypes.
Note:Youcanbundleyourownmodulesfromnpmbyinstallingtheminyourservicefolderandmakingsurethenode_modulesfolderisincludedinyourziparchive.Formoreinformationseethesectiononmoduledependenciesinthechapterondependencies.
ThepathParammethodallowsustospecifyparametersweareexpectinginthepath.Thefirstargumentcorrespondstotheparameternameinthepath,thesecondargumentisajoischematheparameterisexpectedtomatchandthefinalargumentservestodescribetheparameterintheAPIdocumentation.
ThepathparametersareaccessiblefromthepathParamspropertyoftherequestobject.We'reusingatemplatestringtogeneratetheserver'sresponsecontainingtheparameter'svalue.
Gettingstarted
186
NotethatrouteswithpathparametersthatfailtovalidatefortherequestURLwillbeskippedasiftheywouldn'texist.Thisallowsyoutodefinemultipleroutesthatareonlydistinguishedbytheschemasoftheirpathparameters(e.g.aroutetakingonlynumericparametersandonetakinganystringasafallback).
Let'stakethisfurtherandcreatearoutethattakesaJSONrequestbody:
//continued
router.post('/sum',function(req,res){
constvalues=req.body.values;
res.send({
result:values.reduce(function(a,b){
returna+b;
},0)
});
})
.body(joi.object({
values:joi.array().items(joi.number().required()).required()
}).required(),'Valuestoaddtogether.')
.response(joi.object({
result:joi.number().required()
}).required(),'Sumoftheinputvalues.')
.summary('Addupnumbers')
.description('Calculatesthesumofanarrayofnumbervalues.');
Notethatweusedposttodefinethisrouteinsteadofget(whichdoesnotsupportrequestbodies).TryingtosendaGETrequesttothisroute'sURL(intheabsenceofagetrouteforthesamepath)willresultinFoxxrespondingwithanappropriateerrorresponse,indicatingthesupportedHTTPmethods.
AsthisroutenotonlyexpectsaJSONobjectasinputbutalsorespondswithaJSONobjectasoutputweneedtodefinetwoschemas.Wedon'tstrictlyneedaresponseschemabutithelpsdocumentingwhattherouteshouldbeexpectedtorespondwithandwillshowupintheAPIdocumentation.
Becausewe'repassingaschematotheresponsemethodwedon'tneedtoexplicitlytellFoxxwearesendingaJSONresponse.ThepresenceofaschemaintheabsenceofacontenttypealwaysimplieswewantJSON.Thoughwecouldjustadd["application/json"]asanadditionalargumentaftertheschemaifwewantedtomakethismoreexplicit.
Thebodymethodworksthesamewayastheresponsemethodexcepttheschemawillbeusedtovalidatetherequestbody.Iftherequestbodycan'tbeparsedasJSONordoesn'tmatchtheschema,Foxxwillrejecttherequestwithanappropriateerrorresponse.
Creatingcollections
TherealpowerofFoxxcomesfrominteractingwiththedatabaseitself.Inordertobeabletouseacollectionfromwithinourservice,weshouldfirstmakesurethatthecollectionactuallyexists.Therightplacetocreatecollectionsyourserviceisgoingtouseisinasetupscript,whichFoxxwillexecuteforyouwheninstallingorupdatingtheservice.
Firstcreateanewfoldercalled"scripts"intheservicefolder,whichwillbewhereourscriptsaregoingtolive.Forsimplicity'ssake,oursetupscriptwillliveinafilecalledsetup.jsinsidethatfolder:
//continued
'usestrict';
constdb=require('@arangodb').db;
constcollectionName='myFoxxCollection';
if(!db._collection(collectionName)){
db._createDocumentCollection(collectionName);
}
Thescriptusesthedbobjectfromthe@arangodbmodule,whichletsusinteractwiththedatabasetheFoxxservicewasinstalledinandthecollectionsinsidethatdatabase.Becausethescriptmaybeexecutedmultipletimes(i.e.wheneverweupdatetheserviceorwhentheserverisrestarted)weneedtomakesurewedon'taccidentallytrytocreatethesamecollectiontwice(whichwouldresultinanexception);wedothatbyfirstcheckingwhetheritalreadyexistsbeforecreatingit.
Gettingstarted
187
The_collectionmethodlooksupacollectionbynameandreturnsnullifnocollectionwiththatnamewasfound.The_createDocumentCollectionmethodcreatesanewdocumentcollectionbyname(_createEdgeCollectionalsoexistsandworksanalogouslyforedgecollections).
Note:Becausewehavehardcodedthecollectionname,multiplecopiesoftheserviceinstalledalongsideeachotherinthesamedatabasewillsharethesamecollection.Becausethismaynotalwaysbewhatyouwant,theFoxxcontextalsoprovidesthecollectionNamemethodwhichappliesamountpointspecificprefixtoanygivencollectionnametomakeituniquetotheservice.Italsoprovidesthecollectionmethod,whichbehavesalmostexactlylikedb._collectionexceptitalsoappliestheprefixbeforelookingthecollectionup.
Nextweneedtotellourserviceaboutthescriptbyaddingittothemanifestfile:
{
"engines":{
"arangodb":"^3.0.0"
},
"main":"index.js",
"scripts":{
"setup":"scripts/setup.js"
}
}
Theonlythingthathaschangedisthatweaddeda"scripts"fieldspecifyingthepathofthesetupscriptwejustwrote.
Gobacktothewebinterfaceandupdatetheservicewithournewcode,thenchecktheCollectionstab.Ifeverythingworkedright,youshouldseeanewcollectioncalled"myFoxxCollection".
AccessingcollectionsLet'sexpandourservicebyaddingafewmoreroutestoourindex.js:
//continued
constdb=require('@arangodb').db;
consterrors=require('@arangodb').errors;
constfoxxColl=db._collection('myFoxxCollection');
constDOC_NOT_FOUND=errors.ERROR_ARANGO_DOCUMENT_NOT_FOUND.code;
router.post('/entries',function(req,res){
constdata=req.body;
constmeta=foxxColl.save(req.body);
res.send(Object.assign(data,meta));
})
.body(joi.object().required(),'Entrytostoreinthecollection.')
.response(joi.object().required(),'Entrystoredinthecollection.')
.summary('Storeanentry')
.description('Storesanentryinthe"myFoxxCollection"collection.');
router.get('/entries/:key',function(req,res){
try{
constdata=foxxColl.document(req.pathParams.key);
res.send(data)
}catch(e){
if(!e.isArangoError||e.errorNum!==DOC_NOT_FOUND){
throwe;
}
res.throw(404,'Theentrydoesnotexist',e);
}
})
.pathParam('key',joi.string().required(),'Keyoftheentry.')
.response(joi.object().required(),'Entrystoredinthecollection.')
.summary('Retrieveanentry')
.description('Retrievesanentryfromthe"myFoxxCollection"collectionbykey.');
We'reusingthesaveanddocumentmethodsofthecollectionobjecttostoreandretrievedocumentsinthecollectionwecreatedinoursetupscript.Becausewedon'tcarewhatthedocumentslooklikeweallowanyattributesontherequestbodyandjustacceptanobject.
Gettingstarted
188
BecausethekeywillbeautomaticallygeneratedbyArangoDBwhenonewasn'tspecifiedintherequestbody,we'reusingObject.assigntoapplytheattributesofthemetadataobjectreturnedbythesavemethodtothedocumentbeforereturningitfromourfirstroute.
Thedocumentmethodreturnsadocumentinacollectionbyits_keyor_id.HoweverwhennomatchingdocumentexistsitthrowsanArangoErrorexception.BecausewewanttoprovideamoredescriptiveerrormessagethanArangoDBdoesoutofthebox,weneedtohandlethaterrorexplicitly.
AllArangoErrorexceptionshaveatruthyattributeisArangoErrorthathelpsyourecognizingtheseerrorswithouthavingtoworryaboutinstanceofchecks.TheyalsoprovideanerrorNumandanerrorMessage.Ifyouwanttocheckforspecificerrorsyoucanjustimporttheerrorsobjectfromthe@arangodbmoduleinsteadofhavingtomemorizenumericerrorcodes.
Insteadofdefiningourownresponselogicfortheerrorcasewejustuseres.throw,whichmakestheresponseobjectthrowanexceptionFoxxcanrecognizeandconverttotheappropriateserverresponse.WealsopassalongtheexceptionitselfsoFoxxcanprovidemorediagnosticinformationwhenwewantitto.
Wecouldextendthepostroutetosupportarraysofobjectsaswell,eachobjectfollowingacertainschema:
//storeschemainvariabletomakeitre-usable,see.body()
constdocSchema=joi.object().required().keys({
name:joi.string().required(),
age:joi.number().required()
}).unknown();//allowadditionalattributes
router.post('/entries',function(req,res){
constmultiple=Array.isArray(req.body);
constbody=multiple?req.body:[req.body];
letdata=[];
for(vardocofbody){
constmeta=foxxColl.save(doc);
data.push(Object.assign(doc,meta));
}
res.send(multiple?data:data[0]);
})
.body(joi.alternatives().try(
docSchema,
joi.array().items(docSchema)
),'Entryorentriestostoreinthecollection.')
.response(joi.alternatives().try(
joi.object().required(),
joi.array().items(joi.object().required())
),'Entryorentriesstoredinthecollection.')
.summary('Storeentryorentries')
.description('Storeasingleentryormultipleentriesinthe"myFoxxCollection"collection.');
WritingdatabasequeriesStoringandretrievingentriesisfine,butrightnowwehavetomemorizeeachkeywhenwecreateanentry.Let'saddaroutethatgivesusalistofthekeysofallentriessowecanusethosetolookanentryupindetail.
ThenaïveapproachwouldbetousethetoArray()methodtoconverttheentirecollectiontoanarrayandjustreturnthat.Butwe'reonlyinterestedinthekeysandtheremightpotentiallybesomanyentriesthatfirstretrievingeverysingledocumentmightgetunwieldy.Let'swriteashortAQLquerytodothisinstead:
//continued
constaql=require('@arangodb').aql;
router.get('/entries',function(req,res){
constkeys=db._query(aql`
FORentryIN${foxxColl}
RETURNentry._key
`);
res.send(keys);
})
.response(joi.array().items(
joi.string().required()
Gettingstarted
189
).required(),'Listofentrykeys.')
.summary('Listentrykeys')
.description('Assemblesalistofkeysofentriesinthecollection.');
Herewe'reusingtwonewthings:
The_querymethodexecutesanAQLqueryintheactivedatabase.
Theaqltemplatestringhandlerallowsustowritemulti-lineAQLqueriesandalsohandlesqueryparametersandcollectionnames.InsteadofhardcodingthenameofthecollectionwewanttouseinthequerywecansimplyreferencethefoxxCollvariablewedefinedearlier--itrecognizesthevalueasanArangoDBcollectionobjectandknowswearespecifyingacollectionratherthanaregularvalueeventhoughAQLdistinguishesbetweenthetwo.
Note:Ifyouaren'tusedtoJavaScripttemplatestringsandtemplatestringhandlersjustthinkofaqlasafunctionthatreceivesthemultilinestringsplitatevery${}expressionaswellasanarrayofthevaluesofthoseexpressions--that'sactuallyallthereistoit.
Alternatively,here'saversionwithouttemplatestrings(noticehowmuchcleanertheaqlversionwillbeincomparisonwhenyouhavemultiplevariables):
constkeys=db._query(
'FORentryIN@@collRETURNentry._key',
{'@coll':foxxColl}
);
Nextsteps
YounowknowhowtocreateaFoxxservicefromscratch,howtohandleuserinputandhowtoaccessthedatabasefromwithinyourFoxxservicetostore,retrieveandquerydatayoustoreinsideArangoDB.ThisshouldallowyoutobuildmeaningfulAPIsforyourownapplicationsbuttherearemanymorethingsyoucandowithFoxx:
Needtogofaster?Turnondevelopmentmodeandhackonyourcoderightontheserver.
Concernedaboutsecurity?Youcouldaddauthenticationtoyourservicetoprotectaccesstothedatabeforeitevenleavesthedatabase.
Writingasinglepageapp?YoucouldstoresomebasicassetsrightinsideyourFoxxservice.
Needtointegrateexternalservices?YoucanmakeHTTPrequestsfrominsideFoxxandusequeuedjobstoperformthatworkinthebackground.
Tiredofreinventingthewheel?Learnaboutdependencies.
Everythingbroken?Youcanwriteteststomakesureyourlogicremainssound.
Gettingstarted
190
ManifestfilesEveryservicecomeswithamanifest.jsonfileprovidingmetadata.Thefollowingfieldsareallowedinmanifests:
configuration:Object(optional)
Anobjectdefiningtheconfigurationoptionsthisservicerequires.
defaultDocument:string(optional)
Ifspecified,the/(root)routeoftheservicewillautomaticallyredirecttothegivenrelativepath,e.g.:
"defaultDocument":"index.html"
ThiswouldhavethesameeffectascreatingthefollowingrouteinJavaScript:
constcreateRouter=require('@arangodb/foxx/router');
constindexRouter=createRouter();
indexRouter.all('/',function(req,res){
res.redirect('index.html');
});
module.context.use(indexRouter);
Note:Asof3.0.0thisfieldcansafelybeomitted;thevaluenolongerdefaultsto"index.html".
dependencies:Object(optional)andprovides:Object(optional)
Objectsspecifyingotherservicesthisservicehasasdependenciesandwhatdependenciesitcanprovidetootherservices.
engines:Object(optional)
AnobjectindicatingthesemanticversionrangesofArangoDB(orcompatibleenvironments)theservicewillbecompatiblewith,e.g.:
"engines":{
"arangodb":"^3.0.0"
}
ThisshouldcorrectlyindicatetheminimumversionofArangoDBtheservicehasbeentestedagainst.FoxxmaintainsastrictsemanticversioningpolicyasofArangoDB3.0.0soitisgenerallysafetousesemverranges(e.g. 3.0.0tomatchanyversiongreaterorequalto3.0.0andbelow4.0.0)formaximumcompatibility.
files:Object(optional)
Anobjectdefiningfileassetsservedbythisservice.
lib:string(Default:".")
TherelativepathtotheFoxxJavaScriptfilesintheservice,e.g.:
"lib":"lib"
Thiswouldresultinthemainentrypoint(seebelow)andotherJavaScriptpathsbeingresolvedasrelativetothelibfolderinsidetheservicefolder.
main:string(optional)
Therelativepathtothemainentrypointofthisservice(relativetolib,seeabove),e.g.:
"main":"index.js"
ThiswouldresultinFoxxloadingandexecutingthefileindex.jswhentheserviceismountedorstarted.
Servicemanifest
191
Note:whileitistechnicallypossibletoomitthisfield,youwilllikelywanttoprovideanentrypointtoyourserviceasthisistheonlywaytoexposeHTTProutesorexportaJavaScriptAPI.
scripts:Object(optional)
Anobjectdefiningnamedscriptsprovidedbythisservice,whichcaneitherbeuseddirectlyorasqueuedjobsbyotherservices.
tests:stringorArray<string>(optional)
ApathorlistofpathsofJavaScripttestsprovidedforthisservice.
Additionallymanifestscanprovidethefollowingmetadata:
author:string(optional)
Thefullnameoftheauthoroftheservice(i.e.you).Thiswillbeshowninthewebinterface.
contributors:Array<string>(optional)
Alistofnamesofpeoplethathavecontributedtothedevelopmentoftheserviceinsomeway.Thiswillbeshowninthewebinterface.
description:string(optional)
Ahuman-readabledescriptionoftheservice.Thiswillbeshowninthewebinterface.
keywords:Array<string>(optional)
Alistofkeywordsthathelpcategorizethisservice.ThisisusedbytheFoxxStoreinstallerstoorganizeservices.
license:string(optional)
Astringidentifyingthelicenseunderwhichtheserviceispublished,ideallyintheformofanSPDXlicenseidentifier.Thiswillbeshowninthewebinterface.
name:string(optional)
ThenameoftheFoxxservice.AllowedcharactersareA-Z,0-9,theASCIIhyphen(-)andunderscore(_)characters.Thenamemustnotstartwithanumber.Thiswillbeshowninthewebinterface.
thumbnail:string(optional)
Thefilenameofathumbnailthatwillbeusedalongsidetheserviceinthewebinterface.ThisshouldbeaJPEGorPNGimagethatlooksgoodatsizes50x50and160x160.
version:string(optional)
TheversionnumberoftheFoxxservice.Theversionnumbermustfollowthesemanticversioningformat.Thiswillbeshowninthewebinterface.
Examples
{
"name":"example-foxx-service",
"version":"3.0.0-dev",
"license":"MIT",
"description":"Anexampleservicewitharelativelyfull-featuredmanifest.",
"thumbnail":"foxx-icon.png",
"keywords":["demo","service"],
"author":"ArangoDBGmbH",
"contributors":[
"AlanPlum<[email protected]>"
],
"lib":"dist",
"main":"entry.js",
"defaultDocument":"welcome.html",
"engines":{
"arangodb":"^3.0.0"
},
"files":{
Servicemanifest
192
"welcome.html":"assets/index.html",
"hello.jpg":"assets/hello.jpg"
"world.jpg":{
"path":"assets/world.jpg",
"type":"image/jpeg",
"gzip":false
}
},
"tests":"dist/**.spec.js"
}
Servicemanifest
193
FoxxservicecontextTheservicecontextprovidesaccesstomethodsandattributesthatarespecifictoagivenservice.InaFoxxservicethecontextisgenerallyavailableasthemodule.contextvariable.Withinarouter'srequesthandlertherequestandresponseobjects'contextattributealsoprovideaccesstothecontextoftheservicetheroutewasmountedin(whichmaybedifferentfromtheonetheroutehandlerwasdefinedin).
Examples
//inservice/my-foxx-1
constcreateRouter=require('@arangodb/foxx/router');
constrouter=createRouter();
//Seethechapterondependenciesformoreinfoon
//howexportsanddependenciesworkacrossservices
module.exports={routes:router};
router.get(function(req,res){
module.context.mount==='/my-foxx-1';
req.context.mount==='/my-foxx-2';
res.write('Hellofrommy-foxx-1');
});
//inservice/my-foxx-2
constcreateRouter=require('@arangodb/foxx/router');
constrouter2=createRouter();
module.context.use(router2);
router2.post(function(req,res){
module.context.mount==='/my-foxx-2';
req.context.mount==='/my-foxx-2';
res.write('Hellofrommy-foxx-2');
});
constrouter1=module.context.dependencies.myFoxx1.routes;
module.context.use(router1);
Theservicecontextspecifiesthefollowingproperties:
argv:any
Anyargumentspassedinifthecurrentfilewasexecutedasascriptorqueuedjob.
basePath:string
Thefilesystempathoftheservice,i.e.thefolderinwhichtheservicewasinstalledtobyArangoDB.
baseUrl:string
ThebaseURLoftheservice,relativetotheArangoDBserver,e.g./_db/_system/my-foxx.
collectionPrefix:string
TheprefixthatwillbeusedbycollectionandcollectionNametoderivethenamesofservice-specificcollections.Thisisderivedfromtheservice'smountpoint,e.g./my-foxxbecomesmy_foxx.
configuration:Object
Configurationoptionsfortheservice.
dependencies:Object
Configureddependenciesfortheservice.
isDevelopment:boolean
Indicateswhethertheserviceisrunningindevelopmentmode.
Servicecontext
194
isProduction:boolean
TheinverseofisDevelopment.
manifest:Object
Theparsedmanifestfileoftheservice.
mount:string
Themountpointoftheservice,e.g./my-foxx.
apiDocumentationmodule.context.apiDocumentation([options]):Function
DEPRECATED
CreatesarequesthandlerthatservestheAPIdocumentation.
Note:ThismethodhasbeendeprecatedinArangoDB3.1andreplacedwiththemorestraightforwardcreateDocumentationRoutermethodprovidingthesamefunctionality.
Arguments
SeecreateDocumentationRouterbelow.
Examples
//ServetheAPIdocsforthecurrentservice
router.get('/docs/*',module.context.apiDocumentation());
//Notethatthepathmustendwithawildcard
//andtheroutemustuseHTTPGET.
createDocumentationRoutermodule.context.createDocumentationRouter([options]):Router
CreatesarouterthatservestheAPIdocumentation.
Note:Theroutercanbemountedlikeanyotherchildrouter(seeexamplesbelow).
Arguments
options:Object(optional)
Anobjectwithanyofthefollowingproperties:
mount:string(Default:module.context.mount)
Themountpathoftheservicetoservethedocumentationof.
indexFile:string(Default:"index.html")
FilenameoftheHTMLfileservingtheAPIdocumentation.
swaggerRoot:string(optional)
FullpathofthefoldercontainingtheSwaggerassetsandtheindexFile.DefaultstotheSwaggerassetsusedbythewebinterface.
before:Function(optional)
Afunctionthatwillbeexecutedbeforearequestishandled.
Ifthefunctionreturnsfalsetherequestwillnotbeprocessedanyfurther.
Ifthefunctionreturnsanobject,itsattributeswillbeusedtooverridetheoptionsforthecurrentrequest.
Servicecontext
195
Anyotherreturnvaluewillbeignored.
Ifoptionsisafunctionitwillbeusedasthebeforeoption.
IfoptionsisastringitwillbeusedastheswaggerRootoption.
ReturnsaFoxxrouter.
Examples
//ServetheAPIdocsforthecurrentservice
router.use('/docs',module.context.createDocumentationRouter());
//--or--
//ServetheAPIdocsfortheservicetherouterismountedin
router.use('/docs',module.context.createDocumentationRouter(function(req){
return{mount:req.context.mount};
}));
//--or--
//ServetheAPIdocsonlyforusersauthenticatedwithArangoDB
router.use('/docs',module.context.createDocumentationRouter(function(req,res){
if(req.suffix==='swagger.json'&&!req.arangoUser){
res.throw(401,'Notauthenticated');
}
}));
collectionmodule.context.collection(name):ArangoCollection|null
PassesthegivennametocollectionName,thenlooksupthecollectionwiththeprefixedname.
Arguments
name:string
Unprefixednameoftheservice-specificcollection.
Returnsacollectionornullifnocollectionwiththeprefixednameexists.
collectionNamemodule.context.collectionName(name):string
PrefixesthegivennamewiththecollectionPrefixforthisservice.
Arguments
name:string
Unprefixednameoftheservice-specificcollection.
Returnstheprefixedname.
Examples
module.context.mount==='/my-foxx'
module.context.collectionName('doodads')==='my_foxx_doodads'
filemodule.context.file(name,[encoding]):Buffer|string
PassesthegivennametofileName,thenloadsthefilewiththeresultingname.
Servicecontext
196
Arguments
name:string
Nameofthefiletoload,relativetothecurrentservice.
encoding:string(optional)
Encodingofthefile,e.g.utf-8.Ifomittedthefilewillbeloadedasarawbufferinsteadofastring.
Returnsthefile'scontents.
fileNamemodule.context.fileName(name):string
Resolvesthegivenfilenamerelativetothecurrentservice.
Arguments
name:string
Nameofthefile,relativetothecurrentservice.
Returnstheabsolutefilepath.
usemodule.context.use([path],router):Endpoint
Mountsagivenrouterontheservicetoexposetherouter'sroutesontheservice'smountpoint.
Arguments
path:string(Default:"/")
Pathtomounttherouterat,relativetotheservice'smountpoint.
router:Router|Middleware
Arouterormiddlewaretomount.
ReturnsanEndpointforthegivenrouterormiddleware.
Note:Mountingservicesatruntime(e.g.withinrequesthandlersorqueuedjobs)isnotsupported.
Servicecontext
197
FoxxconfigurationFoxxservicescandefineconfigurationparameterstomakethemmorere-usable.
Theconfigurationobjectmapsnamestoconfigurationparameters:
Thekeyisthenameunderwhichtheparameterwillbeavailableontheservicecontext'sconfigurationproperty.
Thevalueisaparameterdefinition.
Theparameterdefinitioncanhavethefollowingproperties:
description:string
Humanreadabledescriptionoftheparameter.
type:string(Default:"string")
Typeoftheconfigurationparameter.Supportedvaluesare:
"integer"or"int":anyfiniteintegernumber.
"boolean"or"bool":thevaluestrueorfalse.
"number":anyfinitedecimalorintegernumber.
"string":anystringvalue.
"json":anywell-formedJSONvalue.
"password":likestringbutwillbedisplayedasamaskedinputfieldinthewebfrontend.
default:any
Defaultvalueoftheconfigurationparameter.
required:(Default:true)
Whethertheparameterisrequired.
Iftheconfigurationhasparametersthatdonotspecifyadefaultvalue,youneedtoconfiguretheservicebeforeitbecomesactive.InthemeantimeafallbackservicelicationwillbemountedthatrespondstoallrequestswithaHTTP500statuscodeindicatingaserver-sideerror.
TheconfigurationparametersofamountedservicecanbeadjustedfromthewebinterfacebyclickingtheConfigurationbuttonintheservicedetails.
Examples
"configuration":{
"currency":{
"description":"Currencysymboltouseforpricesintheshop.",
"default":"$",
"type":"string"
},
"secretKey":{
"description":"Secretkeytouseforsigningsessiontokens.",
"type":"password"
}
}
Configuration
198
DependencymanagementTherearetwothingscommonlycalled"dependencies"inFoxx:
Moduledependencies,i.e.dependenciesonexternalJavaScriptmodules(e.g.fromthepublicnpmregistry)
Foxxdependencies,i.e.dependenciesbetweenFoxxservices
Let'slookattheminmoredetail:
Moduledependencies
Youcanusethenode_modulesfoldertobundlethird-partyFoxx-compatiblenpmandNode.jsmoduleswithyourFoxxservice.Typicallythisisachievedbyaddingapackage.jsonfiletoyourprojectspecifyingnpmdependenciesusingthedependenciesattributeandinstallingthemwiththenpmcommand-linetool.
Makesuretoincludetheactualnode_modulesfolderinyourFoxxservicebundleasArangoDBwillnotdoanythingspecialtoinstallthesedependencies.AlsokeepinmindthatbundlingextraneousmoduleslikedevelopmentdependenciesmaybloatthefilesizeofyourFoxxservicebundle.
Compatibilitycaveats
UnlikeJavaScriptinbrowsersorNode.js,theJavaScriptenvironmentinArangoDBissynchronous.ThismeansanymodulesthatdependonasynchronousbehaviourlikepromisesorsetTimeoutwillnotbehavecorrectlyinArangoDBorFoxx.AdditionallyunlikeNode.jsArangoDBdoesnotsupportnativeextensions.AllmoduleshavetobeimplementedinpureJavaScript.
WhileArangoDBprovidesalotofcompatibilitycodetosupportmoduleswrittenforNode.js,someNode.jsbuilt-inmodulescannotbeprovidedbyArangoDB.ForacloserlookattheNode.jsmodulesArangoDBdoesordoesnotprovidecheckouttheappendixonJavaScriptmodules.
Alsonotethattheserestrictionsnotonlyapplyonthemodulesyouwishtoinstallbutalsothedependenciesofthosemodules.Asaruleofthumb:moduleswrittentoworkinNode.jsandthebrowserthatdonotrelyonasyncbehaviourshouldgenerallywork;modulesthatrelyonnetworkorfilesystemI/Oormakeheavyuseofasyncbehaviourmostlikelywillnot.
Foxxdependencies
Foxxdependenciescanbedeclaredinaservice'smanifestusingtheprovidesanddependenciesfields:
providesliststhedependenciesagivenserviceprovides,i.e.whichAPIsitclaimstobecompatiblewith
dependenciesliststhedependenciesagivenserviceuses,i.e.whichAPIsitsdependenciesneedtobecompatiblewith
Adependencynameshouldgenerallyusethesameformatasanamespaced(org-scoped)NPMmodule,e.g.@foxx/sessions.
DependencynamesrefertotheexternalJavaScriptAPIofaserviceratherthanspecificservicesimplementingthoseAPIs.Somedependencynamesdefinedbyofficiallymaintainedservicesare:
@foxx/auth(version1.0.0)@foxx/api-keys(version1.0.0)@foxx/bugsnag(versions1.0.0and2.0.0)@foxx/mailgun(versions1.0.0and2.0.0)@foxx/postageapp(versions1.0.0and2.0.0)@foxx/postmark(versions1.0.0and2.0.0)@foxx/sendgrid(versions1.0.0and2.0.0)@foxx/oauth2(versions1.0.0and2.0.0)@foxx/segment-io(versions1.0.0and2.0.0)@foxx/sessions(versions1.0.0and2.0.0)@foxx/users(versions1.0.0,2.0.0and3.0.0)
Dependencies
199
Aprovidesdefinitionmapseachprovideddependency'snametotheprovidedversion:
"provides":{
"@foxx/auth":"1.0.0"
}
Adependenciesdefinitionmapsthelocalaliasofagivendependencyagainstitsnameandthesupportedversionrange(eitherasaJSONobjectorashorthandstring):
"dependencies":{
"mySessions":"@foxx/sessions:^2.0.0",
"myAuth":{
"name":"@foxx/auth",
"version":"^1.0.0",
"description":"Thisdescriptionisentirelyoptional.",
"required":false
}
}
Dependenciescanbeconfiguredfromthewebinterfaceinaservice'ssettingstabusingtheDependenciesbutton.
Thevalueforeachdependencyshouldbethedatabase-relativemountpathoftheservice(includingtheleadingslash).Inordertobeusableasthedependencyofanotherservicebothservicesneedtobemountedinthesamedatabase.Aservicecanbeusedtoprovidemultipledependenciesforthesameservice(aslongastheexpectedJavaScriptAPIsdon'tconflict).
Aservicethathasunconfiguredrequireddependenciescannotbeuseduntilallofitsdependencieshavebeenconfigured.
Itispossibletospecifythemountpathofaservicethatdoesnotactuallydeclarethedependencyasprovided.Thereiscurrentlynovalidationbeyondthemanifestformats.
Whenaserviceusesanothermountedserviceasadependencythedependency'smainentryfile'sexportsobjectbecomesavailableinthemodule.context.dependenciesobjectoftheotherservice:
Examples
ServiceAandServiceBaremountedinthesamedatabase.ServiceBhasadependencywiththelocalalias"greeter".ThedependencyisconfiguredtousethemountpathofServiceA.
//EntryfileofServiceA
module.exports={
sayHi(){
return'Hello';
}
};
//SomewhereinServiceB
constgreeter=module.context.dependencies.greeter;
res.write(greeter.sayHi());
Dependencies
200
RoutersconstcreateRouter=require('@arangodb/foxx/router');
RoutersletyoudefineroutesthatextendArangoDB'sHTTPAPIwithcustomendpoints.
RoutersneedtobemountedusingtheusemethodofaservicecontexttoexposetheirHTTProutesataservice'smountpath.
Youcanpassroutersbetweenservicesmountedinthesamedatabaseasdependencies.Youcanevennestrouterswithineachother.
CreatingaroutercreateRouter():Router
Thisreturnsanew,cleanrouterobjectthathasnotyetbeenmountedintheserviceandcanbeexportedlikeanyotherobject.
Requesthandlersrouter.get([path],[...middleware],handler,[name]):Endpoint
router.post([path],[...middleware],handler,[name]):Endpoint
router.put([path],[...middleware],handler,[name]):Endpoint
router.patch([path],[...middleware],handler,[name]):Endpoint
router.delete([path],[...middleware],handler,[name]):Endpoint
router.all([path],[...middleware],handler,[name]):Endpoint
Thesemethodsletyouspecifyroutesontherouter.TheallmethoddefinesaroutethatwillmatchanysupportedHTTPverb,theothermethodsdefineroutesthatonlymatchtheHTTPverbwiththesamename.
Arguments
path:string(Default:"/")
ThepathoftherequesthandlerrelativetothebasepaththeRouterismountedat.Ifomitted,therequesthandlerwillhandlerequeststothebasepathoftheRouter.Forinformationondefiningdynamicroutesseethesectiononpathparametersinthechapteronrouterendpoints.
middleware:Function(optional)
Zeroormoremiddlewarefunctionsthattakethefollowingarguments:
req:Request
Anincomingserverrequestobject.
res:Response
Anoutgoingserverresponseobject.
next:Function
Acallbackthatpassescontrolovertothenextmiddlewarefunctionandreturnswhenthatfunctionhascompleted.
Ifatruthyargumentispassed,thatargumentwillbethrownasanerror.
Ifthereisnonextmiddlewarefunction,thehandlerwillbeinvokedinstead(seebelow).
handler:Function
Afunctionthattakesthefollowingarguments:
req:Request
Anincomingserverrequestobject.
Routers
201
res:Response
Anoutgoingserverresponse.
name:string(optional)
AnamethatcanbeusedtogenerateURLsfortheendpoint.Formoreinformationseethereversemethodoftherequestobject.
ReturnsanEndpointfortheroute.
Examples
Simpleindexroute:
router.get(function(req,res){
res.set('content-type','text/plain');
res.write('HelloWorld!');
});
RestrictingaccesstoauthenticatedArangoDBusers:
router.get('/secrets',function(req,res,next){
if(req.arangoUser){
next();
}else{
res.throw(404,'Secrets?Whatsecrets?');
}
},function(req,res){
res.download('allOurSecrets.zip');
});
Multiplemiddlewarefunctions:
functioncounting(req,res,next){
if(!req.counter)req.counter=0;
req.counter++;
next();
req.counter--;
}
router.get(counting,counting,counting,function(req,res){
res.json({counter:req.counter});//{"counter":3}
});
Mountingchildroutersandmiddlewarerouter.use([path],middleware,[name]):Endpoint
Theusemethodletsyoumountachildrouterormiddlewareatagivenpath.
Arguments
path:string(optional)
ThepathofthemiddlewarerelativetothebasepaththeRouterismountedat.Ifomitted,themiddlewarewillhandlerequeststothebasepathoftheRouter.Forinformationondefiningdynamicroutesseethesectiononpathparametersinthechapteronrouterendpoints.
middleware:Router|Middleware
Anunmountedrouterobjectoramiddleware.
name:string(optional)
AnamethatcanbeusedtogenerateURLsforendpointsofthisrouter.Formoreinformationseethereversemethodoftherequestobject.HasnoeffectifhandlerisaMiddleware.
ReturnsanEndpointforthemiddlewareorchildrouter.
Routers
202
EndpointsEndpointsarereturnedbytheuse,allandHTTPverb(e.g.get,post)methodsofroutersaswellastheusemethodoftheservicecontext.Theycanbeusedtoattachmetadatatomountedroutes,middlewareandchildroutersthataffectshowrequestsandresponsesareprocessedorprovidesAPIdocumentation.
Endpointsshouldonlybeusedtoinvokethefollowingmethods.Endpointmethodscanbechainedtogether(eachmethodreturnstheendpointitself).
headerendpoint.header(name,[schema],[description]):this
Definesarequestheaderrecognizedbytheendpoint.Anyadditionalnon-definedheaderswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.
Iftheendpointisachildrouter,allroutesofthatrouterwillusethisheaderdefinitionunlessoverridden.
Arguments
name:string
Nameoftheheader.Thisshouldbeconsideredcaseinsensitiveasallheadernameswillbeconvertedtolowercase.
schema:Schema(optional)
Aschemadescribingtheformatoftheheadervalue.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.
Thevalueofthisheaderwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.
description:string(optional)
AhumanreadablestringthatwillbeshownintheAPIdocumentation.
Returnstheendpoint.
Examples
router.get(/*...*/)
.header('arangoVersion',joi.number().min(30000).default(30000));
pathParamendpoint.pathParam(name,[schema],[description]):this
Definesapathparameterrecognizedbytheendpoint.Pathparametersareexpectedtobefilledaspartoftheendpoint'smountpath.Anyadditionalnon-definedpathparameterswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.
Iftheendpointisachildrouter,allroutesofthatrouterwillusethisparameterdefinitionunlessoverridden.
Arguments
name:string
Nameoftheparameter.
schema:Schema(optional)
Aschemadescribingtheformatoftheparameter.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.
Endpoints
204
Thevalueofthisparameterwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultintheroutefailingtomatchandbeingignored(resultingina404(NotFound)errorresponseifnootherroutesmatch).
description:string(optional)
AhumanreadablestringthatwillbeshownintheAPIdocumentation.
Returnstheendpoint.
Examples
router.get('/some/:num/here',/*...*/)
.pathParam('num',joi.number().required());
queryParamendpoint.queryParam(name,[schema],[description]):this
Definesaqueryparameterrecognizedbytheendpoint.Anyadditionalnon-definedqueryparameterswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.
Iftheendpointisachildrouter,allroutesofthatrouterwillusethisparameterdefinitionunlessoverridden.
Arguments
name:string
Nameoftheparameter.
schema:Schema(optional)
Aschemadescribingtheformatoftheparameter.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.
Thevalueofthisparameterwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.
description:string(optional)
AhumanreadablestringthatwillbeshownintheAPIdocumentation.
Returnstheendpoint.
Examples
router.get(/*...*/)
.queryParam('num',joi.number().required());
bodyendpoint.body([model],[mimes],[description]):this
Definestherequestbodyrecognizedbytheendpoint.Therecanonlybeonerequestbodydefinitionperendpoint.ThedefinitionwillalsobeshownintheroutedetailsintheAPIdocumentation.
Intheabsenceofarequestbodydefinition,therequestobject'sbodypropertywillbeinitializedtotheunprocessedrawBodybuffer.
Iftheendpointisachildrouter,allroutesofthatrouterwillusethisbodydefinitionunlessoverridden.Iftheendpointisamiddleware,therequestbodywillonlybeparsedonce(i.e.theMIMEtypesoftheroutematchingthesamerequestwillbeignoredbutthebodywillstillbevalidatedagain).
Arguments
model:Model|Schema|null(optional)
Amodelorjoischemadescribingtherequestbody.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.
Endpoints
205
IfthevalueisamodelwithafromClientmethod,thatmethodwillbeappliedtotheparsedrequestbody.
Ifthevalueisaschemaoramodelwithaschema,theschemawillbeusedtovalidatetherequestbodyandthevaluepropertyofthevalidationresultoftheparsedrequestbodywillbeusedinsteadoftheparsedrequestbodyitself.
IfthevalueisamodeloraschemaandtheMIMEtypehasbeenomitted,theMIMEtypewilldefaulttoJSONinstead.
Ifthevalueisexplicitlysettonull,norequestbodywillbeexpected.
Ifthevalueisanarraycontainingexactlyonemodelorschema,therequestbodywillbetreatedasanarrayofitemsmatchingthatmodelorschema.
mimes:Array<string>(optional)
AnarrayofMIMEtypestheroutesupports.
Commonnon-mimealiaseslike"json"or"html"arealsosupportedandwillbeexpandedtotheappropriateMIMEtype(e.g."application/json"and"text/html").
IftheMIMEtypeisrecognizedbyFoxxtherequestbodywillbeparsedintotheappropriatestructurebeforebeingvalidated.CurrentlyonlyJSON,application/x-www-form-urlencodedandmultipartformatsaresupportedinthisway.
IftheMIMEtypeindicatedintherequestheadersdoesnotmatchanyofthesupportedMIMEtypes,thefirstMIMEtypeinthelistwillbeusedinstead.
Failuretoparsetherequestbodywillresultinanautomatic400(BadRequest)errorresponse.
description:string(optional)
AhumanreadablestringthatwillbeshownintheAPIdocumentation.
Returnstheendpoint.
Examples
router.post('/expects/some/json',/*...*/)
.body(
joi.object().required(),
'ThisimpliesJSON.'
);
router.post('/expects/nothing',/*...*/)
.body(null);//Nobodyallowed
router.post('/expects/some/plaintext',/*...*/)
.body(['text/plain'],'Thisbodywillbeastring.');
responseendpoint.response([status],[model],[mimes],[description]):this
Definesaresponsebodyfortheendpoint.Whenusingtheresponseobject'ssendmethodintherequesthandlerofthisroute,thedefinitionwiththematchingstatuscodewillbeusedtogeneratetheresponsebody.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.
Iftheendpointisachildrouter,allroutesofthatrouterwillusethisresponsedefinitionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.
Arguments
status:number|string(Default:200or204)
HTTPstatuscodetheresponseappliesto.Ifastringisprovidedinsteadofanumericstatuscodeitwillbeusedtolookupanumericstatuscodeusingthestatusesmodule.
model:Model|Schema|null(optional)
Amodelorjoischemadescribingtheresponsebody.
Endpoints
206
IfthevalueisamodelwithaforClientmethod,thatmethodwillbeappliedtothedatapassedtoresponse.sendwithintherouteiftheresponsestatuscodematches(butalsoifnostatuscodehasbeenset).
Ifthevalueisaschemaoramodelwithaschema,theactualschemawillnotbeusedtovalidatetheresponsebodyandonlyservestodocumenttheresponseinmoredetailintheAPIdocumentation.
IfthevalueisamodeloraschemaandtheMIMEtypehasbeenomitted,theMIMEtypewilldefaulttoJSONinstead.
Ifthevalueisexplicitlysettonullandthestatuscodehasbeenomitted,thestatuscodewilldefaultto204("nocontent")insteadof200.
Ifthevalueisanarraycontainingexactlyonemodelorschema,theresponsebodywillbeanarrayofitemsmatchingthatmodelorschema.
mimes:Array<string>(optional)
AnarrayofMIMEtypestheroutemightrespondwithforthisstatuscode.
Commonnon-mimealiaseslike"json"or"html"arealsosupportedandwillbeexpandedtotheappropriateMIMEtype(e.g."application/json"and"text/html").
Whenusingtheresponse.sendmethodtheresponsebodywillbeconvertedtotheappropriateMIMEtypeifpossible.
description:string(optional)
Ahuman-readablestringthatbrieflydescribestheresponseandwillbeshownintheendpoint'sdetaileddocumentation.
Returnstheendpoint.
Examples
//Thisexampleonlyprovidesdocumentation
//andimpliesagenericJSONresponsebody.
router.get(/*...*/)
.response(
joi.array().items(joi.string()),
'Alistofdoodadidentifiers.'
);
//Noresponsebodywillbeexpectedhere.
router.delete(/*...*/)
.response(null,'Thedoodadnolongerexists.');
//Anendpointcandefinemultipleresponsetypes
//fordifferentstatuscodes--butnevermorethan
//oneforeachstatuscode.
router.post(/*...*/)
.response('found','Thedoodadislocatedelsewhere.')
.response(201,['text/plain'],'Thedoodadwascreatedsohereisahaiku.');
//Heretheresponsebodywillbesetto
//thequerystring-encodedresultof
//FormModel.forClient({some:'data'})
//becausethestatuscodedefaultsto200.
router.patch(function(req,res){
//...
res.send({some:'data'});
})
.response(FormModel,['application/x-www-form-urlencoded'],'OMG.');
//Inthiscasetheresponsebodywillbesetto
//SomeModel.forClient({some:'data'})because
//thestatuscodehasbeensetto201before.
router.put(function(req,res){
//...
res.status(201);
res.send({some:'data'});
})
.response(201,SomeModel,'Somethingamazinghappened.');
error
Endpoints
207
endpoint.error(status,[description]):this
Documentsanerrorstatusfortheendpoint.
Iftheendpointisachildrouter,allroutesofthatrouterwillusethiserrordescriptionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.
ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.
Arguments
status:number|string
HTTPstatuscodefortheerror(e.g.400for"badrequest").Ifastringisprovidedinsteadofanumericstatuscodeitwillbeusedtolookupanumericstatuscodeusingthestatusesmodule.
description:string(optional)
Ahuman-readablestringthatbrieflydescribestheerrorconditionandwillbeshownintheendpoint'sdetaileddocumentation.
Returnstheendpoint.
Examples
router.get(function(req,res){
//...
res.throw(403,'Validationerroratx.y.z');
})
.error(403,'Indicatesthatavalidationhasfailed.');
summaryendpoint.summary(summary):this
Addsashortdescriptiontotheendpoint'sAPIdocumentation.
Iftheendpointisachildrouter,allroutesofthatrouterwillusethissummaryunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.
ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.
Arguments
summary:string
Ahuman-readablestringthatbrieflydescribestheendpointandwillappearnexttotheendpoint'spathinthedocumentation.
Returnstheendpoint.
Examples
router.get(/*...*/)
.summary('Listalldiscombobulateddoodads')
descriptionendpoint.description(description):this
Addsalongdescriptiontotheendpoint'sAPIdocumentation.
Iftheendpointisachildrouter,allroutesofthatrouterwillusethisdescriptionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.
ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.
Arguments
Endpoints
208
description:string
Ahuman-readablestringthatdescribestheendpointindetailandwillbeshownintheendpoint'sdetaileddocumentation.
Returnstheendpoint.
Examples
//The"dedent"libraryhelpsformatting
//multi-linestringsbyadjustingindentation
//andremovingleadingandtrailingblanklines
constdd=require('dedent');
router.post(/*...*/)
.description(dd`
Thisroutediscombobulatesthedoodadsby
frobnicatingthemoxieoftherequestbody.
`)
deprecatedendpoint.deprecated([deprecated]):this
Markstheendpointasdeprecated.
Iftheendpointisachildrouter,allroutesofthatrouterwillalsobemarkedasdeprecated.Iftheendpointisamiddleware,thismethodhasnoeffect.
ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.
Arguments
deprecated:boolean(Default:true)
Whethertheendpointshouldbemarkedasdeprecated.Ifsettofalsetheendpointwillbeexplicitlymarkedasnotdeprecated.
Returnstheendpoint.
Examples
router.get(/*...*/)
.deprecated();
tagendpoint.tag(...tags):this
MarkstheendpointwiththegiventagsthatwillbeusedtogrouprelatedroutesinthegeneratedAPIdocumentation.
Iftheendpointisachildrouter,allroutesofthatrouterwillalsobemarkedwiththetags.Iftheendpointisamiddleware,thismethodhasnoeffect.
ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.
Arguments
tags:string
Oneormorestringsthatwillbeusedtogrouptheendpoint'sroutes.
Returnstheendpoint.
Examples
router.get(/*...*/)
.tag('auth','restricted');
Endpoints
209
MiddlewareMiddlewareinFoxxreferstofunctionsthatcanbemountedlikeroutesandcanmanipulatetherequestandresponseobjectsbeforeandaftertherouteitselfisinvoked.Theycanalsobeusedtocontrolaccessortoprovidecommonlogiclikeloggingetc.Unlikeroutes,middlewareismountedwiththeusemethodlikearouter.
Insteadofafunctiontheusemethodcanalsoacceptanobjectwitharegisterfunctionthatwilltakeaparameterendpoint,themiddlewarewillbemountedatandreturnstheactualmiddlewarefunction.Thisallowsmanipulatingtheendpointbeforecreatingthemiddleware(e.g.todocumentheaders,requestbodies,pathparametersorqueryparameters).
Examples
RestrictaccesstoArangoDB-authenticatedusers:
module.context.use(function(req,res,next){
if(!req.arangoUser){
res.throw(401,'NotauthenticatedwithArangoDB');
}
next();
});
Anytruthyargumentpassedtothenextfunctionwillbethrownasanerror:
module.context.use(function(req,res,next){
leterr=null;
if(!req.arangoUser){
err=newError('Thisshouldneverhappen');
}
next(err);//throwsiftheerrorwasset
})
Trivialloggingmiddleware:
module.context.use(function(req,res,next){
conststart=Date.now();
try{
next();
}finally{
console.log(`Handledrequestin${Date.now()-start}ms`);
}
});
Morecomplexexampleforheader-basedsessions:
constsessions=module.context.collection('sessions');
module.context.use({
register(endpoint){
endpoint.header('x-session-id',joi.string().optional(),'ThesessionID.');
returnfunction(req,res,next){
constsid=req.get('x-session-id');
if(sid){
try{
req.session=sessions.document(sid);
}catch(e){
deletereq.headers['x-session-id'];
}
}
next();
if(req.session){
if(req.session._rev){
sessions.replace(req.session,req.session);
res.set('x-session-id',req.session._key);
}else{
constmeta=sessions.save(req.session);
res.set('x-session-id',meta._key);
}
Middleware
211
RequestobjectsTherequestobjectspecifiesthefollowingproperties:
arangoUser:string|null
TheauthenticatedArangoDBusernameusedtomaketherequest.ThisvalueisonlysetifauthenticationisenabledinArangoDBandtherequestsetanauthorizationheaderArangoDBwasabletoverify.YouarestronglyencouragedtoimplementyourownauthenticationlogicforyourownservicesbutthispropertycanbeusefulifyouneedtointegratewithArangoDB'sownauthenticationmechanisms.
arangoVersion:number
Thenumericvalueofthex-arango-versionheaderorthenumericversionoftheArangoDBserver(e.g.30102forversion3.1.2)ifnovalidheaderwasprovided.
baseUrl:string
Root-relativebaseURLoftheservice,i.e.theprefix"/_db/"followedbythevalueofdatabase.
body:any
Theprocessedandvalidatedrequestbodyforthecurrentroute.Ifnobodyhasbeendefinedforthecurrentroute,thevaluewillbeidenticaltorawBody.
FordetailsonhowrequestbodiescanbeprocessedandvalidatedbyFoxxseethebodymethodoftheendpointobject.
context:Context
Theservicecontextinwhichtherouterwasmounted(ratherthanthecontextinwhichtheroutewasdefined).
database:string
Thenameofthedatabaseinwhichtherequestisbeinghandled,e.g."_system".
headers:object
Therawheadersobject.
FordetailsonhowrequestheaderscanbevalidatedbyFoxxseetheheadermethodoftheendpointobject.
hostname:string
Thehostname(domainname)indicatedintherequestheaders.
Defaultstothehostnameportion(i.e.excludingtheport)oftheHostheaderandfallsbacktothelisteningaddressoftheserver.
method:string
TheHTTPverbusedtomaketherequest,e.g."GET".
originalUrl:string
Root-relativeURLoftherequest,i.e.pathfollowedbytherawqueryparameters,ifany.
path:string
Database-relativepathoftherequestURL(notincludingthequeryparameters).
pathParams:object
Anobjectmappingthenamesofpathparametersofthecurrentroutetotheirvalidatedvalues.
FordetailsonhowpathparameterscanbevalidatedbyFoxxseethepathParammethodoftheendpointobject.
port:number
Theportindicatedintherequestheaders.
Request
213
Defaultstotheportportion(i.e.excludingthehostname)oftheHostheaderandfallsbacktothelisteningportortheappropriatedefaultport(443forHTTPSor80forHTTP,dependingonsecure)iftheheaderonlyindicatesahostname.
Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettotheportportionoftheX-Forwarded-Hostheader(orapproriatedefaultport)ifpresent.
protocol:string
Theprotocolusedfortherequest.
Defaultsto"https"or"http"dependingonwhetherArangoDBisconfiguredtouseSSLornot.
Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothevalueoftheX-Forwarded-Protoheaderifpresent.
queryParams:object
Anobjectmappingthenamesofqueryparametersofthecurrentroutetotheirvalidatedvalues.
FordetailsonhowqueryparameterscanbevalidatedbyFoxxseethequeryParammethodoftheendpointobject.
rawBody:Buffer
Theraw,unparsed,unvalidatedrequestbodyasabuffer.
remoteAddress:string
TheIPoftheclientthatmadetherequest.
Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothefirstIPlistedintheX-Forwarded-Forheaderifpresent.
remoteAddresses:Array<string>
AlistcontainingtheIPaddressesusedtomaketherequest.
DefaultstothevalueofremoteAddresswrappedinanarray.
Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothelistofIPsspecifiedintheX-Forwarded-Forheaderifpresent.
remotePort:number
Thelisteningportoftheclientthatmadetherequest.
Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettotheportspecifiedintheX-Forwarded-Portheaderifpresent.
secure:boolean
Whethertherequestwasmadeoverasecureconnection(i.e.HTTPS).
Thisissettofalsewhenprotocolis"http"andtruewhenprotocolis"https".
suffix:string
Thetrailingpathrelativetothecurrentrouteifthecurrentrouteendsinawildcard(e.g./something/*).
Note:StartingwithArangoDB3.2ispassedintotheserviceas-is,i.e.percentageescapesequenceslike%2Fwillnolongerbeunescaped.Alsonotethatthesuffixmaycontainpathsegmentslike..whichmayhavespecialmeaningifthesuffixisusedtobuildfilesystempaths.
trustProxy:boolean
Indicateswhethertherequestwasmadeusingatrustedproxy.Iftheoriginserver'saddresswasspecifiedintheArangoDBconfigurationusing--frontend.trusted-proxyortheservice'strustProxysettingisenabled,thiswillbetrue,otherwiseitwillbefalse.
url:string
TheURLoftherequest.
Request
214
xhr:boolean
WhethertherequestindicatesitwasmadewithinabrowserusingAJAX.
ThisissettotrueiftheX-Requested-Withheaderispresentandisacase-insensitivematchforthevalue"xmlhttprequest".
NotethatthisvaluedoesnotguaranteewhethertherequestwasmadefrominsideabrowserorwhetherAJAXwasusedandismerelyaconventionestablishedbyJavaScriptframeworkslikejQuery.
acceptsreq.accepts(types):string|false
req.accepts(...types):string|false
req.acceptsCharsets(charsets):string|false
req.acceptsCharsets(...charsets):string|false
req.acceptsEncodings(encodings):string|false
req.acceptsEncodings(...encodings):string|false
req.acceptsLanguages(languages):string|false
req.acceptsLanguages(...languages):string|false
Thesemethodswrapthecorrespondingcontentnegotiationmethodsoftheacceptsmoduleforthecurrentrequest.
Examples
if(req.accepts(['json','html'])==='html'){
//ClientexplicitlyprefersHTMLoverJSON
res.write('<h1>ClientprefersHTML</h1>');
}else{
//OtherwisejustsendJSON
res.json({success:true});
}
cookiereq.cookie(name,options):string|null
Getsthevalueofacookiebyname.
Arguments
name:string
Nameofthecookie.
options:object(optional)
Anobjectwithanyofthefollowingproperties:
secret:string(optional)
Secretthatwasusedtosignthecookie.
Ifasecretisspecified,thecookie'ssignatureisexpectedtobepresentinasecondcookiewiththesamenameandthesuffix.sig.Otherwisethesignature(ifpresent)willbeignored.
algorithm:string(Default:"sha256")
Algorithmthatwasusedtosignthecookie.
Ifastringispassedinsteadofanoptionsobjectitwillbeinterpretedasthesecretoption.
Returnsthevalueofthecookieornullifthecookieisnotsetoritssignatureisinvalid.
Request
215
get/headerreq.get(name):string
req.header(name):string
Getsthevalueofaheaderbyname.Youcanvalidaterequestheadersusingtheheadermethodoftheendpoint.
Arguments
name:string
Nameoftheheader.
Returnstheheadervalue.
isreq.is(types):string
req.is(...types):string
Thismethodwrapsthe(requestbody)contenttypedetectionmethodofthetype-ismoduleforthecurrentrequest.
Examples
consttype=req.is('html','application/xml','application/*+xml');
if(type===false){//nomatch
handleDefault(req.rawBody);
}elseif(type==='html'){
handleHtml(req.rawBody);
}else{//isXML
handleXml(req.rawBody);
}
jsonreq.json():any
AttemptstoparsetherawrequestbodyasJSONandreturnstheresult.
Itisgenerallymoreusefultodefinearequestbodyontheendpointandusethereq.bodypropertyinstead.
Returnsundefinediftherequestbodyisempty.MaythrowaSyntaxErrorifthebodycouldnotbeparsed.
makeAbsolutereq.makeAbsolute(path,[query]):string
Resolvesthegivenpathrelativetothereq.context.service'smountpathtoafullURL.
Arguments
path:string
Thepathtoresovle.
query:string|object
AstringorobjectwithqueryparameterstoaddtotheURL.
ReturnstheformattedabsoluteURL.
paramsreq.param(name):any
Request
216
Arguments
Looksupaparameterbyname,preferringpathParamsoverqueryParams.
It'sprobablybetterstyletousethereq.pathParamsorreq.queryParamsobjectsdirectly.
name:string
Nameoftheparameter.
Returnsthe(validated)valueoftheparameter.
rangereq.range([size]):Ranges|number
Thismethodwrapstherangeheaderparsingmethodoftherange-parsermoduleforthecurrentrequest.
Arguments
size:number(Default:Infinity)
Lengthofthesatisfiablerange(e.g.numberofbytesinthefullresponse).Ifpresent,rangesexceedingthesizewillbeconsideredunsatisfiable.
ReturnsundefinediftheRangeheaderisabsent,-2iftheheaderispresentbutmalformed,-1iftherangeisinvalid(e.g.startoffsetislargerthanendoffset)orunsatisfiableforthegivensize.
Otherwisereturnsanarrayofobjectswiththepropertiesstartandendvaluesforeachrange.Thearrayhasanadditionalpropertytypeindicatingtherequestrangetype.
Examples
console.log(req.headers.range);//"bytes=40-80"
constranges=req.range(100);
console.log(ranges);//[{start:40,end:80}]
console.log(ranges.type);//"bytes"
reversereq.reverse(name,[params]):string
LooksuptheURLofanamedrouteforthegivenparameters.
Arguments
name:string
Nameoftheroutetolookup.
params:object(optional)
Anobjectcontainingvaluesforthe(pathorquery)parametersoftheroute.
ReturnstheURLoftherouteforthegivenparameters.
Examples
router.get('/items/:id',function(req,res){
/*...*/
},'getItemById');
router.post('/items',function(req,res){
//...
consturl=req.reverse('getItemById',{id:createdItem._key});
res.set('location',req.makeAbsolute(url));
});
Request
217
ResponseobjectsTheresponseobjectspecifiesthefollowingproperties:
body:Buffer|string
Responsebodyasastringorbuffer.Canbesetdirectlyorusingsomeoftheresponsemethods.
context:Context
Theservicecontextinwhichtherouterwasmounted(ratherthanthecontextinwhichtheroutewasdefined).
headers:object
Therawheadersobject.
statusCode:number
Statuscodeoftheresponse.Defaultsto200(bodysetandnotanemptystringorbuffer)or204(otherwise)ifnotchangedfromundefined.
attachmentres.attachment([filename]):this
Setsthecontent-dispositionheadertoindicatetheresponseisadownloadablefilewiththegivenname.
Note:Thisdoesnotactuallymodifytheresponsebodyoraccessthefilesystem.TosendafilefromthefilesystemseethedownloadorsendFilemethods.
Arguments
filename:string(optional)
Nameofthedownloadablefileintheresponsebody.
Ifpresent,theextensionofthefilenamewillbeusedtosettheresponsecontent-typeifithasnotyetbeenset.
Returnstheresponseobject.
cookieres.cookie(name,value,[options]):this
Setsacookiewiththegivenname.
Arguments
name:string
Nameofthecookie.
value:string
Valueofthecookie.
options:object(optional)
Anobjectwithanyofthefollowingproperties:
ttl:number(optional)
Timetoliveofthecookieinseconds.
algorithm:string(Default:"sha256")
Algorithmthatwillbeusedtosignthecookie.
Response
219
secret:string(optional)
Secretthatwillbeusedtosignthecookie.
Ifasecretisspecified,thecookie'ssignaturewillbestoredinasecondcookiewiththesameoptions,thesamenameandthesuffix.sig.Otherwisenosignaturewillbeadded.
path:string(optional)
Pathforwhichthecookieshouldbeissued.
domain:string(optional)
Domainforwhichthecookieshouldbeissued.
secure:boolean(Default:false)
Whetherthecookieshouldbemarkedassecure(i.e.HTTPS/SSL-only).
httpOnly:boolean(Default:false)
WhetherthecookieshouldbemarkedasHTTP-only(ratherthanalsoexposingittoclient-sidecode).
Ifastringispassedinsteadofanoptionsobjectitwillbeinterpretedasthesecretoption.
Ifanumberispassedinsteadofanoptionsobjectitwillbeinterpretedasthettloption.
Returnstheresponseobject.
downloadres.download(path,[filename]):this
Theequivalentofcallingres.attachment(filename).sendFile(path).
Arguments
path:string
Pathtothefileonthelocalfilesystemtobesentastheresponsebody.
filename:string(optional)
Filenametoindicateinthecontent-dispositionheader.
Ifomittedthepathwillbeusedinstead.
Returnstheresponseobject.
getHeaderres.getHeader(name):string
Getsthevalueoftheheaderwiththegivenname.
Arguments
name:string
Nameoftheheadertoget.
Returnsthevalueoftheheaderorundefined.
jsonres.json(data):this
SetstheresponsebodytotheJSONstringvalueofthegivendata.
Response
220
Arguments
data:any
Thedatatobeusedastheresponsebody.
Returnstheresponseobject.
redirectres.redirect([status],path):this
Redirectstheresponsebysettingtheresponselocationheaderandstatuscode.
Arguments
status:number|string(optional)
Responsestatuscodetoset.
Ifthestatuscodeisthestringvalue"permanent"itwillbetreatedasthevalue301.
Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.
Ifthestatuscodeisomittedbuttheresponsestatushasnotalreadybeenset,theresponsestatuswillbesetto302.
path:string
URLtosetthelocationheaderto.
Returnstheresponseobject.
removeHeaderres.removeHeader(name):this
Removestheheaderwiththegivennamefromtheresponse.
Arguments
name:string
Nameoftheheadertoremove.
Returnstheresponseobject.
sendres.send(data,[type]):this
Setstheresponsebodytothegivendatawithrespecttotheresponsedefinitionfortheresponse'scurrentstatuscode.
Arguments
data:any
Thedatatobeusedastheresponsebody.Willbeconvertedaccordingtheresponsedefinitionfortheresponse'scurrentstatuscode(or200)inthefollowingway:
IfthedataisanArangoDBresultset,itwillbeconvertedtoanarrayfirst.
IftheresponsedefinitionspecifiesamodelwithaforClientmethod,thatmethodwillbeappliedtothedatafirst.Ifthedataisanarrayandtheresponsedefinitionhasthemultipleflagset,themethodwillbeappliedtoeachentryindividuallyinstead.
Finallythedatawillbeprocessedbytheresponsetypehandlertoconvertheresponsebodytoastringorbuffer.
type:string(Default:"auto")
Response
221
Content-typeoftheresponsebody.
Ifsetto"auto"thefirstMIMEtypespecifiedintheresponsedefinitionfortheresponse'scurrentstatuscode(or200)willbeusedinstead.
Ifsetto"auto"andnoresponsedefinitionexists,theMIMEtypewillbedeterminedthefollowingway:
IfthedataisabuffertheMIMEtypewillbesettobinary(application/octet-stream).
IfthedataisanobjecttheMIMEtypewillbesettoJSONandthedatawillbeconvertedtoaJSONstring.
OtherwisetheMIMEtypewillbesettoHTMLandthedatawillbeconvertedtoastring.
Returnstheresponseobject.
sendFileres.sendFile(path,[options]):this
Sendsafilefromthelocalfilesystemastheresponsebody.
Arguments
path:string
Pathtothefileonthelocalfilesystemtobesentastheresponsebody.
Ifnocontent-typeheaderhasbeensetyet,theextensionofthefilenamewillbeusedtosetthevalueofthatheader.
options:object(optional)
Anobjectwithanyofthefollowingproperties:
lastModified:boolean(optional)
Ifsettotrueorifnolast-modifiedheaderhasbeensetyetandthevalueisnotsettofalsethelast-modifiedheaderwillbesettothemodificationdateofthefileinmilliseconds.
Returnstheresponseobject.
Examples
//Sendthefile"favicon.ico"fromthisservice'sfolder
res.sendFile(module.context.fileName('favicon.ico'));
sendStatusres.sendStatus(status):this
Sendsaplaintextresponseforthegivenstatuscode.Theresponsestatuswillbesettothegivenstatuscode,theresponsebodywillbesettothestatusmessagecorrespondingtothatstatuscode.
Arguments
status:number|string
Responsestatuscodetoset.
Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.
Returnstheresponseobject.
setHeader/setres.setHeader(name,value):this
res.set(name,value):this
Response
222
res.set(headers):this
Setsthevalueoftheheaderwiththegivenname.
Arguments
name:string
Nameoftheheadertoset.
value:string
Valuetosettheheaderto.
headers:object
Headerobjectmappingheadernamestovalues.
Returnstheresponseobject.
statusres.status(status):this
Setstheresponsestatustothegivenstatuscode.
Arguments
status:number|string
Responsestatuscodetoset.
Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.
Returnstheresponseobject.
throwres.throw(status,[reason],[options]):void
ThrowsanHTTPexceptionforthegivenstatus,whichwillbehandledbyFoxxtoservetheappropriateJSONerrorresponse.
Arguments
status:number|string
Responsestatuscodetoset.
Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.
Ifthestatuscodeisinthe500-range(500-599),itsstacktracewillalwaysbeloggedasifitwereanunhandledexception.
Ifdevelopmentmodeisenabled,theerror'sstacktracewillbeloggedasawarningifthestatuscodeisinthe400-range(400-499)orasaregularmessageotherwise.
reason:string(optional)
Messagefortheexception.
Ifomitted,thestatusmessagecorrespondingtothestatuscodewillbeusedinstead.
options:object(optional)
Anobjectwithanyofthefollowingproperties:
cause:Error(optional)
Causeoftheexceptionthatwillbeloggedaspartoftheerror'sstacktrace(recursively,iftheexceptionalsohasacausepropertyandsoon).
Response
223
extra:object(optional)
AdditionalpropertiesthatwillbeaddedtotheerrorresponsebodygeneratedbyFoxx.
Ifdevelopmentmodeisenabled,anexceptionpropertywillbeaddedtothisvaluecontainingtheerrormessageandastacktracepropertywillbeaddedcontaininganarraywitheachlineoftheerror'sstacktrace.
Ifanerrorispassedinsteadofanoptionsobjectitwillbeinterpretedasthecauseoption.Ifnoreasonwasprovidedtheerror'smessagewillbeusedasthereasoninstead.
Returnsnothing.
typeres.type([type]):string
Setstheresponsecontent-typetothegiventypeifprovidedorreturnsthepreviouslysetcontent-type.
Arguments
type:string(optional)
Content-typeoftheresponsebody.
Unlikeres.set('content-type',type)fileextensionscanbeprovidedasvaluesandwillbetranslatedtothecorrespondingMIMEtype(e.g.jsonbecomesapplication/json).
Returnsthecontent-typeoftheresponsebody.
varyres.vary(names):this
res.vary(...names):this
Thismethodwrapsthevaryheadermanipulationmethodofthevarymoduleforthecurrentresponse.
Thegivennameswillbeaddedtotheresponse'svaryheaderifnotalreadypresent.
Returnstheresponseobject.
Examples
res.vary('user-agent');
res.vary('cookie');
res.vary('cookie');//duplicateswillbeignored
//--or--
res.vary('user-agent','cookie');
//--or--
res.vary(['user-agent','cookie']);
writeres.write(data):this
Appendsthegivendatatotheresponsebody.
Arguments
data:string|Buffer
Datatoappend.
Response
224
Ifthedataisabuffertheresponsebodywillbeconvertedtoabufferfirst.
Iftheresponsebodyisabufferthedatawillbeconvertedtoabufferfirst.
IfthedataisanobjectitwillbeconvertedtoaJSONstringfirst.
Ifthedataisanyothernon-stringvalueitwillbeconvertedtoastringfirst.
Returnstheresponseobject.
Response
225
UsingGraphQLinFoxxconstcreateGraphQLRouter=require('@arangodb/foxx/graphql');
Foxxbundlesversion0.6ofthegraphql-syncmodule,whichisasynchronouswrapperfortheofficialJavaScriptGraphQLreferenceimplementation,toallowwritingGraphQLschemasdirectlyinsideFoxx.
Additionallythe@arangodb/foxx/graphqlletsyoucreateroutersforservingGraphQLrequests,whichcloselymimicksthebehaviouroftheexpress-graphqlmodule.
Formoreinformationongraphql-syncseethegraphql-jsAPIreference(notethatgraphql-syncneverwrapsresultsinpromises).
ForanexampleofaGraphQLschemainFoxxthatresolvesfieldsusingthedatabaseseetheGraphQLexampleservice(alsoavailablefromtheFoxxstore).
Examples
constgraphql=require('graphql-sync');
constgraphqlSchema=newgraphql.GraphQLSchema({
//...
});
//Mountingagraphqlendpointdirectlyinaservice:
module.context.use('/graphql',createGraphQLRouter({
schema:graphqlSchema,
graphiql:true
}));
//Orattheservice'srootURL:
module.context.use(createGraphQLRouter({
schema:graphqlSchema,
graphiql:true
}));
//Orinsideanexistingrouter:
router.get('/hello',function(req,res){
res.write('Helloworld!');
});
router.use('/graphql',createGraphQLRouter({
schema:graphqlSchema,
graphiql:true
}));
Note:ArangoDBaimsforstabilitywhichmeansbundleddependencieswillgenerallynotbeupdatedasquicklyastheirmaintainersmakeupdatesavailableonGitHuborNPM.StartingwithArangoDB3.2,ifyouwanttouseanewerversionthantheonebundledwithyourtargetversionofArangoDB,youcanprovideyourownversionofthelibrarybypassingitviathegraphqloption:
constgraphql=require('graphql-sync');
constgraphqlSchema=newgraphql.Schema({
//...
});
module.context.use(createGraphQLRouter({
schema:graphqlSchema,
graphiql:true,
graphql:graphql
}))
Startingwithgraphql0.12youcanalsousetheofficialgraphqllibraryifyouincludeitinthenode_modulesfolderofyourservicebundle:
constgraphql=require('graphql');//0.12orlater
constgraphqlSchema=newgraphql.Schema({
//...
});
module.context.use(createGraphQLRouter({
schema:graphqlSchema,
graphiql:true,
UsingGraphQL
226
graphql:graphql
}))
CreatingaroutercreateGraphQLRouter(options):Router
ThisreturnsanewrouterobjectwithPOSTandGETroutesforservingGraphQLrequests.
Arguments
options:object
Anobjectwithanyofthefollowingproperties:
schema:GraphQLSchema
AGraphQLSchemaobjectfromgraphql-sync.
context:any(optional)
TheGraphQLcontextthatwillbepassedtothegraphql()functionfromgraphql-synctohandleGraphQLqueries.
rootValue:object(optional)
TheGraphQLrootvaluethatwillbepassedtothegraphql()functionfromgraphql-synctohandleGraphQLqueries.
pretty:boolean(Default:false)
Iftrue,JSONresponseswillbepretty-printed.
formatError:Function(optional)
Afunctionthatwillbeusedtoformaterrorsproducedbygraphql-sync.Ifomitted,theformatErrorfunctionfromgraphql-syncwillbeusedinstead.
validationRules:Array<any>(optional)
AdditionalvalidationrulesqueriesmustsatisfyinadditiontothosedefinedintheGraphQLspec.
graphiql:boolean(Default:false)
Iftrue,theGraphiQLexplorerwillbeservedwhenloadeddirectlyfromabrowser.
graphql:object(optional)
Ifyouneedtouseyourowncopyofthegraphql-syncmoduleinsteadoftheonebundledwithArangoDB,hereyoucanpassitindirectly.
IfaGraphQLSchemaobjectispassedinsteadofanoptionsobjectitwillbeinterpretedastheschemaoption.
Generatedroutes
TherouterhandlesGETandPOSTrequeststoitsrootpathandacceptsthefollowingparameters,whichcanbeprovidedeitherasqueryparametersorasthePOSTrequestbody:
query:string
AGraphQLquerythatwillbeexecuted.
variables:object|string(optional)
AnobjectorastringcontainingaJSONobjectwithruntimevaluestouseforanyGraphQLqueryvariables.
operationName:string(optional)
Iftheprovidedquerycontainsmultiplenamedoperations,thisspecifieswhichoperationshouldbeexecuted.
raw:boolean(Default:false)
UsingGraphQL
227
ForcesaJSONresponseevenifgraphiqlisenabledandtherequestwasmadeusingabrowser.
ThePOSTrequestbodycanbeprovidedasJSONorasquerystringusingapplication/x-www-form-urlencoded.Arequestbodypassedasapplication/graphqlwillbeinterpretedasthequeryparameter.
UsingGraphQL
228
SessionMiddlewareconstsessionMiddleware=require('@arangodb/foxx/sessions');
ThesessionmiddlewareaddsthesessionandsessionStoragepropertiestotherequestobjectanddealswithserializinganddeserializingthesessionaswellasextractingsessionidentifiersfromincomingrequestsandinjectingthemintooutgoingresponses.
Examples
//Createasessionmiddleware
constsessions=sessionsMiddleware({
storage:module.context.collection('sessions'),
transport:['header','cookie']
});
//Firstenablethemiddlewareforthisservice
module.context.use(sessions);
//Nowmounttheroutersthatusethesession
constrouter=createRouter();
module.context.use(router);
router.get('/',function(req,res){
res.send(`Hello${req.session.uid||'anonymous'}!`);
},'hello');
router.post('/login',function(req,res){
req.session.uid=req.body;
req.sessionStorage.save(req.session);
res.redirect(req.reverse('hello'));
});
.body(['text/plain'],'Username');
CreatingasessionmiddlewaresessionMiddleware(options):Middleware
Createsasessionmiddleware.
Arguments
options:Object
Anobjectwiththefollowingproperties:
storage:Storage
Storagethatwillbeusedtopersistthesessions.
ThestorageisalsoexposedasthesessionStorageonallrequestobjectsandasthestoragepropertyofthemiddleware.
IfastringorcollectionispassedinsteadofaStorage,itwillbeusedtocreateaCollectionStorage.
transport:Transport|Array<Transport>
Transportorarrayoftransportsthatwillbeusedtoextractthesessionidentifiersfromincomingrequestsandinjectthemintooutgoingresponses.Whenattemptingtoextractasessionidentifier,thetransportswillbeusedintheorderspecifieduntilamatchisfound.Wheninjecting(orclearing)sessionidentifiers,alltransportswillbeinvoked.
Thetransportsarealsoexposedasthetransportpropertyofthemiddleware.
Ifthestring"cookie"ispassedinsteadofaTransport,theCookieTransportwillbeusedwiththedefaultsettingsinstead.
Ifthestring"header"ispassedinsteadofaTransport,theHeaderTransportwillbeusedwiththedefaultsettingsinstead.
autoCreate:boolean(Default:true)
Ifenabledthesessionstorage'snewmethodwillbeinvokedtocreateanemptysessionwheneverthetransportfailedtoreturnasessionfortheincomingrequest.Otherwisethesessionwillbeinitializedasnull.
Sessionsmiddleware
229
SessionStoragesSessionstoragesareusedbythesessionsmiddlewaretopersistsessionsacrossrequests.SessionstoragesmustimplementthefromClientandforClientmethodsandcanoptionallyimplementthenewmethod.
Thebuilt-insessionstoragesgenerallyprovidethefollowingattributes:
uid:string(Default:null)
Auniqueidentifierindicatingtheactiveuser.
created:number(Default:Date.now())
Thenumerictimestampofwhenthesessionwascreated.
data:any(Default:null)
Arbitrarydatatopersistedinthesession.
newstorage.new():Session
Generatesanewsessionobjectrepresentinganemptysession.Theemptysessionobjectshouldnotbepersistedunlessnecessary.Thereturnvaluewillbeexposedbythemiddlewareasthesessionpropertyoftherequestobjectifnosessionidentifierwasreturnedbythesessiontransportsandauto-creationisnotexplicitlydisabledinthesessionmiddleware.
Examples
new(){
return{
uid:null,
created:Date.now(),
data:null
};
}
fromClientstorage.fromClient(sid):Session|null
Resolvesordeserializesasessionidentifiertoasessionobject.
Arguments
sid:string
Sessionidentifiertoresolveordeserialize.
Returnsasessionobjectrepresentingthesessionwiththegivensessionidentifierthatwillbeexposedbythemiddlewareasthesessionpropertyoftherequestobject.Thismethodwillonlybecalledifanyofthesessiontransportsreturnedasessionidentifier.Ifthesessionidentifierisinvalidorexpired,themethodshouldreturnanullvaluetoindicatenomatchingsession.
Examples
fromClient(sid){
returndb._collection('sessions').firstExample({_key:sid});
}
forClientstorage.forClient(session):string|null
Sessionstorages
231
Derivesasessionidentifierfromthegivensessionobject.
Arguments
session:Session
Sessiontoderiveasessionidentifierfrom.
Returnsasessionidentifierforthesessionrepresentedbythegivensessionobject.Thismethodwillbecalledwiththesessionpropertyoftherequestobjectunlessthatpropertyisempty(e.g.null).
Examples
forClient(session){
if(!session._key){
constmeta=db._collection('sessions').save(session);
returnmeta._key;
}
db._collection('sessions').replace(session._key,session);
returnsession._key;
}
Sessionstorages
232
CollectionSessionStorageconstcollectionStorage=require('@arangodb/foxx/sessions/storages/collection');
Thecollectionsessionstoragepersistssessionstoacollectioninthedatabase.
CreatingastoragecollectionStorage(options):Storage
CreatesaStoragethatcanbeusedinthesessionsmiddleware.
Arguments
options:Object
Anobjectwiththefollowingproperties:
collection:ArangoCollection
Thecollectionthatshouldbeusedtopersistthesessions.Ifastringispassedinsteadofacollectionitisassumedtobethefullyqualifiednameofacollectioninthecurrentdatabase.
ttl:number(Default:60*60)
Thetimeinsecondssincethelastupdateuntilasessionwillbeconsideredexpired.
pruneExpired:boolean(Default:false)
Whetherexpiredsessionsshouldberemovedfromthecollectionwhentheyareaccessedinsteadofsimplybeingignored.
autoUpdate:boolean(Default:true)
Whethersessionsshouldbeupdatedinthecollectioneverytimetheyareaccessedtokeepthemfromexpiring.Disablingthisoptionwillimproveperformancebutmeansyouwillhavetotakecareofkeepingyoursessionsaliveyourself.
Ifastringorcollectionispassedinsteadofanoptionsobject,itwillbeinterpretedasthecollectionoption.
prunestorage.prune():Array<string>
Removesallexpiredsessionsfromthecollection.ThismethodshouldbecalledevenifthepruneExpiredoptionisenabledtocleanupabandonedsessions.
Returnsanarrayofthekeysofallsessionsthatwereremoved.
savestorage.save(session):Session
Saves(replaces)thegivensessionobjectinthecollection.Thismethodneedstobeinvokedexplicitlyaftermakingchangestothesessionorthechangeswillnotbepersisted.Assignsanew_keytothesessionifitpreviouslydidnothaveone.
Arguments
session:Session
Asessionobject.
Returnsthemodifiedsession.
clear
Sessionstorages
233
storage.clear(session):boolean
Removesthesessionfromthecollection.Hasnoeffectifthesessionwasalreadyremovedorhasnotyetbeensavedtothecollection(i.e.hasno_key).
Arguments
session:Session
Asessionobject.
Returnstrueifthesessionwasremovedorfalseifithadnoeffect.
Sessionstorages
234
JWTSessionStorageconstjwtStorage=require('@arangodb/foxx/sessions/storages/jwt');
TheJWTsessionstorageconvertssessionstoandfromJSONWebTokens.
Examples
//PassinasecuresecretfromtheFoxxconfiguration
constsecret=module.context.configuration.jwtSecret;
constsessions=sessionsMiddleware({
storage:jwtStorage(secret),
transport:'header'
});
module.context.use(sessions);
CreatingastoragejwtStorage(options):Storage
CreatesaStoragethatcanbeusedinthesessionsmiddleware.
Note:whilethe"none"algorithm(i.e.nosignature)issupportedthisdummyalgorithmprovidesnosecurityandallowsclientstomakearbitrarymodificationstothepayloadandshouldnotbeusedunlessyouarecertainyouspecificallyneedit.
Arguments
options:Object
Anobjectwiththefollowingproperties:
algorithm:string(Default:"HS512")
Thealgorithmtouseforsigningthetoken.
Supportedvalues:
"HS256"(HMAC-SHA256)"HS384"(HMAC-SHA384)"HS512"(HMAC-SHA512)"none"(nosignature)
secret:string
Thesecrettouseforsigningthetoken.
Thisfieldisforbiddenwhenusingthe"none"algorithmbutrequiredotherwise.
ttl:number(Default:3600)
Themaximumlifetimeofthetokeninseconds.Youmaywanttokeepthisshortasanewtokenisgeneratedoneveryrequestallowingclientstorefreshtokensautomatically.
verify:boolean(Default:true)
Ifsettofalsethesignaturewillnotbeverifiedbutstillgenerated(unlessusingthe"none"algorithm).
maxExp:number(Default:Infinity)
LargestvaluethatwillbeacceptedinanincomingJWTexp(expiration)field.
Ifastringispassedinsteadofanoptionsobjectitwillbeinterpretedasthesecretoption.
Sessionstorages
235
SessionTransportsSessiontransportsareusedbythesessionsmiddlewaretostoreandretrievesessionidentifiersinrequestsandresponses.Sessiontransportsmustimplementthegetand/orsetmethodsandcanoptionallyimplementtheclearmethod.
gettransport.get(request):string|null
Retrievesasessionidentifierfromarequestobject.
Ifpresentthismethodwillautomaticallybeinvokedforeachtransportuntilatransportreturnsasessionidentifier.
Arguments
request:Request
Requestobjecttoextractasessionidentifierfrom.
Returnsthesessionidentifierornullifthetransportcannotfindasessionidentifierintherequest.
Examples
get(req){
returnreq.get('x-session-id')||null;
}
settransport.set(response,sid):void
Attachesasessionidentifiertoaresponseobject.
Ifpresentthismethodwillautomaticallybeinvokedattheendofarequestregardlessofwhetherthesessionwasmodifiedornot.
Arguments
response:Response
Responseobjecttoattachasessionidentifierto.
sid:string
Sessionidentifiertoattachtotheresponse.
Returnsnothing.
Examples
set(res){
res.set('x-session-id',value);
}
cleartransport.clear(response):void
Attachesapayloadindicatingthatthesessionhasbeenclearedtotheresponseobject.Thiscanbeusedtoclearasessioncookiewhenthesessionhasbeendestroyed(e.g.duringlogout).
Ifpresentthismethodwillautomaticallybeinvokedinsteadofsetwhenthereq.sessionattributewasremovedbytheroutehandler.
Arguments
Sessiontransports
236
response:Response
Responseobjecttoremovethesessionidentifierfrom.
Returnsnothing.
Sessiontransports
237
CookieSessionTransportconstcookieTransport=require('@arangodb/foxx/sessions/transports/cookie');
Thecookietransportstoressessionidentifiersincookiesontherequestandresponseobject.
Examples
//PassinasecuresecretfromtheFoxxconfiguration
constsecret=module.context.configuration.cookieSecret;
constsessions=sessionsMiddleware({
storage:module.context.collection('sessions'),
transport:cookieTransport({
name:'FOXXSESSID',
ttl:60*60*24*7,//oneweekinseconds
algorithm:'sha256',
secret:secret
})
});
module.context.use(sessions);
CreatingatransportcookieTransport([options]):Transport
CreatesaTransportthatcanbeusedinthesessionsmiddleware.
Arguments
options:Object(optional)
Anobjectwiththefollowingproperties:
name:string(Default:"sid")
Thenameofthecookie.
ttl:number(optional)
Cookielifetimeinseconds.NotethatthisdoesnotaffectthestorageTTL(i.e.howlongthesessionitselfisconsideredvalid),justhowlongthecookieshouldbestoredbytheclient.
algorithm:string(optional)
Thealgorithmusedtosignandverifythecookie.Ifnoalgorithmisspecified,thecookiewillnotbesignedorverified.Seethecookiemethodontheresponseobject.
secret:string(optional)
Secrettouseforthesignedcookie.Willbeignoredifnoalgorithmisprovided.
path:string(optional)
Pathforwhichthecookieshouldbeissued.
domain:string(optional)
Domainforwhichthecookieshouldbeissued.
secure:boolean(Default:false)
Whetherthecookieshouldbemarkedassecure(i.e.HTTPS/SSL-only).
httpOnly:boolean(Default:false)
WhetherthecookieshouldbemarkedasHTTP-only(ratherthanalsoexposingittoclient-sidecode).
Ifastringispassedinsteadofanoptionsobject,itwillbeinterpretedasthenameoption.
Sessiontransports
238
HeaderSessionTransportconstheaderTransport=require('@arangodb/foxx/sessions/transports/header');
Theheadertransportstoressessionidentifiersinheadersontherequestandresponseobjects.
Examples
constsessions=sessionsMiddleware({
storage:module.context.collection('sessions'),
transport:headerTransport('X-FOXXSESSID')
});
module.context.use(sessions);
CreatingatransportheaderTransport([options]):Transport
CreatesaTransportthatcanbeusedinthesessionsmiddleware.
Arguments
options:Object(optional)
Anobjectwiththefollowingproperties:
name:string(Default:X-Session-Id)
Nameoftheheaderthatcontainsthesessionidentifier(notcasesensitive).
Ifastringispassedinsteadofanoptionsobject,itwillbeinterpretedasthenameoption.
Sessiontransports
240
StaticfileassetsThemostflexiblewaytoservefilesinyourFoxxserviceistosimplypassthemthroughinyourrouterusingthecontextobject'sfileNamemethodandtheresponseobject'ssendFilemethod:
router.get('/some/filename.png',function(req,res){
constfilePath=module.context.fileName('some-local-filename.png');
res.sendFile(filePath);
});
Whileallowingforgreatercontrolofhowthefileshouldbesenttotheclientandwhoshouldbeabletoaccessit,doingthisforallyourstaticassetscangettedious.
AlternativelyyoucanspecifyfileassetsthatshouldbeservedbyyourFoxxservicedirectlyintheservicemanifestusingthefilesattribute:
"files":{
"/some/filename.png":{
"path":"some-local-filename.png",
"type":"image/png",
"gzip":false
},
"/favicon.ico":"bookmark.ico",
"/static":"my-assets-folder"
}
Eachentryinthefilesattributecanrepresenteitherasinglefileoradirectory.Whenservingentiredirectories,thekeyactsasaprefixandrequeststothatprefixwillberesolvedwithinthegivendirectory.
Options
path:string
Therelativepathofthefileorfolderwithintheservice.
type:string(optional)
TheMIMEcontenttypeofthefile.Defaultstoanintelligentguessbasedonthefilename'sextension.
gzip:boolean(Default:false)
Ifsettotruethefilewillbeservedwithgzip-encodingifsupportedbytheclient.Thiscanbeusefulwhenservingtextfileslikeclient-sideJavaScript,CSSorHTML.
Ifastringisprovidedinsteadofanobject,itwillbeinterpretedasthepathoption.
Servingfiles
241
WritingtestsFoxxprovidesoutoftheboxsupportforrunningtestsagainstaninstalledserviceusingtheMochatestrunner.
TestfileshavefullaccesstotheservicecontextandallArangoDBAPIsbutlikescriptscannotdefineFoxxroutes.
Runningtests
Aninstalledservice'stestscanbeexecutedfromtheadministrativewebinterface:
1. Openthe"Services"tabofthewebinterface2. Clickontheinstalledservicetobetested3. Clickonthe"Settings"tab4. Clickontheflaskiconinthetopright5. Accepttheconfirmationdialog
Notethatrunningtestsinaproductiondatabaseisnotrecommendedandmayresultindatalossifthetestsaccessthedatabase.
Whenrunningaserviceindevelopmentmodespecialcareneedstobetakenasperformingrequeststotheservice'sownroutesaspartofthetestsuitesmayresultintestsbeingexecutedwhilethedatabaseisinaninconsistentstate,leadingtounexpectedbehaviour.
TestfilepathsInordertotellFoxxaboutfilescontainingtestsuites,oneormorepatternsneedtobespecifiedinthetestsoptionoftheservicemanifest:
{
"tests":[
"**/test_*.js",
"**/*_test.js"
]
}
Thesepatternscanbeeitherrelativefilepathsor"globstar"patternswhere
*matcheszeroormorecharactersinafilename**matcheszeroormorenesteddirectories
Forexample,giventhefollowingdirectorystructure:
++test/
|++a/
||+-a1.js
||+-a2.js
||+-test.js
|+-b.js
|+-c.js
|+-d_test.js
+-e_test.js
+-test.js
Thefollowingpatternswouldmatchthefollowingfiles:
test.js:
test.js
test/*.js:
/test/b.js
/test/c.js
/test/d_test.js
Writingtests
242
test/**/*.js:
/test/a/a1.js
/test/a/a2.js
/test/a/test.js
/test/b.js
/test/c.js
/test/d_test.js
**/test.js:
/test/a/test.js
**/*test.js:
/test/a/test.js
/test/d_test.js
/e_test.js
/test.js
Evenifmultiplepatternsmatchthesamefilethetestsinthatfilewillonlyberunonce.
Theorderoftestsisalwaysdeterminedbythefilepaths,nottheorderinwhichtheyarematchedorspecifiedinthemanifest.
TeststructureMochatestsuitescanbedefinedusingoneofthreeinterfaces:BDD,TDDorExports.
TheQUnitinterfaceofMochaisnotsupportedinArangoDB.
LikeallArangoDBcode,testcodeisalwayssynchronous.
BDDinterface
TheBDDinterfacedefinestestsuitesusingthedescribefunctionandeachtestcaseisdefinedusingtheitfunction:
'usestrict';
constassert=require('assert');
consttrueThing=true;
describe('Truethings',()=>{
it('aretrue',()=>{
assert.equal(trueThing,true);
});
});
TheBDDinterfacealsooffersthealiascontextfordescribeandspecifyforit.
Testfixturescanbehandledusingbeforeandafterforsuite-widefixturesandbeforeEachandafterEachforper-testfixtures:
describe('Falsethings',()=>{
letfalseThing;
before(()=>{
falseThing=!true;
});
it('arefalse',()=>{
assert.equal(falseThing,false);
});
});
TDDinterface
TheTDDinterfacedefinestestsuitesusingthesuitefunctionandeachtestcaseisdefinedusingthetestfunction:
'usestrict';
constassert=require('assert');
consttrueThing=true;
suite('Truethings',()=>{
test('aretrue',()=>{
Writingtests
243
assert.equal(trueThing,true);
});
});
TestfixturescanbehandledusingsuiteSetupandsuiteTeardownforsuite-widefixturesandsetupandteardownforper-testfixtures:
suite('Falsethings',()=>{
letfalseThing;
suiteSetup(()=>{
falseThing=!true;
});
test('arefalse',()=>{
assert.equal(falseThing,false);
});
});
Exportsinterface
TheExportsinterfacedefinestestcasesasmethodsofplainobjectpropertiesofthemodule.exportsobject:
'usestrict';
constassert=require('assert');
consttrueThing=true;
exports['Truethings']={
'aretrue':function(){
assert.equal(trueThing,true);
}
};
Thekeysbefore,after,beforeEachandafterEacharespecial-casedandbehavelikethecorrespondingfunctionsintheBDDinterface:
letfalseThing;
exports['Falsethings']={
before(){
falseThing=false;
},
'arefalse':function(){
assert.equal(falseThing,false);
}
};
Assertions
ArangoDBprovidestwobundledmodulestodefineassertions:
assertcorrespondstotheNode.jsassertmodule,providinglow-levelassertionsthatcanoptionallyspecifyanerrormessage.
chaiisthepopularChaiAssertionLibrary,providingbothBDDandTDDstyleassertionsusingafamiliarsyntax.
Writingtests
244
Cross-OriginResourceSharing(CORS)TouseCORSinyourFoxxservicesyoufirstneedtoconfigureArangoDBforCORS.Asof3.2Foxxwillthenautomaticallywhitelistallresponseheadersastheyareused.
IfyouwantmorecontroloverthewhitelistorareusinganolderversionofArangoDByoucansetthefollowingresponseheadersinyourrequesthandler:
access-control-expose-headers:acomma-separatedlistofresponseheaders.Thisdefaultstoalistofallheaderstheresponseisactuallyusing(butnotincludinganyaccess-controlheaders).
access-control-allow-credentials:canbesetto"false"toforbidexposingcookies.ThedefaultvaluedependsonwhetherArangoDBtruststheorigin.Seethenotesonhttp.trusted-origin.
NotethatitisnotpossibletooverridetheseheadersfortheCORSpreflightresponse.Itisthereforenotpossibletoacceptcredentialsorcookiesonlyforindividualroutes,servicesordatabases.TheoriginneedstobetrustedaccordingtothegeneralArangoDBconfiguration(seeabove).
CrossOrigin
245
FoxxscriptsandqueuedjobsFoxxletsyoudefinescriptsthatcanbeexecutedaspartoftheinstallationandremovalprocess,invokedmanuallyorscheduledtorunatalatertimeusingthejobqueue.
Toregisteryourscript,justaddascriptssectiontoyourservicemanifest:
{
...
"scripts":{
"setup":"scripts/setup.js",
"send-mail":"scripts/send-mail.js"
}
...
}
Thescriptsyoudefineinyourservicemanifestcanbeinvokedfromthewebinterfaceintheservice'ssettingspagewiththeScriptsdropdown.
Youcanalsousethescriptsasqueuedjobs:
'usestrict';
constqueues=require('@arangodb/foxx/queues');
queues.get('default').push(
{mount:'/my-service-mount-point',name:'send-mail'},
{to:'[email protected]',body:'Hello'}
);
ScriptargumentsandreturnvaluesIfthescriptwasinvokedwithanyarguments,youcanaccessthemusingthemodule.context.argvarray.
Toreturndatafromyourscript,youcanassignthedatatomodule.exportsasusual.PleasenotethatthisdatawillbeconvertedtoJSON.
Anyerrorsraisedbythescriptwillbehandleddependingonhowthescriptwasinvoked:
ifthescriptwasinvokedfromtheHTTPAPI(e.g.usingthewebinterface),itwillreturnanerrorresponseusingtheexception'sstatusCodepropertyifspecifiedor500.ifthescriptwasinvokedfromaFoxxjobqueue,thejob'sfailurecounterwillbeincrementedandthejobwillberescheduledormarkedasfailedifnoattemptsremain.
Examples
Let'ssayyouwanttodefineascriptthattakestwonumericvaluesandreturnstheresultofmultiplyingthem:
'usestrict';
constassert=require('assert');
constargv=module.context.argv;
assert.equal(argv.length,2,'Expectedexactlytwoarguments');
assert.equal(typeofargv[0],'number','Expectedfirstargumenttobeanumber');
assert.equal(typeofargv[1],'number','Expectedsecondargumenttobeanumber');
module.exports=argv[0]*argv[1];
LifecycleScriptsFoxxrecognizeslifecyclescriptsiftheyaredefinedandwillinvokethemduringtheinstallation,updateandremovalprocessoftheserviceifyouwant.
Scriptsandqueuedjobs
246
Thefollowingscriptsarecurrentlyrecognizedaslifecyclescriptsbytheirname:"setup"and"teardown".
SetupScript
ThesetupscriptwillbeexecutedwithoutargumentsduringtheinstallationofyourFoxxservice.
Thesetupscriptmaybeexecutedmorethanonceandshouldthereforebetreatedasreentrant.Runningthesamesetupscriptagainshouldnotresultinanyerrorsorduplicatedata.
Thesetupscriptistypicallyusedtocreatecollectionsyourserviceneedsorinsertseeddatalikeinitialadministrativeuseraccountsandsoon.
Examples
'usestrict';
constdb=require('@arangodb').db;
consttextsCollectionName=module.context.collectionName('texts');
//`textsCollectionName`isnowtheprefixednameofthisservice's"texts"collection.
//e.g."example_texts"iftheservicehasbeenmountedat`/example`
if(db._collection(textsCollectionName)===null){
constcollection=db._create(textsCollectionName);
collection.save({text:'entry1fromcollectiontexts'});
collection.save({text:'entry2fromcollectiontexts'});
collection.save({text:'entry3fromcollectiontexts'});
}else{
console.debug(`collection${texts}alreadyexists.Leavingituntouched.`);
}
TeardownScript
TheteardownscriptwillbeexecutedwithoutargumentsduringtheremovalofyourFoxxservice.
Itcanalsooptionallybeexecutedbeforeupgradinganservice.
Thisscripttypicallyremovesthecollectionsand/ordocumentscreatedbyyourservice'ssetupscript.
Examples
'usestrict';
constdb=require('@arangodb').db;
consttextsCollection=module.context.collection('texts');
if(textsCollection){
textsCollection.drop();
}
Queuesconstqueues=require('@arangodb/foxx/queues')
Foxxallowsdefiningjobqueuesthatletyouperformsloworexpensiveactionsasynchronously.Thesequeuescanbeusedtosende-mails,callexternalAPIsorperformotheractionsthatyoudonotwanttoperformdirectlyorwanttoretryonfailure.
enableordisabletheFoxxqueuesfeature--foxx.queuesflag
Iftrue,theFoxxqueueswillbeavailableandjobsinthequeueswillbeexecutedasynchronously.
Thedefaultistrue.Whensettofalsethequeuemanagerwillbedisabledandanyjobsarepreventedfrombeingprocessed,whichmayreduceCPUloadabit.PleasenotethatFoxxjobqueuesaredatabase-specific.Queuesandjobsarealwaysrelativetothedatabaseinwhichtheyarecreatedoraccessed.
pollintervalforFoxxqueues--foxx.queues-poll-intervalvalue
Scriptsandqueuedjobs
247
ThepollintervalfortheFoxxqueuesmanager.Thevalueisspecifiedinseconds.LowervalueswillmeanmoreimmediateandmorefrequentFoxxqueuejobexecution,butwillmakethequeuethreadwakeupandquerythequeuesmoreoften.Whensettoalowvalue,thequeuethreadmightcauseCPUload.
Thedefaultis1second.IfFoxxqueuesarenotusedmuch,thenthisvaluemaybeincreasedtomakethequeuesthreadwakeupless.Forthelow-levelfunctionalityseethechapteronthetaskmanagementmodule.
Creatingorupdatingaqueue
queues.create(name,[maxWorkers]):Queue
Returnsthequeueforthegivenname.Ifthequeuedoesnotexist,anewqueuewiththegivennamewillbecreated.IfaqueuewiththegivennamealreadyexistsandmaxWorkersisset,thequeue'smaximumnumberofworkerswillbeupdated.Thequeuewillbecreatedinthecurrentdatabase.
Arguments
name:string
Nameofthequeuetocreate.
maxWorkers:number(Default:1)
Themaximumnumberofworkers.
Examples
//Createaqueuewiththedefaultnumberofworkers(i.e.one)
constqueue1=queues.create("my-queue");
//Createaqueuewithagivennumberofworkers
constqueue2=queues.create("another-queue",2);
//Updatethenumberofworkersofanexistingqueue
constqueue3=queues.create("my-queue",10);
//queue1andqueue3refertothesamequeue
assertEqual(queue1,queue3);
Fetchinganexistingqueue
queues.get(name):Queue
Returnsthequeueforthegivenname.Ifthequeuedoesnotexistanexceptionisthrowninstead.
Thequeuewillbelookedupinthecurrentdatabase.
Arguments
name:string
Nameofthequeuetofetch.
Examples
Ifthequeuedoesnotyetexistanexceptionisthrown:
queues.get("some-queue");
//Error:Queuedoesnotexist:some-queue
//at...
Otherwisethequeuewillbereturned:
constqueue1=queues.create("some-queue");
constqueue2=queues.get("some-queue");
assertEqual(queue1,queue2);
Deletingaqueuequeues.delete(name):boolean
Scriptsandqueuedjobs
248
Returnstrueifthequeuewasdeletedsuccessfully.Ifthequeuedidnotexist,itreturnsfalseinstead.Thequeuewillbelookedupanddeletedinthecurrentdatabase.
Whenaqueueisdeleted,jobsonthatqueuewillnolongerbeexecuted.
Deletingaqueuewillnotdeleteanyjobsonthatqueue.
Arguments
name:string
Nameofthequeuetodelete.
Examples
constqueue=queues.create("my-queue");
queues.delete("my-queue");//true
queues.delete("my-queue");//false
Addingajobtoaqueue
queue.push(script,data,[opts]):string
Thejobwillbeaddedtothespecifiedqueueinthecurrentdatabase.
Returnsthejobid.
Arguments
script:object
Ajobtypedefinition,consistingofanobjectwiththefollowingproperties:
name:string
Nameofthescriptthatwillbeinvoked.
mount:string
Mountpathoftheservicethatdefinesthescript.
backOff:Function|number(Default:1000)
Eitherafunctionthattakesthenumberoftimesthejobhasfailedbeforeasinputandreturnsthenumberofmillisecondstowaitbeforetryingthejobagain,orthedelaytobeusedtocalculateanexponentialback-off,or0fornodelay.
maxFailures:number|Infinity(Default:0):
Numberoftimesasinglerunofajobwillbere-triedbeforeitismarkedas"failed".AnegativevalueorInfinitymeansthatthejobwillbere-triedonfailureindefinitely.
schema:Schema(optional)
Schematovalidateajob'sdataagainstbeforeenqueuingthejob.
preprocess:Function(optional)
Functiontopre-processajob's(validated)databeforeserializingitinthequeue.
data:any
Jobdataofthejob;mustbeserializabletoJSON.
opts:object(optional)
Objectwithanyofthefollowingproperties:
success:Function(optional)
Functiontobecalledafterthejobhasbeencompletedsuccessfully.
Scriptsandqueuedjobs
249
failure:Function(optional)
Functiontobecalledafterthejobhasfailedtoomanytimes.
delayUntil:number|Date(Default:Date.now())
Timestampinmilliseconds(orDateinstance)untilwhichtheexecutionofthejobshouldbedelayed.
backOff:Function|number(Default:1000)
Seescript.backOff.
maxFailures:number|Infinity(Default:0):
Seescript.maxFailures.
repeatTimes:number|Function(Default:0)
Ifsettoapositivenumber,thejobwillberepeatedthismanytimes(notcountingrecoverywhenusingmaxFailures).IfsettoanegativenumberorInfinity,thejobwillberepeatedindefinitely.Ifsetto0thejobwillnotberepeated.
repeatUntil:number|Date(optional)
Ifthejobissettoautomaticallyrepeat,thiscanbesettoatimestampinmilliseconds(orDateinstance)afterwhichthejobwillnolongerrepeat.Settingthisvaluetozero,anegativevalueorInfinityhasnoeffect.
repeatDelay:number(Default:0)
Ifthejobissettoautomaticallyrepeat,thiscanbesettoanon-negativevaluetosetthenumberofmillisecondsforwhichthejobwillbedelayedbeforeitisstartedagain.
NotethatifyoupassafunctionforthebackOffcalculation,successcallbackorfailurecallbackoptionsthefunctionwillbeserializedtothedatabaseasastringandthereforemustnotrelyonanyexternalscopeorexternalvariables.
Whenthejobissettoautomaticallyrepeat,thefailurecallbackwillonlybeexecutedwhenarunofthejobhasfailedmorethanmaxFailurestimes.NotethatifthejobfailsandmaxFailuresisset,itwillberescheduledaccordingtothebackOffuntilithaseitherfailedtoomanytimesorcompletedsuccessfullybeforebeingscheduledaccordingtotherepeatDelayagain.RecoveryattemptsbymaxFailuresdonotcounttowardsrepeatTimes.
Thesuccessandfailurecallbacksreceivethefollowingarguments:
result:any
Thereturnvalueofthescriptforthecurrentrunofthejob.
jobData:any
Thedatapassedtothismethod.
job:object
ArangoDBdocumentrepresentingthejob'scurrentstate.
Examples
Let'ssaywehaveanservicemountedat/mailerthatprovidesascriptcalledsend-mail:
'usestrict';
constqueues=require('@arangodb/foxx/queues');
constqueue=queues.create('my-queue');
queue.push(
{mount:'/mailer',name:'send-mail'},
{to:'[email protected]',body:'Helloworld'}
);
Thiswillnotwork,becauselogwasdefinedoutsidethecallbackfunction(thecallbackmustbeserializabletoastring):
//WARNING:THISDOESNOTWORK!
'usestrict';
constqueues=require('@arangodb/foxx/queues');
Scriptsandqueuedjobs
250
constqueue=queues.create('my-queue');
constlog=require('console').log;//outsidethecallback'sfunctionscope
queue.push(
{mount:'/mailer',name:'send-mail'},
{to:'[email protected]',body:'Helloworld'},
{success:function(){
log('Yay!');//throws'logisnotdefined'
}}
);
Here'sanexampleofajobthatwillbeexecutedevery5secondsuntiltomorrow:
'usestrict';
constqueues=require('@arangodb/foxx').queues;
constqueue=queues.create('my-queue');
queue.push(
{mount:'/mailer',name:'send-mail'},
{to:'[email protected]',body:'Helloworld'},
{
repeatTimes:Infinity,
repeatUntil:Date.now()+(24*60*60*1000),
repeatDelay:5*1000
}
);
Fetchingajobfromthequeue
queue.get(jobId):Job
Createsaproxyobjectrepresentingajobwiththegivenjobid.
Thejobwillbelookedupinthespecifiedqueueinthecurrentdatabase.
ReturnsthejobforthegivenjobId.Propertiesofthejobobjectwillbefetchedwhenevertheyarereferencedandcannotbemodified.
Arguments
jobId:string
Theidofthejobtocreateaproxyobjectfor.
Examples
constjobId=queue.push({mount:'/logger',name:'log'},'HelloWorld!');
constjob=queue.get(jobId);
assertEqual(job.id,jobId);
Deletingajobfromthequeue
queue.delete(jobId):boolean
Deletesajobwiththegivenjobid.Thejobwillbelookedupanddeletedinthespecifiedqueueinthecurrentdatabase.
Arguments
jobId:string
Theidofthejobtodelete.
Returnstrueifthejobwasdeletedsuccessfully.Ifthejobdidnotexistitreturnsfalseinstead.
Fetchinganarrayofjobsinaqueue
Examples
constlogScript={mount:'/logger',name:'log'};
queue.push(logScript,'HelloWorld!',{delayUntil:Date.now()+50});
assertEqual(queue.pending(logScript).length,1);
//50mslater...
Scriptsandqueuedjobs
251
assertEqual(queue.pending(logScript).length,0);
assertEqual(queue.progress(logScript).length,1);
//evenlater...
assertEqual(queue.progress(logScript).length,0);
assertEqual(queue.complete(logScript).length,1);
Fetchinganarrayofpendingjobsinaqueuequeue.pending([script]):Array<string>
Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"pending",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.
Arguments
script:object(optional)
Anobjectwiththefollowingproperties:
name:stringNameofthescript.
mount:stringMountpathoftheservicedefiningthescript.
Fetchinganarrayofjobsthatarecurrentlyinprogress
queue.progress([script])
Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"progress",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.
Arguments
script:object(optional)
Anobjectwiththefollowingproperties:
name:stringNameofthescript.
mount:stringMountpathoftheservicedefiningthescript.
Fetchinganarrayofcompletedjobsinaqueuequeue.complete([script]):Array<string>
Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"complete",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.
Arguments
script:object(optional)
Anobjectwiththefollowingproperties:
name:stringNameofthescript.
mount:stringMountpathoftheservicedefiningthescript.
Fetchinganarrayoffailedjobsinaqueue
queue.failed([script]):Array<string>
Scriptsandqueuedjobs
252
Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"failed",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.
Arguments
script:object(optional)
Anobjectwiththefollowingproperties:
name:stringNameofthescript.
mount:stringMountpathoftheservicedefiningthescript.
Fetchinganarrayofalljobsinaqueue
queue.all([script]):Array<string>
Returnsanarrayofjobidsofalljobsinthegivenqueue,optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.
Arguments
script:object(optional)
Anobjectwiththefollowingproperties:
name:stringNameofthescript.
mount:stringMountpathoftheservicedefiningthescript.
Abortingajob
job.abort():void
Abortsanon-completedjob.
Setsajob'sstatusto"failed"ifitisnotalready"complete",withoutcallingthejob'sonFailurecallback.
Scriptsandqueuedjobs
253
Migrating2.xservicesto3.0WhenmigratingservicesfromolderversionsofArangoDBitisgenerallyrecommendedyoumakesuretheyworkinlegacycompatibilitymode,whichcanalsoserveasastop-gapsolution.
ThischapteroutlinesthemajordifferencesintheFoxxAPIbetweenArangoDB2.8andArangoDB3.0.
Generalchanges
TheconsoleobjectinlaterversionsofArangoDB2.ximplementedaspecialFoxxconsoleAPIandwouldoptionallylogmessagestoacollection.ArangoDB3.0restorestheoriginalbehaviourwhereconsoleisthesameobjectavailablefromtheconsolemodule.
Migrating2.xservices
254
Migratingfrompre-2.8WhenmigratingfromaversionolderthanArangoDB2.8pleasenotethatstartingwithArangoDB2.8thebehaviouroftherequirefunctionmorecloselymimicsthebehaviourobservedinNode.jsandmodulebundlersforbrowsers,e.g.:
Inafile/routes/examples.js(relativetotherootfolderoftheservice):
require('./my-module')willbeattemptedtoberesolvedinthefollowingorder:
1. /routes/my-module(relativetoserviceroot)2. /routes/my-module.js(relativetoserviceroot)3. /routes/my-module.json(relativetoserviceroot)4. /routes/my-module/index.js(relativetoserviceroot)5. /routes/my-module/index.json(relativetoserviceroot)
require('lodash')willbeattemptedtoberesolvedinthefollowingorder:
1. /routes/node_modules/lodash(relativetoserviceroot)2. /node_modules/lodash(relativetoserviceroot)3. ArangoDBmodulelodash4. Nodecompatibilitymodulelodash5. BundledNPMmodulelodash
require('/abs/path')willbeattemptedtoberesolvedinthefollowingorder:
1. /abs/path(relativetofilesystemroot)2. /abs/path.js(relativetofilesystemroot)3. /abs/path.json(relativetofilesystemroot)4. /abs/path/index.js(relativetofilesystemroot)5. /abs/path/index.json(relativetofilesystemroot)
ThisbehaviourisincompatiblewiththesourcecodegeneratedbytheFoxxgeneratorinthewebinterfacebeforeArangoDB2.8.
Note:Theorg/arangodbmoduleisaliasedtothenewname@arangodbinArangoDB3.0.0andthe@arangodbmodulewasaliasedtotheoldnameorg/arangodbinArangoDB2.8.0.Eitheronewillworkin2.8and3.0butoutsideoflegacyservicesyoushoulduse@arangodbgoingforward.
FoxxqueueInArangoDB2.6FoxxintroducedanewwaytodefinequeuedjobsusingFoxxscriptstoreplacethefunction-basedjobtypedefinitionswhichwerecausingproblemswhenrestartingtheserver.Thefunction-basedjobshavebeenremovedin2.7andarenolongersupportedatall.
CoffeeScriptArangoDB3.0nolongerprovidesbuilt-insupportforCoffeeScriptsourcefiles,eveninlegacycompatibilitymode.IfyouwanttouseanalternativelanguagelikeCoffeeScript,makesuretopre-compiletherawsourcefilestoJavaScriptandusethecompiledJavaScriptfilesintheservice.
TherequestmoduleThe@arangodb/requestmodulewhenusedwiththejsonoptionpreviouslyoverwrotethestringinthebodypropertyoftheresponseobjectoftheresponsewiththeparsedJSONbody.In2.8thiswaschangedsotheparsedJSONbodyisaddedasthejsonpropertyoftheresponseobjectinadditiontooverwritingthebodyproperty.In3.0andlater(includinglegacycompatibilitymode)thebodypropertyisnolongeroverwrittenandmustusethejsonpropertyinstead.Notethatthisonlyaffectscodeusingthejsonoptionwhenmakingtherequest.
Migratingfrompre-2.8
255
BundledNPMmodules
ThebundledNPMmoduleshavebeenupgradedandmayincludebackwards-incompatiblechanges,especiallytheAPIofjoihaschangedseveraltimes.Ifindoubtyoushouldbundleyourownversionsofthesemodulestoensurespecificversionswillbeused.
Theutilitymodulelodashisnowavailableandshouldbeusedinsteadofunderscore,butbothmoduleswillcontinuetobeprovided.
Migratingfrompre-2.8
256
ManifestManyofthefieldsthatwererequiredinArangoDB2.xarenowoptionalandcanbesafelyomitted.
ToavoidcompatibilityproblemswithfutureversionsofArangoDByoushouldalwaysspecifytheenginesfield,e.g.:
{
"engines":{
"arangodb":"^3.0.0"
}
}
Controllers&exports
PreviouslyFoxxdistinguishedbetweenexportsandcontrollers,eachofwhichcouldbespecifiedasanobject.InArangoDB3.0thesehavebeenmergedintoasinglemainfieldspecifyinganentryfile.
Theeasiestwaytomigrateservicesusingmultipleexportsand/orcontrollersistocreateaseparateentryfilethatimportsthesefiles:
Old(manifest.json):
{
"exports":{
"doodads":"doodads.js",
"dingbats":"dingbats.js"
},
"controllers":{
"/doodads":"routes/doodads.js",
"/dingbats":"routes/dingbats.js",
"/":"routes/root.js"
}
}
New(manifest.json):
{
"main":"index.js"
}
New(index.js):
'usestrict';
module.context.use('/doodads',require('./routes/doodads'));
module.context.use('/dingbats',require('./routes/dingbats'));
module.context.use('/',require('./routes/root'));
module.exports={
doodads:require('./doodads'),
dingbats:require('./dingbats')
};
IndexredirectIfyoupreviouslydidnotdefinethedefaultDocumentfield,pleasenotethatinArangoDB3.0thefieldwillnolongerdefaulttothevalueindex.htmlwhenomitted:
Old:
{
//nodefaultDocument
}
manifest.json
257
New:
{
"defaultDocument":"index.html"
}
ThisalsomeansitisnolongernecessarytospecifythedefaultDocumentfieldwithanemptyvaluetopreventtheredirectandbeabletoserverequestsatthe/(root)pathofthemountpoint:
Old:
{
"defaultDocument":""
}
New:
{
//nodefaultDocument
}
AssetsTheassetsfieldisnolongersupportedinArangoDB3.0outsideoflegacycompatibilitymode.
Ifyoupreviouslyusedthefieldtoserveindividualfilesas-isyoucansimplyusethefilesfieldinstead:
Old:
{
"assets":{
"client.js":{
"files":["assets/client.js"],
"contentType":"application/javascript"
}
}
}
New:
{
"files":{
"client.js":{
"path":"assets/client.js",
"type":"application/javascript"
}
}
}
Ifyoureliedonbeingabletospecifymultiplefilesthatshouldbeconcatenated,youwillhavetousebuildtoolsoutsideofArangoDBtopreparethesefilesaccordingly.
RootelementTherootElementfieldisnolongersupportedandhasbeenremovedentirely.
IfyourcontrollersreliedonthisfieldbeingavailableyouneedtoadjustyourschemasandroutestobeabletohandlethefullJSONstructureofincomingdocuments.
Systemservices
manifest.json
258
TheisSystemfieldisnolongersupported.ThepresenceorabsenceofthefieldhadnoeffectinmostrecentversionsofArangoDB2.xandhasnowbeenremovedentirely.
manifest.json
259
TheapplicationcontextTheglobalapplicationContextvariableavailableinFoxxmoduleshasbeenreplacedwiththecontextattributeofthemodulevariable.Forconsistencyitisnowreferredtoastheservicecontextthroughoutthisdocumentation.
SomemethodsoftheservicecontexthavechangedinArangoDB3.0:
fileName()nowbehaveslikepath()didinArangoDB2.xpath()hasbeenremoved(usefileName()instead)foxxFileName()hasbeenremoved(usefileName()instead)
Additionallytheversionandnameattributeshavebeenremovedandcannowonlybeaccessedviathemanifestattribute(asmanifest.versionandmanifest.name).Notethatthecorrespondingmanifestfieldsarenowoptionalandmaybeomitted.
Theoptionsattributehasalsobeenremovedasitshouldbeconsideredanimplementationdetail.Youshouldinsteadaccessthedependenciesandconfigurationattributesdirectly.
Theinternal_prefixattribute(whichwasanaliasforbasePath)andtheinternalcommentandclearCommentsmethods(whichwereusedbythemagicaldocumentationcommentsinArangoDB2.x)havealsobeenremoved.
Theinternal_serviceattribute(whichprovidesaccesstotheserviceitself)hasbeenrenamedtoservice.
applicationContext
260
RepositoriesandmodelsPreviouslyFoxxwasheavilybuiltaroundtheconceptofrepositoriesandmodels,whichprovidedcomplexbutrarelynecessaryabstractionsontopofArangoDBcollectionsanddocuments.InArangoDB3.0thesehavebeenremovedentirely.
Repositoriesvscollections
RepositoriesmostlywrappedmethodsthatalreadyexistedonArangoDBcollectionobjectsandprimarilydealtwithconvertingbetweenplainArangoDBdocumentsandFoxxmodelinstances.InArangoDB3.0youcansimplyusethesecollectionsdirectlyandtreatdocumentsasplainJavaScriptobjects.
Old:
'usestrict';
constFoxx=require('org/arangodb/foxx');
constmyRepo=newFoxx.Repository(
applicationContext.collection('myCollection'),
{model:Foxx.Model}
);
//...
constmodels=myRepo.byExample({color:'green'});
res.json(models.map(function(model){
returnmodel.forClient();
}));
New:
'usestrict';
constmyDocs=module.context.collection('myCollection');
//...
constdocs=myDocs.byExample({color:'green'});
res.json(docs);
Schemavalidation
ThemainpurposeofmodelsinArangoDB2.xwastovalidateincomingdatausingjoischemas.InmorerecentversionsofArangoDB2.xitwasalreadypossibletopasstheseschemasdirectlyinmostplaceswhereamodelwasexpectedasanargument.Theonlydifferenceisthatschemasshouldnowbeconsideredthedefault.
IfyoupreviouslyreliedontheautomaticvalidationofFoxxmodelinstanceswhensettingattributesorinstantiatingmodelsfromuntrusteddata,youcansimplyusetheschema'svalidatemethoddirectly.
Old:
'usestrict';
constjoi=require('joi');
constmySchema={
name:joi.string().required(),
size:joi.number().required()
};
constFoxx=require('org/arangodb/foxx');
constMyModel=Foxx.Model.extend({schema:mySchema});
//...
constmodel=newMyModel(req.json());
if(!model.isValid){
res.status(400);
RepositoriesandModels
261
res.write('Badrequest');
return;
}
New:
'usestrict';
constjoi=require('joi');
//Notethisisnowwrappedinajoi.object()
constmySchema=joi.object({
name:joi.string().required(),
size:joi.number().required()
}).required();
//...
constresult=mySchema.validate(req.body);
if(result.errors){
res.status(400);
res.write('Badrequest');
return;
}
Migratingmodels
Whilemostusecasesformodelscannowbereplacedwithplainjoischemas,thereisstilltheconceptofa"model"inFoxxinArangoDB3.0althoughitisquitedifferentfromFoxxmodelsinArangoDB2.x.
AmodelinFoxxnowreferstoaplainJavaScriptobjectwithanoptionalschemaattributeandtheoptionalmethodsforClientandfromClient.Modelscanbeusedinsteadofplainjoischemastodefinerequestandresponsebodiesbuttherearenomodel"instances"inArangoDB3.0.
Old:
'usestrict';
const_=require('underscore');
constjoi=require('joi');
constFoxx=require('org/arangodb/foxx');
constMyModel=Foxx.Model.extend({
schema:{
name:joi.string().required(),
size:joi.number().required()
},
forClient(){
return_.omit(this.attributes,['_key','_id','_rev']);
}
});
//...
ctrl.get(/*...*/)
.bodyParam('body',{type:MyModel});
New:
'usestrict';
const_=require('lodash');
constjoi=require('joi');
constMyModel={
schema:joi.object({
name:joi.string().required(),
size:joi.number().required()
}).required(),
forClient(data){
return_.omit(data,['_key','_id','_rev']);
}
};
RepositoriesandModels
262
//...
router.get(/*...*/)
.body(MyModel);
TriggersWhensaving,updating,replacingordeletingmodelsinArangoDB2.xusingtherepositorymethodstherepositoryandmodelwouldfireeventsthatcouldbesubscribedtoinordertoperformside-effects.
Notethatevenin2.xtheseeventswouldnotfirewhenusingqueriesormanipulatingdocumentsinanyotherwaythanusingthespecificrepositorymethodsthatoperatedonindividualdocuments.
ThisbehaviourisnolongeravailableinArangoDB3.0butcanbeemulatedbyusinganEventEmitterdirectlyifitisnotpossibletosolvetheproblemdifferently:
Old:
'usestrict';
constFoxx=require('org/arangodb/foxx');
constMyModel=Foxx.Model.extend({
//...
},{
afterRemove(){
console.log(this.get('name'),'wasremoved');
}
});
//...
constmodel=myRepo.firstExample({name:'myName'});
myRepo.remove(model);
//->"myNamewasremovedsuccessfully"
New:
'usestrict';
constEventEmitter=require('events');
constemitter=newEventEmitter();
emitter.on('afterRemove',function(doc){
console.log(doc.name,'wasremoved');
});
//...
constdoc=myDocs.firstExample({name:'myName'});
myDocs.remove(doc);
emitter.emit('afterRemove',doc);
//->"myNamewasremovedsuccessfully"
Orsimply:
'usestrict';
functionafterRemove(doc){
console.log(doc.name,'wasremoved');
}
//...
constdoc=myDocs.firstExample({name:'myName'});
myDocs.remove(doc);
afterRemove(doc);
//->"myNamewasremovedsuccessfully"
RepositoriesandModels
263
ControllersvsroutersFoxxControllershavebeenreplacedwithrouters.Thisismorethanacosmeticchangeastherearesignificantdifferencesinbehaviour:
Controllerswereautomaticallymountedwhenthefiledefiningthemwasexecuted.Routersneedtobeexplicitlymountedusingthemodule.context.usemethod.Routerscanalsobeexported,importedandevennested.Thismakesiteasiertosplitupcomplexroutingtreesacrossmultiplefiles.
Old:
'usestrict';
constFoxx=require('org/arangodb/foxx');
constctrl=newFoxx.Controller(applicationContext);
ctrl.get('/hello',function(req,res){
//...
});
New:
'usestrict';
constcreateRouter=require('org/arangodb/foxx/router');
constrouter=createRouter();
//Ifyouareimportingthisfilefromyourentryfile("main"):
module.exports=router;
//Otherwise:module.context.use(router);
router.get('/hello',function(req,res){
//...
});
Somegeneralchangesinbehaviourthatmighttripyouup:
WhenspecifyingpathparameterswithschemasFoxxwillnowignoretherouteiftheschemadoesnotmatch(i.e./hello/foxxwillnolongermatch/hello/:numifnumspecifiesaschemathatdoesn'tmatchthevalue"foxx").Withcontrollersthiscouldpreviouslyresultinusersseeinga400(badrequest)errorwhentheyshouldinsteadbeserveda404(notfound)response.
WhenarequestismadewithanHTTPverbnotsupportedbyanendpoint,Foxxwillnowrespondwitha405(methodnotallowed)errorwithanappropriateAllowedheaderlistingthesupportedHTTPverbsforthatendpoint.
FoxxwillnolongerparseyourJSDoccommentstogenerateroutedocumentation(usethesummaryanddescriptionmethodsoftheendpointinstead).
TheapiDocumentationmethodnowlivesontheservicecontextandbehavesslightlydifferently.
ThereisnorouterequivalentfortheactivateAuthenticationandactivateSessionsmethods.Insteadyoushouldusethesessionmiddleware(seethesectiononsessionsbelow).
Thereisnodelaliasforthedeletemethodonrouters.IthasalwaysbeensafetousekeywordsasmethodnamesinFoxx,sotheuseofthisaliaswasalreadydiscouragedbefore.
TheallRoutesproxyisnolotavailableonroutersbutcaneasilybereplacedwithmiddlewareorchildrouters.
Controllers
264
TherequestcontextWhendefiningarouteonacontrollerthecontrollerwouldreturnanobjectcalledrequestcontext.Routersreturnasimilarobjectcalledendpoint.Routersalsoreturnendpointswhenmountingchildroutersormiddleware,asdoestheusemethodoftheservicecontext.
ThemaindifferencesbetweenthenewendpointsandtheobjectsreturnedbycontrollersinpreviousversionsofArangoDBare:
bodyParamisnowsimplycalledbody;itisnolongerneccessaryorpossibletogivethebodyanameandtherequestbodywillnotshowupintherequestparameters.It'salsopossibletospecifyaMIMEtype
body,queryParamandpathParamnowtakepositionargumentsinsteadofanobject.Forspecificsseetheendpointdocumentation.
notesisnowcalleddescriptionandtakesasinglestringargument.
onlyIfandonlyIfAuthenticatedarenolongeravailable;theycanbeemulatedwithmiddlewareifnecessary:
Old:
ctrl.get(/*...*/)
.onlyIf(function(req){
if(!req.user){
thrownewError('Notauthenticated!');
}
});
New:
router.use(function(req,res,next){
if(!req.arangoUser){
res.throw(403,'Notauthenticated!');
}
next();
});
router.get(/*...*/);
Controllers
265
ErrorhandlingTheerrorResponsemethodprovidedbycontrollerrequestcontextshasnoequivalentinrouterendpoints.Ifyouwanttohandlespecificerrortypeswithspecificstatuscodesyouneedtocatchthemexplicitly,eitherintherouteorinamiddleware:
Old:
ctrl.get('/puppies',function(req,res){
//Exceptionisthrownhere
})
.errorResponse(TooManyPuppiesError,400,'Somethingwentwrong!');
New:
ctrl.get('/puppies',function(req,res){
try{
//Exceptionisthrownhere
}catch(e){
if(!(einstanceofTooManyPuppiesError)){
throwe;
}
res.throw(400,'Somethingwentwrong!');
}
})
//The"error"methodmerelydocumentsthemeaning
//ofthestatuscodeandhasnoothereffect.
.error(400,'Throwniftherearetoomanypuppies.');
Notethaterrorscreatedwithhttp-errorsarestillhandledbyFoxxintelligently.Infactres.throwisjustahelpermethodforcreatingandthrowingtheseerrors.
Controllers
266
Before,afterandaroundThebefore,afterandaroundmethodscaneasilybereplacedbymiddleware:
Old:
letstart;
ctrl.before(function(req,res){
start=Date.now();
});
ctrl.after(function(req,res){
console.log('Requesthandledin',(Date.now()-start),'ms');
});
New:
router.use(function(req,res,next){
letstart=Date.now();
next();
console.log('Requesthandledin',(Date.now()-start),'ms');
});
Notethatunlikearoundmiddlewarereceivesthenextfunctionasthethirdargument(the"opts"argumenthasnoequivalent).
Controllers
267
RequestobjectsThenamesofsomeattributesoftherequestobjecthavebeenadjustedtomorecloselyalignwiththoseofthecorrespondingmethodsontheendpointobjectsandestablishedconventionsinotherJavaScriptframeworks:
req.urlParametersisnowcalledreq.pathParams
req.parametersisnowcalledreq.queryParams
req.params()isnowcalledreq.param()
req.requestTypeisnowcalledreq.method
req.compatibilityisnowcalledreq.arangoVersion
req.userisnowcalledreq.arangoUser
Someattributeshavebeenremovedorchanged:
req.cookieshasbeenremovedentirely(usereq.cookie(name))
req.requestBodyhasbeenremovedentirely(seebelow)
req.suffixisnowastringratherthananarray
Additionallythereq.serverandreq.clientattributesarenolongeravailable.Theinformationisnowexposedinawaythatcan(optionally)transparentlyhandleproxyforwardingheaders:
req.hostnamedefaultstoreq.server.address
req.portdefaultstoreq.server.port
req.remoteAddressdefaultstoclient.address
req.remotePortdefaultstoclient.port
Finally,thereq.cookiemethodnowtakesthesignedoptionsdirectly.
Old:
constsid=req.cookie('sid',{
signed:{
secret:'keyboardcat',
algorithm:'sha256'
}
});
New:
constsid=req.cookie('sid',{
secret:'keyboardcat',
algorithm:'sha256'
});
RequestbodiesThereq.bodyisnolongeramethodandnolongerautomaticallyparsesJSONrequestbodiesunlessarequestbodywasdefined.Thereq.rawBodynowcorrespondstothereq.rawBodyBufferofArangoDB2.xandisalsonolongeramethod.
Old:
ctrl.post('/',function(req,res){
constdata=req.body();
//...
});
Controllers
268
New:
router.post('/',function(req,res){
constdata=req.body;
//...
})
.body(['json']);
Orsimply:
constjoi=require('joi');
router.post('/',function(req,res){
constdata=req.body;
//...
})
.body(joi.object().optional());
Multipartrequests
Thereq.requestPartsmethodhasbeenremovedentirely.Ifyouneedtoacceptmultipartrequestbodies,youcansimplydefinetherequestbodyusingamultipartMIMEtypelikemultipart/form-data:
Old:
ctrl.post('/',function(req,res){
constparts=req.requestParts();
//...
});
New:
router.post('/',function(req,res){
constparts=req.body;
//...
})
.body(['multipart/form-data']);
Controllers
269
ResponseobjectsTheresponseobjecthasalotofnewmethodsinArangoDB3.0butotherwiseremainssimilartotheresponseobjectofpreviousversions:
Theres.sendmethodbehavesverydifferentlyfromhowthemethodwiththesamenamebehavedinArangoDB2.x:theconversionnowtakestheresponsebodydefinitionoftherouteintoaccount.Thereisanewmethodres.writethatimplementstheoldbehaviour.
Notethatconsecutivecallstores.writewillappendtotheresponsebodyratherthanreplacingitlikeres.send.
Theres.contentTypepropertyisalsonolongeravailable.IfyouwanttosettheMIMEtypeoftheresponsebodytoanexplicitvalueyoushouldsetthecontent-typeheaderinstead:
Old:
res.contentType='application/json';
res.body=JSON.stringify(results);
New:
res.set('content-type','application/json');
res.body=JSON.stringify(results);
Orsimply:
//setsthecontenttypetoJSON
//ifithasnotalreadybeenset
res.json(results);
Theres.cookiemethodnowtakesthesignedoptionsaspartoftheregularoptionsobject.
Old:
res.cookie('sid','abcdef',{
ttl:60*60,
signed:{
secret:'keyboardcat',
algorithm:'sha256'
}
});
New:
res.cookie('sid','abcdef',{
ttl:60*60,
secret:'keyboardcat',
algorithm:'sha256'
});
Controllers
270
DependencyinjectionThereisnoequivalentoftheaddInjectormethodavailableinArangoDB2.xcontrollers.Mostusecasescanbesolvedbysimplyusingplainvariablesbutifyouneedsomethingmoreflexibleyoucanalsousemiddleware:
Old:
ctrl.addInjector('magicNumber',function(){
returnMath.random();
});
ctrl.get('/',function(req,res,injected){
res.json(injected.magicNumber);
});
New:
functionmagicMiddleware(name){
return{
register(){
letmagic;
returnfunction(req,res,next){
if(!magic){
magic=Math.random();
}
req[name]=magic;
next();
};
}
};
}
router.use(magicMiddleware('magicNumber'));
router.get('/',function(req,res){
res.json(req.magicNumber);
});
Orsimply:
constmagicNumber=Math.random();
router.get('/',function(req,res){
res.json(magicNumber);
});
Controllers
271
SessionsThectrl.activateSessionsmethodandtherelatedutil-sessions-localFoxxservicehavebeenreplacedwiththeFoxxsessionsmiddleware.Itisnolongerpossibletousethebuilt-insessionstoragebutyoucansimplypassinanydocumentcollectiondirectly.
Old:
constlocalSessions=applicationContext.dependencies.localSessions;
constsessionStorage=localSessions.sessionStorage;
ctrl.activateSessions({
sessionStorage:sessionStorage,
cookie:{secret:'keyboardcat'}
});
ctrl.destroySession('/logout',function(req,res){
res.json({message:'Goodbye!'});
});
New:
constsessionMiddleware=require('@arangodb/foxx/sessions');
constcookieTransport=require('@arangodb/foxx/sessions/transports/cookie');
router.use(sessionMiddleware({
storage:module.context.collection('sessions'),
transport:cookieTransport('keyboardcat')
}));
router.post('/logout',function(req,res){
req.sessionStorage.clear(req.session);
res.json({message:'Goodbye!'});
});
Sessions
272
AuthandOAuth2Theutil-simple-authandutil-oauth2FoxxserviceshavebeenreplacedwiththeFoxxauthandFoxxOAuth2modules.Itisnolongernecessarytoinstalltheseservicesasdependenciesinordertousethefunctionality.
Old:
'usestrict';
constauth=applicationContext.dependencies.simpleAuth;
//...
constvalid=auth.verifyPassword(authData,password);
New:
'usestrict';
constcreateAuth=require('@arangodb/foxx/auth');
constauth=createAuth();//Usedefaultconfiguration
//...
constvalid=auth.verifyPassword(authData,password);
AuthandOAuth2
273
FoxxqueriesThecreateQuerymethodhasbeenremoved.ItcanbetriviallyreplacedwithplainJavaScriptfunctionsanddirectcallstothedb._querymethod:
Old:
'usestrict';
constFoxx=require('org/arangodb/foxx');
constquery=Foxx.createQuery({
query:'FORuIN_usersSORTu.userASCRETURNu[@propName]',
params:['propName'],
transform:function(results,uppercase){
return(
uppercase
?results[0].toUpperCase()
:results[0].toLowerCase()
);
}
});
query('user',true);
New:
'usestrict';
constdb=require('@arangodb').db;
constaql=require('@arangodb').aql;
functionquery(propName,uppercase){
constresults=db._query(aql`
FORuIN_users
SORTu.userASC
RETURNu[${propName}]
`);
return(
uppercase
?results[0].toUpperCase()
:results[0].toLowerCase()
);
}
query('user',true);
FoxxQueries
274
Legacycompatibilitymodefor2.8servicesArangoDB3continuestosupportFoxxserviceswrittenforArangoDB2.8byrunningtheminaspeciallegacycompatibilitymodethatprovidesaccesstosomeofthemodulesandAPIsnolongerprovidedin3.0andbeyond.
Note:LegacycompatibilitymodeisstrictlyintendedasatemporarystopgapsolutionforsupportingexistingserviceswhileupgradingtoArangoDB3.0andshouldnotbeconsideredapermanentfeatureofArangoDBorFoxx.
Inordertomarkanexistingserviceasalegacyservice,justmakesurethefollowingattributeisdefinedintheservicemanifest:
"engines":{
"arangodb":"^2.8.0"
}
ThissemanticversionrangedenotesthattheserviceisknowntoworkwithArangoDB2.8.0andsupportsallnewerversionsofArangoDBuptobutnotincluding3.0.0(noranydevelopmentversionof3.0.0andgreater).
Anysimilarversionrangethedoesnotinclude3.0.0orgreaterwillhavethesameeffect(e.g. 2.5.0willalsotriggerthelegacycompatibilitymode,aswill1.2.3,but>=2.8.0willnotasitindicatescompatibilitywithallversionsgreaterorequal2.8.0,notjustthosewithinthe2.xversionrange).
FeaturessupportedinlegacycompatibilitymodeLegacycompatibilitymodesupportstheoldmanifestformat,specifically:
mainisignoredcontrollerswillbemountedasin2.8exportswillbeexecutedasin2.8
AdditionallytheisSystemattributewillbeignoredifpresentbutdoesnotresultinawarninginlegacycompatibilitymode.
TheFoxxconsoleisavailableastheconsolepseudo-globalvariable(shadowingtheglobalconsoleobject).
TheservicecontextisavailableastheapplicationContextpseudo-globalvariableinthecontrollers,exports,scriptsandtestsasin2.8.Thefollowingadditionalpropertiesareavailableontheservicecontextinlegacycompatibilitymode:
path()isanaliasfor3.xfileName()(usingpath.jointobuildfilepaths)fileName()behavesasin2.x(usingfs.safeJointobuildfilepaths)foxxFileName()isanaliasfor2.xfileNameversionexposestheservicemanifest'sversionattributenameexposestheservicemanifest'snameattributeoptionsexposestheservice'srawoptions
Thefollowingmethodsareremovedontheservicecontextinlegacycompatibilitymode:
use()--use@arangodb/foxx/controllerinsteadapiDocumentation()--usecontroller.apiDocumentation()insteadregisterType()--notsupportedinlegacycompatibilitymode
Thefollowingmodulesthathavebeenremovedorreplacedin3.0.0areavailableinlegacycompatibilitymode:
@arangodb/foxx/authentication
@arangodb/foxx/console
@arangodb/foxx/controller
@arangodb/foxx/model
@arangodb/foxx/query
@arangodb/foxx/repository
@arangodb/foxx/schema
@arangodb/foxx/sessions
@arangodb/foxx/template_middleware
Legacycompatibilitymode
275
The@arangodb/foxxmodulealsoprovidesthesameexportsasin2.8,namely:
Controllerfrom@arangodb/foxx/controllercreateQueryfrom@arangodb/foxx/queryModelfrom@arangodb/foxx/modelRepositoryfrom@arangodb/foxx/repositorytoJSONSchemafrom@arangodb/foxx/schemagetExportsandrequireAppfrom@arangodb/foxx/managerqueuesfrom@arangodb/foxx/queues
Anyfeaturenotsupportedin2.8willalsonotworkinlegacycompatibilitymode.WhenmigratingfromanolderversionofArangoDBitisagoodideatomigratetoArangoDB2.8firstforaneasierupgradepath.
AdditionallypleasenotethedifferenceslaidoutinthechapterMigratingfrompre-2.8inthemigrationguide.
Legacycompatibilitymode
276
UsermanagementFoxxdoesnotprovideanyusermanagementoutoftheboxbutitisveryeasytorollyourownsolution:
Thesessionmiddlewareprovidesmechanismsforaddingsessionlogictoyourservice,usinge.g.acollectionorJSONWebTokenstostorethesessionsbetweenrequests.
Theauthmoduleprovidesutilitiesforbasicpasswordverificationandhashing.
Thefollowingexampleservicedemonstrateshowusermanagementcanbeimplementedusingthesebasicbuildingblocks.
Settingupthecollections
Let'ssaywewanttostoresessionsandusersincollections.Wecanusethesetupscripttomakesurethesecollectionsarecreatedbeforetheserviceismounted.
Firstaddasetupscripttoyourmanifestifitisn'talreadydefined:
"scripts":{
"setup":"scripts/setup.js"
}
Thencreatethesetupscriptwiththefollowingcontent:
'usestrict';
constdb=require('@arangodb').db;
constsessions=module.context.collectionName('sessions');
constusers=module.context.collectionName('users');
if(!db._collection(sessions)){
db._createDocumentCollection(sessions);
}
if(!db._collection(users)){
db._createDocumentCollection(users);
}
db._collection(users).ensureIndex({
type:'hash',
fields:['username'],
unique:true
});
Creatingtherouter
Thefollowingmainfiledemonstratesbasicusermanagement:
'usestrict';
constjoi=require('joi');
constcreateAuth=require('@arangodb/foxx/auth');
constcreateRouter=require('@arangodb/foxx/router');
constsessionsMiddleware=require('@arangodb/foxx/sessions');
constauth=createAuth();
constrouter=createRouter();
constusers=module.context.collection('users');
constsessions=sessionsMiddleware({
storage:module.context.collection('sessions'),
transport:'cookie'
});
module.context.use(sessions);
module.context.use(router);
router.get('/whoami',function(req,res){
Usermanagement
277
try{
constuser=users.document(req.session.uid);
res.send({username:user.username});
}catch(e){
res.send({username:null});
}
})
.description('Returnsthecurrentlyactiveusername.');
router.post('/login',function(req,res){
//Thismayreturnauserobjectornull
constuser=users.firstExample({
username:req.body.username
});
constvalid=auth.verify(
//Pretendtovalidateevenifnouserwasfound
user?user.authData:{},
req.body.password
);
if(!valid)res.throw('unauthorized');
//Logtheuserin
req.session.uid=user._key;
req.sessionStorage.save(req.session);
res.send({sucess:true});
})
.body(joi.object({
username:joi.string().required(),
password:joi.string().required()
}).required(),'Credentials')
.description('Logsaregistereduserin.');
router.post('/logout',function(req,res){
if(req.session.uid){
req.session.uid=null;
req.sessionStorage.save(req.session);
}
res.send({success:true});
})
.description('Logsthecurrentuserout.');
router.post('/signup',function(req,res){
constuser=req.body;
try{
//Createanauthenticationhash
user.authData=auth.create(user.password);
deleteuser.password;
constmeta=users.save(user);
Object.assign(user,meta);
}catch(e){
//Failedtosavetheuser
//We'llassumetheUniqueConstrainthasbeenviolated
res.throw('badrequest','Usernamealreadytaken',e);
}
//Logtheuserin
req.session.uid=user._key;
req.sessionStorage.save(req.session);
res.send({success:true});
})
.body(joi.object({
username:joi.string().required(),
password:joi.string().required()
}).required(),'Credentials')
.description('Createsanewuserandlogsthemin.');
Usermanagement
278
RelatedmodulesThesearesomeofthemodulesoutsideofFoxxyouwillfindusefulwhenwritingFoxxservices.
AdditionallytherearemodulesprovidingsomelevelofcompatibilitywithNode.jsaswellasanumberofbundledNPMmodules(likelodashandjoi).FormoreinformationonthesemodulesseetheJavaScriptmodulesappendix.
The @arangodbmodulerequire('@arangodb')
ThismoduleprovidesaccesstovariousArangoDBinternalsaswellasthreeofthemostimportantexportsnecessarytoworkwiththedatabaseinFoxx:db,aqlanderrors.
YoucanfindafulldescriptionofthismoduleintheArangoDBmoduleappendix.
The @arangodb/requestmodulerequire('@arangodb/request')
ThismoduleprovidesafunctionformakingHTTPrequeststoexternalservices.Notethatwhilethisallowscommunicatingwiththird-partyservicesitmayaffectdatabaseperformancebyblockingFoxxrequestsasArangoDBwaitsfortheremoteservicetorespond.Ifyouroutinelymakerequeststoslowexternalservicesandarenotdirectlyinterestedintheresponseitisprobablyabetterideatodelegatetheactualrequest/responsecycletoagatewayservicerunningoutsideArangoDB.
Youcanfindafulldescriptionofthismoduleintherequestmoduleappendix.
The @arangodb/general-graphmodulerequire('@arangodb/general-graph')
ThismoduleprovidesaccesstoArangoDBgraphdefinitionsandvariouslow-levelgraphoperationsinJavaScript.FormorecomplexqueriesitisprobablybettertouseAQLbutthismodulecanbeusefulinyoursetupandteardownscriptstocreateanddestroygraphdefinitions.
Formoreinformationseethechapteronthegeneralgraphmodule.
Relatedmodules
279
AuthenticationconstcreateAuth=require('@arangodb/foxx/auth');
Authenticatorsallowimplementingbasicpasswordmechanismusingsimplebuilt-inhashingfunctions.
ForafullexampleofsessionswithauthenticationandregistrationseetheexampleinthechapteronUserManagement.
CreatinganauthenticatorcreateAuth([options]):Authenticator
Createsanauthenticator.
Arguments
options:Object(optional)
Anobjectwiththefollowingproperties:
method:string(Default:"sha256")
Thehashingalgorithmtousetocreatepasswordhashes.Theauthenticatorwillbeabletoverifypasswordsagainsthashesusinganysupportedhashingalgorithm.Thisonlyaffectsnewhashescreatedbytheauthenticator.
Supportedvalues:
"md5"
"sha1"
"sha224"
"sha256"
"sha384"
"sha512"
saltLength:number(Default:16)
Lengthofthesaltsthatwillbegeneratedforpasswordhashes.
Returnsanauthenticator.
Creatingauthenticationdataobjectsauth.create(password):AuthData
Createsanauthenticationdataobjectforthegivenpasswordwiththefollowingproperties:
method:string
Themethodusedtogeneratethehash.
salt:string
Arandomsaltusedtogeneratethishash.
hash:string
Thehashstringitself.
Arguments
password:string
Apasswordtohash.
Returnstheauthenticationdataobject.
Authentication
280
Validatingpasswordsagainstauthenticationdataobjectsauth.verify([hash,[password]]):boolean
Verifiesthegivenpasswordagainstthegivenhashusingaconstanttimestringcomparison.
Arguments
hash:AuthData(optional)
Aauthenticationdataobjectgeneratedwiththecreatemethod.
password:string(optional)
Apasswordtoverifyagainstthehash.
Returnstrueifthehashmatchesthegivenpassword.Returnsfalseotherwise.
Authentication
281
OAuth1.0aconstcreateOAuth1Client=require('@arangodb/foxx/oauth1');
TheOAuth1moduleprovidesabstractionsoverOAuth1.0aproviderslikeTwitter,XINGandTumblr.
Examples
Thefollowingextendstheusermanagementexample:
constrouter=createRouter();
constoauth1=createOAuth1Client({
//We'lluseTwitterforthisexample
requestTokenEndpoint:'https://api.twitter.com/oauth/request_token',
authEndpoint:'https://api.twitter.com/oauth/authorize',
accessTokenEndpoint:'https://api.twitter.com/oauth/access_token',
activeUserEndpoint:'https://api.twitter.com/1.1/account/verify_credentials.json',
clientId:'keyboardcat',
clientSecret:'keyboardcat'
});
module.context.use('/oauth1',router);
//Seetheusermanagementexampleforsettingupthe
//sessionsandusersobjectsusedinthisexample
router.use(sessions);
router.post('/auth',function(req,res){
consturl=req.reverse('oauth1_callback');
constoauth_callback=req.makeAbsolute(url);
constrequestToken=oauth1.fetchRequestToken(oauth_callback);
if(requestToken.oauth_callback_confirmed!=='true'){
res.throw(500,'CouldnotfetchOAuthrequesttoken');
}
//Setrequesttokencookieforfiveminutes
res.cookie('oauth1_request_token',requestToken.oauth_token,{ttl:60*5});
//Redirecttotheprovider'sauthorizationURL
res.redirect(303,oauth1.getAuthUrl(requestToken.oauth_token));
});
router.get('/auth',function(req,res){
//MakesureCSRFcookiematchestheURL
constexpectedToken=req.cookie('oauth1_request_token');
if(!expectedToken||req.queryParams.oauth_token!==expectedToken){
res.throw(400,'CSRFmismatch.');
}
constauthData=oauth1.exchangeRequestToken(
req.queryParams.oauth_token,
req.queryParams.oauth_verifier
);
consttwitterToken=authData.oauth_token;
consttwitterSecret=authData.oauth_token_secret;
//Fetchtheactiveuser'sprofileinfo
constprofile=oauth1.fetchActiveUser(twitterToken,twitterSecret);
consttwitterId=profile.screen_name;
//TrytofindanexistinguserwiththeuserID
//(thisrequirestheuserscollection)
letuser=users.firstExample({twitterId});
if(user){
//UpdatethetwitterTokenifithaschanged
if(
user.twitterToken!==twitterToken||
user.twitterSecret!==twitterSecret
){
users.update(user,{twitterToken,twitterSecret});
}
}else{
//Createanewuserdocument
user={
username:`twitter:${twitterId}`,
twitterId,
twitterToken
OAuth1.0a
282
}
constmeta=users.save(user);
Object.assign(user,meta);
}
//Logtheuserin(thisrequiresthesessionmiddleware)
req.session.uid=user._key;
req.session.twitterToken=authData.twitterToken;
req.session.twitterSecret=authData.twitterSecret;
req.sessionStorage.save(req.session);
//Redirecttothedefaultroute
res.redirect(303,req.makeAbsolute('/'));
},'oauth1_callback')
.queryParam('oauth_token',joi.string().optional())
.queryParam('oauth_verifier',joi.string().optional());
CreatinganOAuth1.0aclientcreateOAuth1Client(options):OAuth1Client
CreatesanOAuth1.0aclient.
Arguments
options:Object
Anobjectwiththefollowingproperties:
requestTokenEndpoint:string
Thefully-qualifiedURLoftheprovider'sTemporaryCredentialsRequestendpoint.ThisURLisusedtofetchtheunauthenticatedtemporarycredentialsthatwillbeusedtogeneratetheauthorizationredirectfortheuser.
authEndpoint:string
Thefully-qualifiedURLoftheprovider'sResourceOwnerAuthorizationendpoint.ThisistheURLtheuserwillberedirectedtoinordertoauthorizetheOAuthconsumer(i.e.yourservice).
accessTokenEndpoint:string
Thefully-qualifiedURLoftheprovider'sTokenRequestendpoint.ThisURLisusedtoexchangetheauthenticatedtemporarycredentialsreceivedfromtheauthorizationredirectfortheactualtokencredentialsthatcanbeusedtomakerequeststotheAPIserver.
activeUserEndpoint:string(optional)
Thefully-qualifiedURLoftheprovider'sendpointforfetchingdetailsaboutthecurrentuser.
clientId:string
Theapplication'sClientID(orConsumerKey)fortheprovider.
clientSecret:string
Theapplication'sClientSecret(orConsumerSecret)fortheprovider.
signatureMethod:string(Default:"HMAC-SHA1")
ThecryptographicmethodthatwillbeusedtosignOAuth1.0arequests.Only"HMAC-SHA1-"and"PLAINTEXT"aresupportedatthistime.
Notethatmanyprovidersmaynotimplement"PLAINTEXT"asitexposestheClientSecretandoauth_token_secretinsteadofgeneratingasignature.
ReturnsanOAuth1.0aclientforthegivenprovider.
SettingupOAuth1.0aforTwitter
IfyouwanttouseTwitterastheOAuth1.0aprovider,usethefollowingoptions:
OAuth1.0a
283
requestTokenEndpoint:https://api.twitter.com/oauth/request_tokenauthEndpoint:https://api.twitter.com/oauth/authorizeaccessTokenEndpoint:https://api.twitter.com/oauth/access_tokenactiveUserEndpoint:https://api.twitter.com/1.1/account/verify_credentials.json
YoualsoneedtoobtainaclientIDandclientsecretfromTwitter:
1. CreatearegularaccountatTwitteroruseanexistingaccountyouown.2. VisittheTwitterApplicationManagementdashboardandsigninwithyourTwitteraccount.3. ClickonCreateNewAppandfollowtheinstructionsprovided.TheCallbackURLshouldmatchyouroauth_callbacklater.You
maybepromptedtoaddamobilephonenumbertoyouraccountandverifyit.4. OpentheKeysandAccessTonestab,thennotedowntheConsumerKeyandConsumerSecret.5. SettheoptionclientIdtotheConsumerKeyandtheoptionclientSecrettotheConsumerSecret.
Notethatifyouonlyneedread-onlyaccesstopublicinformation,youcanalsousetheclientIdandclientSecretdirectlywithoutOAuth1.0a.
SeeTwitterRESTAPIReferenceDocumentation.
SettingupOAuth1.0aforXING
IfyouwanttouseXINGastheOAuth1.0aprovider,usethefollowingoptions:
requestTokenEndpoint:https://api.xing.com/v1/request_tokenauthEndpoint:https://api.xing.com/v1/authorizeaccessTokenEndpoint:https://api.xing.com/v1/access_tokenactiveUserEndpoint:https://api.xing.com/v1/users/me
YoualsoneedtoobtainaclientIDandclientsecretfromXING:
1. CreatearegularaccountatXINGoruseanexistingaccountyouown.2. VisittheXINGDeveloperpageandsigninwithyourXINGaccount.3. ClickonCreateappandnotedowntheConsumerkeyandConsumersecret.4. SettheoptionclientIdtotheConsumerkeyandtheoptionclientSecrettotheConsumersecret.
SeeXINGDeveloperDocumentation.
SettingupOAuth1.0aforTumblr
IfyouwanttouseTumblrastheOAuth1.0aprovider,usethefollowingoptions:
requestTokenEndpoint:https://www.tumblr.com/oauth/request_tokenauthEndpoint:https://www.tumblr.com/oauth/authorizeaccessTokenEndpoint:https://www.tumblr.com/oauth/access_tokenactiveUserEndpoint:https://api.tumblr.com/v2/user/info
YoualsoneedtoobtainaclientIDandclientsecretfromTumblr:
1. CreatearegularaccountatTumblroruseanexistingaccountyouown.2. VisittheTumblrApplicationsdashboard.3. ClickonRegisterapplication,thenfollowtheinstructionsprovided.TheDefaultcallbackURLshouldmatchyouroauth_callback
later.4. NotedowntheOAuthConsumerKeyandSecretKey.Thesecretmaybehiddenbydefault.5. SettheoptionclientIdtotheOAuthConsumerKeyandtheoptionclientSecrettotheSecretKey.
SeeTumblrAPIDocumentation.
Fetchanunauthenticatedrequesttokenoauth1.fetchRequestToken(oauth_callback,opts)
Fetchesanoauth_tokenthatcanbeusedtocreateanauthorizationURLthatredirectstothegivenoauth_callbackonconfirmation.
OAuth1.0a
284
PerformsaPOSTresponsetotherequestTokenEndpoint.
Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.
Arguments
oauth_callback:string
Thefully-qualifiedURLofyourapplication'sOAuth1.0acallback.
opts:Object(optional)
Anobjectwithadditionalqueryparameterstoincludeintherequest.
SeeRFC5849.
Returnstheparsedresponseobject.
GettheauthorizationURLoauth1.getAuthUrl(oauth_token,opts):string
GeneratestheauthorizationURLfortheauthorizationendpoint.
Arguments
oauth_token:string
Theoauth_tokenpreviouslyreturnedbyfetchRequestToken.
opts:(optional)
AnobjectwithadditionalqueryparameterstoaddtotheURL.
SeeRFC5849.
Returnsafully-qualifiedURLfortheauthorizationendpointoftheproviderbyappendingtheoauth_tokenandanyadditionalargumentsfromoptstotheauthEndpoint.
Examples
constrequestToken=oauth1.fetchRequestToken(oauth_callback);
if(requestToken.oauth_callback_confirmed!=='true'){
thrownewError('ProvidercouldnotconfirmOAuth1.0callback');
}
constauthUrl=oauth1.getAuthUrl(requestToken.oauth_token);
Exchangeanauthenticatedrequesttokenforanaccesstokenoauth1.exchangeRequestToken(oauth_token,oauth_verifier,opts)
TakesapairofauthenticatedtemporarycredentialspassedtothecallbackURLbytheproviderandexchangesitforanoauth_tokenandoauth_token_secretthancanbeusedtoperformauthenticatedrequeststotheOAuth1.0aprovider.
PerformsaPOSTresponsetotheaccessTokenEndpoint.
Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.
Arguments
oauth_token:string
Theoauth_tokenpassedtothecallbackURLbytheprovider.
oauth_verifier:string
Theoauth_verifierpassedtothecallbackURLbytheprovider.
opts:Object(optional)
OAuth1.0a
285
Anobjectwithadditionalqueryparameterstoincludeintherequest.
SeeRFC5849.
Returnstheparsedresponseobject.
Fetchtheactiveuseroauth1.fetchActiveUser(oauth_token,oauth_token_secret,opts):Object
Fetchesdetailsoftheactiveuser.
PerformsaGETresponsetotheactiveUserEndpoint.
Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.
ReturnsnulliftheactiveUserEndpointisnotconfigured.
Arguments
oauth_token:string
AnOAuth1.0aaccesstokenasreturnedbyexchangeRequestToken.
oauth_token_secret:string
AnOAuth1.0aaccesstokensecretasreturnedbyexchangeRequestToken.
opts:Object(optional)
Anobjectwithadditionalqueryparameterstoincludeintherequest.
SeeRFC5849.
Returnstheparsedresponseobject.
Examples
constauthData=oauth1.exchangeRequestToken(oauth_token,oauth_verifier);
constuserData=oauth1.fetchActiveUser(authData.oauth_token,authData.oauth_token_secret);
Createanauthenticatedrequestobjectoauth1.createSignedRequest(method,url,parameters,oauth_token,oauth_token_secret)
CreatesarequestobjectthatcanbeusedtoperformarequesttotheOAuth1.0aproviderwiththeprovidedtokencredentials.
Arguments
method:string
HTTPmethodtherequestwilluse,e.g."POST".
url:string
Thefully-qualifiedURLoftheprovidertherequestwillbeperformedagainst.
TheURLmayoptionallycontainanynumberofqueryparameters.
parameters:string|Object|null
Anadditionalobjectorquerystringcontainingqueryparametersorbodyparametersthatwillbepartofthesignedrequest.
oauth_token:string
AnOAuth1.0aaccesstokenasreturnedbyexchangeRequestToken.
oauth_token_secret:string
OAuth1.0a
286
AnOAuth1.0aaccesstokensecretasreturnedbyexchangeRequestToken.
Returnsanobjectwiththreeproperties:
url:ThenormalizedURLwithoutanyqueryparameters.
qs:Anormalizedquerystringcontainingallparametersandqueryparameters.
headers:Anobjectcontainingthefollowingproperties:
accept:Thestring"application/json".
authorization:AnOAuthauthorizationheadercontainingallOAuthparametersandtherequestsignature.
Examples
Fetchalistoftweetsmentioning@arangodb:
constrequest=require('@arangodb/request');
constreq=oauth1.createSignedRequest(
'GET',
'https://api.twitter.com/1.1/search/tweets.json',
{q:'@arangodb'},
authData.oauth_token,
authData.oauth_token_secret
);
constres=request(req);
console.log(res.json.statuses);
Signingamorecomplexrequest:
consturl='https://api.example.com/v1/timeline?visible=public';
constparams={hello:'world',longcat:'islong'};
constreq=oauth1.createSignedRequest(
'POST',
url,//URLincludesaqueryparameterthatwillbesigned
params,//Requestbodyneedstobesignedtoo
authData.oauth_token,
authData.oauth_token_secret
);
constres=request.post(url,{
form:params,
headers:{
accept:'application/x-www-form-urlencoded',
//Authorizationheaderincludesthesignature
authorization:req.headers.authorization
}
});
console.log(res.json);
OAuth1.0a
287
OAuth2.0constcreateOAuth2Client=require('@arangodb/foxx/oauth2');
TheOAuth2moduleprovidesabstractionsoverOAuth2.0providerslikeFacebook,GitHubandGoogle.
Examples
Thefollowingextendstheusermanagementexample:
constcrypto=require('@arangodb/crypto');
constrouter=createRouter();
constoauth2=createOAuth2Client({
//We'lluseFacebookforthisexample
authEndpoint:'https://www.facebook.com/dialog/oauth',
tokenEndpoint:'https://graph.facebook.com/oauth/access_token',
activeUserEndpoint:'https://graph.facebook.com/v2.0/me',
clientId:'keyboardcat',
clientSecret:'keyboardcat'
});
module.context.use('/oauth2',router);
//Seetheusermanagementexampleforsettingupthe
//sessionsandusersobjectsusedinthisexample
router.use(sessions);
router.post('/auth',function(req,res){
constcsrfToken=crypto.genRandomAlphaNumbers(32);
consturl=req.reverse('oauth2_callback',{csrfToken});
constredirect_uri=req.makeAbsolute(url);
//SetCSRFcookieforfiveminutes
res.cookie('oauth2_csrf_token',csrfToken,{ttl:60*5});
//Redirecttotheprovider'sauthorizationURL
res.redirect(303,oauth2.getAuthUrl(redirect_uri));
});
router.get('/auth',function(req,res){
//Someproviderspasserrorsasqueryparameter
if(req.queryParams.error){
res.throw(500,`Providererror:${req.queryParams.error}`)
}
//MakesureCSRFcookiematchestheURL
constexpectedToken=req.cookie('oauth2_csrf_token');
if(!expectedToken||req.queryParams.csrfToken!==expectedToken){
res.throw(400,'CSRFmismatch.');
}
//MakesuretheURLcontainsagranttoken
if(!req.queryParams.code){
res.throw(400,'Providerdidnotpassgranttoken.');
}
//Reconstructtheredirect_uriusedforthegranttoken
consturl=req.reverse('oauth2_callback');
constredirect_uri=req.makeAbsolute(url);
//Fetchanaccesstokenfromtheprovider
constauthData=oauth2.exchangeGrantToken(
req.queryParams.code,
redirect_uri
);
constfacebookToken=authData.access_token;
//Fetchtheactiveuser'sprofileinfo
constprofile=oauth2.fetchActiveUser(facebookToken);
constfacebookId=profile.id;
//TrytofindanexistinguserwiththeuserID
//(thisrequirestheuserscollection)
letuser=users.firstExample({facebookId});
if(user){
//UpdatethefacebookTokenifithaschanged
if(user.facebookToken!==facebookToken){
users.update(user,{facebookToken});
}
}else{
OAuth2.0
288
//Createanewuserdocument
user={
username:`fb:${facebookId}`,
facebookId,
facebookToken
}
constmeta=users.save(user);
Object.assign(user,meta);
}
//Logtheuserin(thisrequiresthesessionmiddleware)
req.session.uid=user._key;
req.session.facebookToken=authData.facebookToken;
req.sessionStorage.save(req.session);
//Redirecttothedefaultroute
res.redirect(303,req.makeAbsolute('/'));
},'oauth2_callback')
.queryParam('error',joi.string().optional())
.queryParam('csrfToken',joi.string().optional())
.queryParam('code',joi.string().optional());
CreatinganOAuth2.0clientcreateOAuth2Client(options):OAuth2Client
CreatesanOAuth2.0client.
Arguments
options:Object
Anobjectwiththefollowingproperties:
authEndpoint:string
Thefully-qualifiedURLoftheprovider'sauthorizationendpoint.
tokenEndpoint:string
Thefully-qualifiedURLoftheprovider'stokenendpoint.
refreshEndpoint:string(optional)
Thefully-qualifiedURLoftheprovider'srefreshtokenendpoint.
activeUserEndpoint:string(optional)
Thefully-qualifiedURLoftheprovider'sendpointforfetchingdetailsaboutthecurrentuser.
clientId:string
Theapplication'sClientID(orAppID)fortheprovider.
clientSecret:string
Theapplication'sClientSecret(orAppSecret)fortheprovider.
ReturnsanOAuth2.0clientforthegivenprovider.
SettingupOAuth2.0forFacebook
IfyouwanttouseFacebookastheOAuth2.0provider,usethefollowingoptions:
authEndpoint:https://www.facebook.com/dialog/oauthtokenEndpoint:https://graph.facebook.com/oauth/access_tokenactiveUserEndpoint:https://graph.facebook.com/v2.0/me
YoualsoneedtoobtainaclientIDandclientsecretfromFacebook:
1. CreatearegularaccountatFacebookoruseanexistingaccountyouown.2. VisittheFacebookDeveloperspage.
OAuth2.0
289
3. ClickonAppsinthemenu,thenselectRegisterasaDeveloper(theonlyoption)andfollowtheinstructionsprovided.Youmayneedtoverifyyouraccountbyphone.
4. ClickonAppsinthemenu,thenselectCreateaNewAppandfollowtheinstructionsprovided.5. Opentheappdashboard,thennotedowntheAppIDandAppSecret.Thesecretmaybehiddenbydefault.6. ClickonSettings,thenAdvancedandenteroneormoreValidOAuthredirectURIs.Atleastoneofthemmustmatchyourredirect_urilater.Don'tforgettosaveyourchanges.
7. SettheoptionclientIdtotheAppIDandtheoptionclientSecrettotheAppSecret.
SettingupOAuth2.0forGitHub
IfyouwanttouseGitHubastheOAuth2.0provider,usethefollowingoptions:
authEndpoint:https://github.com/login/oauth/authorize?scope=usertokenEndpoint:https://github.com/login/oauth/access_tokenactiveUserEndpoint:https://api.github.com/user
YoualsoneedtoobtainaclientIDandclientsecretfromGitHub:
1. CreatearegularaccountatGitHuboruseanexistingaccountyouown.2. GotoAccountSettings>Applications>Registernewapplication.3. ProvideanauthorizationcallbackURL.Thismustmatchyourredirect_urilater.4. Fillintheotherrequireddetailsandfollowtheinstructionsprovided.5. Opentheapplicationpage,thennotedowntheClientIDandClientSecret.6. SettheoptionclientIdtotheClientIDandtheoptionclientSecrettotheClientSecret.
SettingupOAuth2.0forGoogle
IfyouwanttouseGoogleastheOAuth2.0provider,usethefollowingoptions:
authEndpoint:https://accounts.google.com/o/oauth2/auth?access_type=offline&scope=profiletokenEndpoint:https://accounts.google.com/o/oauth2/tokenactiveUserEndpoint:https://www.googleapis.com/plus/v1/people/me
YoualsoneedtoobtainaclientIDandclientsecretfromGoogle:
1. CreatearegularaccountatGoogleoruseanexistingaccountyouown.2. VisittheGoogleDevelopersConsole.3. ClickonCreateProject,thenfollowtheinstructionsprovided.4. Whenyourprojectisready,opentheprojectdashboard,thenclickonEnableanAPI.5. EnabletheGoogle+APItoallowyourapptodistinguishbetweendifferentusers.6. OpentheCredentialspageandclickCreatenewClientID,thenfollowtheinstructionsprovided.AtleastoneAuthorizedRedirectURImustmatchyourredirect_urilater.AtleastoneAuthorizedJavaScriptOriginmustmatchyourapp'sfully-qualifieddomain.
7. WhentheClientIDisready,notedowntheClientIDandClientsecret.8. SettheoptionclientIdtotheClientIDandtheoptionclientSecrettotheClientsecret.
GettheauthorizationURLoauth2.getAuthUrl(redirect_uri,args):string
GeneratestheauthorizationURLfortheauthorizationendpoint.
Arguments
redirect_uri:string
Thefully-qualifiedURLofyourapplication'sOAuth2.0callback.
args:(optional)
Anobjectwithanyofthefollowingproperties:
response_type:string(Default:"code")
OAuth2.0
290
SeeRFC6749.
Returnsafully-qualifiedURLfortheauthorizationendpointoftheproviderbyappendingtheclientIDandanyadditionalargumentsfromargstotheauthEndpoint.
Exchangeagrantcodeforanaccesstokenoauth2.exchangeGrantToken(code,redirect_uri)
Exchangesagrantcodeforanaccesstoken.
PerformsaPOSTresponsetothetokenEndpoint.
Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.
Arguments
code:string
Agrantcodereturnedbytheprovider'sauthorizationendpoint.
redirect_uri:string
TheoriginalcallbackURLwithwhichthecodewasrequested.
args:Object(optional)
Anobjectwithanyofthefollowingproperties:
grant_type:string(Default:"authorization_code")
SeeRFC6749.
Returnstheparsedresponseobject.
Fetchtheactiveuseroauth2.fetchActiveUser(access_token):Object
Fetchesdetailsoftheactiveuser.
PerformsaGETresponsetotheactiveUserEndpoint.
Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.
ReturnsnulliftheactiveUserEndpointisnotconfigured.
Arguments
access_token:string
AnOAuth2.0accesstokenasreturnedbyexchangeGrantToken.
Returnstheparsedresponseobject.
Examples
constauthData=oauth2.exchangeGrantToken(code,redirect_uri);
constuserData=oauth2.fetchActiveUser(authData.access_token);
OAuth2.0
291
TransactionsStartingwithversion1.3,ArangoDBprovidessupportforuser-definabletransactions.
TransactionsinArangoDBareatomic,consistent,isolated,anddurable(ACID).
TheseACIDpropertiesprovidethefollowingguarantees:
Theatomicityprinciplemakestransactionseithercompleteintheirentiretyorhavenoeffectatall.Theconsistencyprincipleensuresthatnoconstraintsorotherinvariantswillbeviolatedduringorafteranytransaction.Theisolationpropertywillhidethemodificationsofatransactionfromothertransactionsuntilthetransactioncommits.Finally,thedurabilitypropositionmakessurethatoperationsfromtransactionsthathavecommittedwillbemadepersistent.TheamountoftransactiondurabilityisconfigurableinArangoDB,asisthedurabilityoncollectionlevel.
Transactions
292
TransactioninvocationArangoDBtransactionsaredifferentfromtransactionsinSQL.
InSQL,transactionsarestartedwithexplicitBEGINorSTARTTRANSACTIONcommand.Followinganyseriesofdataretrievalormodificationoperations,anSQLtransactionisfinishedwithaCOMMITcommand,orrolledbackwithaROLLBACKcommand.Theremaybeclient/servercommunicationbetweenthestartandthecommit/rollbackofanSQLtransaction.
InArangoDB,atransactionisalwaysaserver-sideoperation,andisexecutedontheserverinonego,withoutanyclientinteraction.Alloperationstobeexecutedinsideatransactionneedtobeknownbytheserverwhenthetransactionisstarted.
TherearenoindividualBEGIN,COMMITorROLLBACKtransactioncommandsinArangoDB.Instead,atransactioninArangoDBisstartedbyprovidingadescriptionofthetransactiontothedb._executeTransactionJavaScriptfunction:
db._executeTransaction(description);
Thisfunctionwillthenautomaticallystartatransaction,executeallrequireddataretrievaland/ormodificationoperations,andattheendautomaticallycommitthetransaction.Ifanerroroccursduringtransactionexecution,thetransactionisautomaticallyaborted,andallchangesarerolledback.
Executetransaction
executesatransactiondb._executeTransaction(object)
Executesaserver-sidetransaction,asspecifiedbyobject.
objectmusthavethefollowingattributes:
collections:asub-objectthatdefineswhichcollectionswillbeusedinthetransaction.collectionscanhavetheseattributes:read:asinglecollectionoralistofcollectionsthatwillbeusedinthetransactioninread-onlymodewrite:asinglecollectionoralistofcollectionsthatwillbeusedinthetransactioninwriteorreadmode.
action:aJavascriptfunctionorastringwithJavascriptcodecontainingalltheinstructionstobeexecutedinsidethetransaction.Ifthecoderunsthroughsuccessfully,thetransactionwillbecommittedattheend.Ifthecodethrowsanexception,thetransactionwillberolledbackandalldatabaseoperationswillberolledback.
Additionally,objectcanhavethefollowingoptionalattributes:
waitForSync:booleanflagindicatingwhetherthetransactionisforcedtobesynchronous.lockTimeout:anumericvaluethatcanbeusedtosetatimeoutforwaitingoncollectionlocks.Ifnotspecified,adefaultvaluewillbeused.SettinglockTimeoutto0willmakeArangoDBnottimeoutwaitingforalock.params:optionalargumentspassedtothefunctionspecifiedinaction.
ThefollowingattributescanbeusedfortransactionsintheRocksDBstorageengine:
maxTransactionSize:transactionsizelimitinbytesintermediateCommitSize:maximumtotalsizeofoperationsafterwhichanintermediatecommitisperformedautomaticallyintermediateCommitCount:maximumnumberofoperationsafterwhichanintermediatecommitisperformedautomatically
Declarationofcollections
Allcollectionswhicharetoparticipateinatransactionneedtobedeclaredbeforehand.Thisisanecessitytoensureproperlockingandisolation.
Collectionscanbeusedinatransactioninwritemodeorinread-onlymode.
Ifanydatamodificationoperationsaretobeexecuted,thecollectionmustbedeclaredforuseinwritemode.Thewritemodeallowsmodifyingandreadingdatafromthecollectionduringthetransaction(i.e.thewritemodeincludesthereadmode).
Contrary,usingacollectioninread-onlymodewillonlyallowperformingreadoperationsonacollection.Anyattempttowriteintoacollectionusedinread-onlymodewillmakethetransactionfail.
Transactioninvocation
293
Collectionsforatransactionaredeclaredbyprovidingtheminthecollectionsattributeoftheobjectpassedtothe_executeTransactionfunction.Thecollectionsattributehasthesub-attributesreadandwrite:
db._executeTransaction({
collections:{
write:["users","logins"],
read:["recommendations"]
}
});
readandwriteareoptionalattributes,andonlyneedtobespecifiediftheoperationsinsidethetransactionsdemandforit.
Thecontentsofreadorwritecaneachbelistsarrayscollectionnamesorasinglecollectionname(asastring):
db._executeTransaction({
collections:{
write:"users",
read:"recommendations"
}
});
Note:Itiscurrentlyoptionaltospecifycollectionsforread-onlyaccess.Evenwithoutspecifyingthem,itisstillpossibletoreadfromsuchcollectionsfromwithinatransaction,butwithrelaxedisolation.PleaserefertoTransactionsLockingformoredetails.
Inordertomakeatransactionfailwhenanon-declaredcollectionisusedinsideforreading,theoptionalallowImplicitsub-attributeofcollectionscanbesettofalse:
db._executeTransaction({
collections:{
read:"recommendations",
allowImplicit:false/*thisdisallowsreadaccesstoothercollections
thanspecified*/
},
action:function(){
vardb=require("@arangodb").db;
returndb.foobar.toArray();/*willfailbecausedb.foobarmustnotbeaccessed
forreadinginsidethistransaction*/
}
});
ThedefaultvalueforallowImplicitistrue.Write-accessingcollectionsthathavenotbeendeclaredinthecollectionsarrayisneverpossible,regardlessofthevalueofallowImplicit.
Declarationofdatamodificationandretrievaloperations
AlldatamodificationandretrievaloperationsthataretobeexecutedinsidethetransactionneedtobespecifiedinaJavascriptfunction,usingtheactionattribute:
db._executeTransaction({
collections:{
write:"users"
},
action:function(){
//alloperationsgohere
}
});
AnyvalidJavascriptcodeisallowedinsideactionbutthecodemayonlyaccessthecollectionsdeclaredincollections.actionmaybeaJavascriptfunctionasshownabove,orastringrepresentationofaJavascriptfunction:
db._executeTransaction({
collections:{
write:"users"
},
action:"function(){doSomething();}"
});
Transactioninvocation
294
Pleasenotethatanyoperationsspecifiedinactionwillbeexecutedontheserver,inaseparatescope.Variableswillbeboundlate.AccessinganyJavaScriptvariablesdefinedontheclient-sideorinsomeotherservercontextfrominsideatransactionmaynotwork.Instead,anyvariablesusedinsideactionshouldbedefinedinsideactionitself:
db._executeTransaction({
collections:{
write:"users"
},
action:function(){
vardb=require(...).db;
db.users.save({...});
}
});
Whenthecodeinsidetheactionattributeisexecuted,thetransactionisalreadystartedandallrequiredlockshavebeenacquired.Whenthecodeinsidetheactionattributefinishes,thetransactionwillautomaticallycommit.Thereisnoexplicitcommitcommand.
Tomakeatransactionabortandrollbackallchanges,anexceptionneedstobethrownandnotcaughtinsidethetransaction:
db._executeTransaction({
collections:{
write:"users"
},
action:function(){
vardb=require("@arangodb").db;
db.users.save({_key:"hello"});
//willabortandrollbackthetransaction
throw"doh!";
}
});
Thereisnoexplicitabortorrollbackcommand.
Asmentionedearlier,atransactionwillcommitautomaticallywhentheendoftheactionfunctionisreachedandnoexceptionhasbeenthrown.Inthiscase,theusercanreturnanylegalJavaScriptvaluefromthefunction:
db._executeTransaction({
collections:{
write:"users"
},
action:function(){
vardb=require("@arangodb").db;
db.users.save({_key:"hello"});
//willcommitthetransactionandreturnthevalue"hello"
return"hello";
}
});
Customexceptions
Onemaywishtodefinecustomexceptionsinsideofatransaction.Tohavetheexceptionpropagateupwardsproperly,pleasethrowananinstanceofbaseJavaScriptErrorclassoraderivative.Tospecifyanerrornumber,includeitastheerrorNumberfield.Asanexample:
db._executeTransaction({
collections:{},
action:function(){
varerr=newError('Myerrorcontext');
err.errorNumber=1234;
throwerr;
}
});
Transactioninvocation
295
Note:Inpreviousversions,customexceptionswhichdidnothaveanError-likeformweresimplyconvertedtostringsandexposedintheexceptionfieldofthereturnederror.Thisisnolongerthecase,asithadthepotentialtoleakunwantedinformationifimproperlyused.
Examples
Thefirstexamplewillwrite3documentsintoacollectionnamedc1.Thec1collectionneedstobedeclaredinthewriteattributeofthecollectionsattributepassedtotheexecuteTransactionfunction.
Theactionattributecontainstheactualtransactioncodetobeexecuted.Thiscodecontainsalldatamodificationoperations(3inthisexample).
//setup
db._create("c1");
db._executeTransaction({
collections:{
write:["c1"]
},
action:function(){
vardb=require("@arangodb").db;
db.c1.save({_key:"key1"});
db.c1.save({_key:"key2"});
db.c1.save({_key:"key3"});
}
});
db.c1.count();//3
Abortingthetransactionbythrowinganexceptionintheactionfunctionwillrevertallchanges,soasifthetransactionneverhappened:
//setup
db._create("c1");
db._executeTransaction({
collections:{
write:["c1"]
},
action:function(){
vardb=require("@arangodb").db;
db.c1.save({_key:"key1"});
db.c1.count();//1
db.c1.save({_key:"key2"});
db.c1.count();//2
throw"doh!";
}
});
db.c1.count();//0
Theautomaticrollbackisalsoexecutedwhenaninternalexceptionisthrownatsomepointduringtransactionexecution:
//setup
db._create("c1");
db._executeTransaction({
collections:{
write:["c1"]
},
action:function(){
vardb=require("@arangodb").db;
db.c1.save({_key:"key1"});
//willthrowduplicateakeyerror,notexplicitlyrequestedbytheuser
db.c1.save({_key:"key1"});
//we'llnevergethere...
}
});
db.c1.count();//0
Transactioninvocation
296
Asrequiredbytheconsistencyprinciple,abortingorrollingbackatransactionwillalsorestoresecondaryindexestothestateattransactionstart.
Cross-collectiontransactions
There'salsothepossibilitytorunatransactionacrossmultiplecollections.Inthiscase,multiplecollectionsneedtobedeclaredinthecollectionsattribute,e.g.:
//setup
db._create("c1");
db._create("c2");
db._executeTransaction({
collections:{
write:["c1","c2"]
},
action:function(){
vardb=require("@arangodb").db;
db.c1.save({_key:"key1"});
db.c2.save({_key:"key2"});
}
});
db.c1.count();//1
db.c2.count();//1
Again,throwinganexceptionfrominsidetheactionfunctionwillmakethetransactionabortandrollbackallchangesinallcollections:
//setup
db._create("c1");
db._create("c2");
db._executeTransaction({
collections:{
write:["c1","c2"]
},
action:function(){
vardb=require("@arangodb").db;
for(vari=0;i<100;++i){
db.c1.save({_key:"key"+i});
db.c2.save({_key:"key"+i});
}
db.c1.count();//100
db.c2.count();//100
//abort
throw"doh!"
}
});
db.c1.count();//0
db.c2.count();//0
Transactioninvocation
297
PassingparameterstotransactionsArbitraryparameterscanbepassedtotransactionsbysettingtheparamsattributewhendeclaringthetransaction.Thisfeatureishandytore-usethesametransactioncodeformultiplecallsbutwithdifferentparameters.
Abasicexample:
db._executeTransaction({
collections:{},
action:function(params){
returnparams[1];
},
params:[1,2,3]
});
Theaboveexamplewillreturn2.
Someexamplethatusescollections:
db._executeTransaction({
collections:{
write:"users",
read:["c1","c2"]
},
action:function(params){
vardb=require('@arangodb').db;
vardoc=db.c1.document(params['c1Key']);
db.users.save(doc);
doc=db.c2.document(params['c2Key']);
db.users.save(doc);
},
params:{
c1Key:"foo",
c2Key:"bar"
}
});
Passingparameters
298
LockingandIsolationTransactionsneedtospecifyfromwhichcollectionstheywillreaddataandwhichcollectionstheyintenddomodify.Thiscanbedonebysettingtheread,write,orexclusiveattributesinthecollectionsattributeofthetransaction:
db._executeTransaction({
collections:{
read:"users",
write:["test","log"]
},
action:function(){
constdb=require("@arangodb").db;
db.users.toArray().forEach(function(doc){
db.log.insert({value:"removeduser:"+doc.name});
db.test.remove(doc._key);
});
}
});
writeheremeanswriteaccesstothecollection,andalsoincludesanyreadaccesses.exclusiveisasynonymforwriteintheMMFilesengine,becausebothexclusiveandwritewillacquirecollection-levellocksinthisengine.IntheRocksDBengine,exclusivemeansexclusivewriteaccesstothecollection,andwritemeans(shared)writeaccesstothecollection,whichcanbeinterleavedwithwriteaccessesbyotherconcurrenttransactions.
MMFilesengine
TheMMFilesengineusesthefollowinglockingmechanismstoserializetransactionsonthesamedata:
Allcollectionsspecifiedinthecollectionsattributearelockedintherequestedmode(readorwrite)attransactionstart.Lockingofmultiplecollectionsisperformedinalphabeticalorder.Whenatransactioncommitsorrollsback,alllocksarereleasedinreverseorder.Thelockingorderisdeterministictoavoiddeadlocks.
Whilelocksareheld,modificationsbyothertransactionstothecollectionsparticipatinginthetransactionareprevented.Atransactionwillthusseeaconsistentviewoftheparticipatingcollections'data.
Additionally,atransactionwillnotbeinterruptedorinterleavedwithanyotherongoingoperationsonthesamecollection.Thismeanseachtransactionwillruninisolation.Atransactionshouldneverseeuncommittedorrolledbackmodificationsbyothertransactions.Additionally,readsinsideatransactionarerepeatable.
Notethattheaboveistrueonlyforallcollectionsthataredeclaredinthecollectionsattributeofthetransaction.
RocksDBengine
TheRocksDBenginedoesnotlockanycollectionsparticipatinginatransactionforread.Readoperationscanruninparalleltootherreadorwriteoperationsonthesamecollections.
Forallcollectionsthatareusedinwritemode,theRocksDBenginewillinternallyacquirea(shared)readlock.Thismeansthatmanywriterscanmodifydatainthesamecollectioninparallel(andalsoruninparalleltoongoingreads).However,iftwoconcurrenttransactionsattempttomodifythesamedocumentorindexentry,therewillbeawrite-writeconflict,andoneofthetransactionswillabortwitherror1200("conflict").Itisthenuptoclientapplicationstoretrythefailedtransactionoracceptthefailure.
Inordertoguardlong-runningorcomplextransactionsagainstconcurrentoperationsonthesamedata,theRocksDBengineallowstoaccesscollectionsinexclusivemode.Exclusiveaccesseswillinternallyacquireawrite-lockonthecollections,sotheyarenotexecutedinparallelwithanyotherwriteoperations.Readoperationscanstillbecarriedoutbyotherconcurrenttransactions.
Lazilyaddingcollections
Theremightbesituationswhendeclaringallcollectionsaprioriisnotpossible,forexample,becausefurthercollectionsaredeterminedbyadynamicAQLqueryinsidethetransaction,forexampleaqueryusingAQLgraphtraversal.
Lockingandisolation
299
Inthiscase,itwouldbeimpossibletoknowbeforehandwhichcollectiontolock,andthusitislegaltonotdeclarecollectionsthatwillbeaccessedinthetransactioninread-onlymode.Accessinganon-declaredcollectioninread-onlymodeduringatransactionwilladdthecollectiontothetransactionlazily,andfetchdatafromthecollectionasusual.However,asthecollectionisaddedlazily,thereisnoisolationfromotherconcurrentoperationsortransactions.Readsfromsuchcollectionsarepotentiallynon-repeatable.
Examples:
db._executeTransaction({
collections:{
read:"users"
},
action:function(){
constdb=require("@arangodb").db;
/*ExecuteanAQLquerythattraversesagraphstartingata"users"vertex.
Itisyetunknownintowhichothercollectionsthequerymighttraverse*/
db._createStatement({
query:`FORvINANY"users/1234"connectionsRETURNv`
}).execute().toArray().forEach(function(d){
/*...*/
});
}
});
Thisautomaticlazyadditionofcollectionstoatransactionalsointroducesthepossibilityofdeadlocks.Deadlocksmayoccurifthereareconcurrenttransactionsthattrytoacquirelocksonthesamecollectionslazily.
Inordertomakeatransactionfailwhenanon-declaredcollectionisusedinsideatransactionforreading,theoptionalallowImplicitsub-attributeofcollectionscanbesettofalse:
db._executeTransaction({
collections:{
read:"users",
allowImplicit:false
},
action:function(){
/*Thebelowquerywillnowfailbecausethecollection"connections"hasnot
beenspecifiedinthelistofcollectionsusedbythetransaction*/
constdb=require("@arangodb").db;
db._createStatement({
query:`FORvINANY"users/1234"connectionsRETURNv`
}).execute().toArray().forEach(function(d){
/*...*/
});
}
});
ThedefaultvalueforallowImplicitistrue.Write-accessingcollectionsthathavenotbeendeclaredinthecollectionsarrayisneverpossible,regardlessofthevalueofallowImplicit.
Ifusers/1234hasanedgeinconnections,linkingittoanotherdocumentintheuserscollection,thenthefollowingexplicitdeclarationwillwork:
db._executeTransaction({
collections:{
read:["users","connections"],
allowImplicit:false
},
/*...*/
Iftheedgepointstoadocumentinanothercollectionhowever,thenthequerywillfail,unlessthatothercollectionisaddedtothedeclarationaswell.
Notethatifadocumenthandleisusedasstartingpointforatraversal,e.g.FORvINANY"users/not_linked"...orFORvINANY{_id:"users/not_linked"}...,thennoerrorisraisedinthecaseofthestartvertexnothavinganyedgestofollow,withallowImplicit:falseandusersnotbeingdeclaredforreadaccess.AQLonlyseesastringanddoesnotconsideritareadaccess,unlessthereareedgesconnectedtoit.FORvINANYDOCUMENT("users/not_linked")...willfailevenwithoutedges,asitisalwaysconsideredtobeareadaccesstotheuserscollection.
Lockingandisolation
300
DeadlocksandDeadlockdetection
Adeadlockisasituationinwhichtwoormoreconcurrentoperations(usertransactionsorAQLqueries)trytoaccessthesameresources(collections,documents)andneedtowaitfortheotherstofinish,butnoneofthemcanmakeanyprogress.
AgoodexampleforadeadlockistwoconcurrentlyexecutingtransactionsT1andT2thattrytoaccessthesamecollectionsbutthatneedtowaitforeachother.Inthisexample,transactionT1willwritetocollectionc1,butwillalsoreaddocumentsfromcollectionc2withoutannouncingit:
db._executeTransaction({
collections:{
write:"c1"
},
action:function(){
constdb=require("@arangodb").db;
/*writeintoc1(announced)*/
db.c1.insert({foo:"bar"});
/*someoperationherethattakeslongtoexecute...*/
/*readfromc2(unannounced)*/
db.c2.toArray();
}
});
TransactionT2announcestowriteintocollectionc2,butwillalsoreaddocumentsfromcollectionc1withoutannouncingit:
db._executeTransaction({
collections:{
write:"c2"
},
action:function(){
vardb=require("@arangodb").db;
/*writeintoc2(announced)*/
db.c2.insert({bar:"baz"});
/*someoperationherethattakeslongtoexecute...*/
/*readfromc1(unannounced)*/
db.c1.toArray();
}
});
Intheaboveexample,adeadlockwilloccuriftransactionT1andT2havebothacquiredtheirwritelocks(T1forcollectionc1andT2forcollectionc2)andarethentryingtoreadfromtheotherother(T1willreadfromc2,T2willreadfromc1).T1willthentrytoacquirethereadlockoncollectionc2,whichispreventedbytransactionT2.T2howeverwillwaitforthereadlockoncollectionc1,whichispreventedbytransactionT1.
Incaseofsuchdeadlock,therewouldbenoprogressforanyoftheinvolvedtransactions,andnoneoftheinvolvedtransactionscouldevercomplete.Thisiscompletelyundesirable,sotheautomaticdeadlockdetectionmechanisminArangoDBwillautomaticallyabortoneofthetransactionsinvolvedinsuchdeadlock.Abortingmeansthatallchangesdonebythetransactionwillberolledbackanderror29(deadlockdetected)willbethrown.
Clientcode(AQLqueries,usertransactions)thataccessesmorethanonecollectionshouldbeawareofthepotentialofdeadlocksandshouldhandletheerror29(deadlockdetected)properly,eitherbypassingtheexceptiontothecallerorretryingtheoperation.
Toavoidbothdeadlocksandnon-repeatablereads,allcollectionsusedinatransactionshouldbespecifiedinthecollectionsattributewhenknowninadvance.Incasethisisnotpossiblebecausecollectionsareaddeddynamicallyinsidethetransaction,deadlocksmayoccurandthedeadlockdetectionmaykickinandabortthetransaction.
TheRocksDBengineusesdocument-levellocksandthereforewillnothaveadeadlockproblemoncollectionlevel.Iftwoconcurrenttransactionshowevermodifythesamedocumentsorindexentries,theRocksDBenginewillsignalawrite-writeconflictandabortoneofthetransactionswitherror1200("conflict")automatically.
Lockingandisolation
301
DurabilityTransactionsareexecutedinmainmemoryfirstuntilthereiseitherarollbackoracommit.Onrollback,nodatawillbewrittentodisk,buttheoperationsfromthetransactionwillbereversedinmemory.
Oncommit,allmodificationsdoneinthetransactionwillbewrittentothecollectiondatafiles.ThesewriteswillbesynchronizedtodiskifanyofthemodifiedcollectionshasthewaitForSyncpropertysettotrue,orifanyindividualoperationinthetransactionwasexecutedwiththewaitForSyncattribute.Additionally,transactionsthatmodifydatainmorethanonecollectionareautomaticallysynchronizedtodisk.Thissynchronizationisdonetonotonlyensuredurability,buttoalsoensureconsistencyincaseofaservercrash.
Thatmeansifyouonlymodifydatainasinglecollection,andthatcollectionhasitswaitForSyncpropertysettofalse,thewholetransactionwillnotbesynchronizedtodiskinstantly,butwithasmalldelay.
Thereisthusthepotentialriskoflosingdatabetweenthecommitofthetransactionandtheactual(delayed)disksynchronization.ThisisthesameaswritingintocollectionsthathavethewaitForSyncpropertysettofalseoutsideofatransaction.IncaseofacrashwithwaitForSyncsettofalse,theoperationsperformedinthetransactionwilleitherbevisiblecompletelyornotatall,dependingonwhetherthedelayedsynchronizationhadkickedinornot.
ToensuredurabilityoftransactionsonacollectionthathavethewaitForSyncpropertysettofalse,youcansetthewaitForSyncattributeoftheobjectthatispassedtoexecuteTransaction.ThiswillforceasynchronizationofthetransactiontodiskevenforcollectionsthathavewaitForSyncsettofalse:
db._executeTransaction({
collections:{
write:"users"
},
waitForSync:true,
action:function(){...}
});
Analternativeistoperformanoperationwithanexplicitsyncrequestinatransaction,e.g.
db.users.save({_key:"1234"},true);
Inthiscase,thetruevaluewillmakethewholetransactionbesynchronizedtodiskatthecommit.
Inanycase,ArangoDBwillgiveusersthechoiceofwhetherornottheywantfulldurabilityforsinglecollectiontransactions.Usingthedelayedsynchronization(i.e.waitForSyncwithavalueoffalse)willpotentiallyincreasethroughputandperformanceoftransactions,butwillintroducetheriskoflosingthelastcommittedtransactionsinthecaseofacrash.
Incontrast,transactionsthatmodifydatainmorethanonecollectionareautomaticallysynchronizedtodisk.Thiscomesatthecostofseveraldisksync.Foramulti-collectiontransaction,thecalltothe_executeTransactionfunctionwillonlyreturnafterthedataofallmodifiedcollectionshasbeensynchronizedtodiskandthetransactionhasbeenmadefullydurable.Thisnotonlyreducestheriskoflosingdataincaseofacrashbutalsoensuresconsistencyafterarestart.
Incaseofaservercrash,anymulti-collectiontransactionsthatwerenotyetcommittedorinpreparationtobecommittedwillberolledbackonserverrestart.
Formulti-collectiontransactions,therewillbeatleastonedisksyncoperationpermodifiedcollection.Multi-collectiontransactionsthushaveapotentiallyhighercostthansinglecollectiontransactions.Thereisnoconfigurationtoturnoffdisksynchronizationformulti-collectiontransactionsinArangoDB.Thedisksyncspeedofthesystemwillthusbethemostimportantfactorfortheperformanceofmulti-collectiontransactions.
Durability
303
Limitations
InGeneral
TransactionsinArangoDBhavebeendesignedwithparticularusecasesinmind.Theywillbemainlyusefulforshortandsmalldataretrievaland/ormodificationoperations.
Theimplementationisnotoptimizedforverylong-runningorveryvoluminousoperations,andmaynotbeusableforthesecases.
Onelimitationisthatatransactionoperationinformationmustfitintomainmemory.Thetransactioninformationconsistsofrecordpointers,revisionnumbersandrollbackinformation.Theactualdatamodificationoperationsofatransactionarewrittentothewrite-aheadloganddonotneedtofitentirelyintomainmemory.
Ongoingtransactionswillalsopreventthewrite-aheadlogsfrombeingfullygarbage-collected.Informationinthewrite-aheadlogfilescannotbewrittentocollectiondatafilesorbediscardedwhiletransactionsareongoing.
Toensureprogressofthewrite-aheadloggarbagecollection,transactionsshouldbekeptassmallaspossible,andbigtransactionsshouldbesplitintomultiplesmallertransactions.
TransactionsinArangoDBcannotbenested,i.e.atransactionmustnotstartanothertransaction.Ifanattemptismadetocallatransactionfrominsidearunningtransaction,theserverwillthrowerror1651(nestedtransactionsdetected).
ItisalsodisallowedtoexecuteusertransactiononsomeofArangoDB'sownsystemcollections.Thisshouldn'tbeaproblemforregularusageassystemcollectionswillnotcontainuserdataandthereisnoneedtoaccessthemfromwithinausertransaction.
Someoperationsarenotallowedinsidetransactionsingeneral:
creationanddeletionofdatabases(db._createDatabase(),db._dropDatabase())creationanddeletionofcollections(db._create(),db._drop(),db.<collection>.rename())creationanddeletionofindexes(db.<collection>.ensureIndex(),db.<collection>.dropIndex())
Ifanattemptismadetocarryoutanyoftheseoperationsduringatransaction,ArangoDBwillabortthetransactionwitherrorcode1653(disallowedoperationinsidetransaction).
Finally,allcollectionsthatmaybemodifiedduringatransactionmustbedeclaredbeforehand,i.e.usingthecollectionsattributeoftheobjectpassedtothe_executeTransactionfunction.Ifanyattemptismadetocarryoutadatamodificationoperationonacollectionthatwasnotdeclaredinthecollectionsattribute,thetransactionwillbeabortedandArangoDBwillthrowerror1652unregisteredcollectionusedintransaction.Itislegaltonotdeclareread-onlycollections,butthisshouldbeavoidedifpossibletoreducetheprobabilityofdeadlocksandnon-repeatablereads.
PleaserefertoLockingandIsolationformoredetails.
InClustersUsingasingleinstanceofArangoDB,multi-document/multi-collectionqueriesareguaranteedtobefullyACID.ThisismorethanmanyotherNoSQLdatabasesystemssupport.Inclustermode,single-documentoperationsarealsofullyACID.Multi-document/multi-collectionqueriesinaclusterarenotACID,whichisequallythecasewithcompetingdatabasesystems.TransactionsinaclusterwillbesupportedinafutureversionofArangoDBandmaketheseoperationsfullyACIDaswell.Notethatfornon-shardedcollectionsinacluster,thetransactionalpropertiesofasingleserverapply(fullyACID).
TransactionsintheRocksDBstorageengineDataofongoingtransactionsisstoredinRAM.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatacreatedormodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.
ThefollowingglobaloptionscanbeusedtocontroltheRAMusageandautomaticintermediatecommitsfortheRocksDBengine:
Limitations
304
--rocksdb.max-transaction-size
Transactionsizelimit(inbytes).TransactionsstoreallkeysandvaluesinRAM,solargetransactionsruntheriskofcausingout-of-memorysitations.Thissettingallowsyoutoensurethatdoesnothappenbylimitingthesizeofanyindividualtransaction.TransactionswhoseoperationswouldconsumemoreRAMthanthisthresholdvaluewillabortautomaticallywitherror32("resourcelimitexceeded").
--rocksdb.intermediate-commit-size
Ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.
--rocksdb.intermediate-commit-count
Ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.
Theabovevaluescanalsobeadjustedpertransaction,bysettingthefollowingattributesinthecalltodb._executeTransaction():
maxTransactionSize:transactionsizelimitinbytesintermediateCommitSize:maximumtotalsizeofoperationsafterwhichanintermediatecommitisperformedautomaticallyintermediateCommitCount:maximumnumberofoperationsafterwhichanintermediatecommitisperformedautomatically
Limitations
305
DeploymentInthischapterwedescribevariouspossibilitiestodeployArangoDB.Inparticularfortheclustermode,therearedifferentwaysandwewanttohighlighttheiradvantagesanddisadvantages.Weevendocumentindetail,howtosetupaclusterbysimplystartingvariousArangoDBprocessesondifferentmachines,eitherdirectlyorusingDockercontainers.
SingleinstanceCluster
DC/OS,ApacheMesosandMarathonGeneric&DockerAdvancedTopics
StandaloneAgencyTestsetuponalocalmachineStartingprocessesondifferentmachinesLaunchinganArangoDBclusterusingDockercontainers
MultipleDatacenters
Deployment
306
Singleinstancedeployment
ThelatestofficialbuildsofArangoDBforallsupportedoperatingsystemsmaybeobtainedfromhttps://www.arangodb.com/download/.
Linuxremarks
Besidestheofficialimageswhichareprovidedforthemostpopularlinuxdistributionstherearealsoavarietyofunofficialimagesprovidedbythecommunity.Wearetrackingmostofthecommunitycontributions(includingneworupdatedimages)inournewsletter:
https://www.arangodb.com/category/newsletter/
Windowsremarks
PleasenotethatArangoDBwillonlyworkon64bit.
Docker
ThesimplestwaytodeployArangoDBisusingDocker.TogetageneralunderstandingofDockerhavealookattheirexcellentdocumentation.
Authentication
TostarttheofficialDockercontaineryouwillhavetodecideonanauthenticationmethod.Otherwisethecontainerwon'tstart.
ProvideoneoftheargumentstoDockerasanenvironmentvariable.
Therearethreeoptions:
1. ARANGO_NO_AUTH=1
Disableauthenticationcompletely.Usefulforlocaltestingorforoperatinginatrustednetwork(withoutapublicinterface).
2. ARANGO_ROOT_PASSWORD=password
StartArangoDBwiththegivenpasswordforroot
3. ARANGO_RANDOM_ROOT_PASSWORD=1
LetArangoDBgeneratearandomrootpassword
Togetgoingquickly:
dockerrun-eARANGO_RANDOM_ROOT_PASSWORD=1arangodb/arangodb
ForanindepthguideaboutDockerandArangoDBpleasechecktheofficialdocumentation:https://hub.docker.com/r/arangodb/arangodb/.Notethatweareusingtheimagearangodb/arangodbherewhichisalwaysthemostcurrentone.Thereisalsothe"official"onecalledarangodbwhosedocumentationishere:https://hub.docker.com/_/arangodb/
Singleinstance
307
ClusterMesos,DC/OS:DistributeddeploymentusingApacheMesos
Generic&Docker:AutomaticnativeclusterswithArangoDBStarter
AdvancedTopics:StandaloneAgency,local/distributed/Dockerclusters
Cluster
308
DistributeddeploymentusingApacheMesos
ArangoDBhasasophisticatedandyeteasywaytouseclustermode.Toleveragethefullclusterfeatureset(monitoring,scaling,automaticfailoverandautomaticreplacementoffailednodes)youhavetorunArangoDBonsomekindofclustermanagementsystem.CurrentlyArangoDBreliesonApacheMesosinthatmatter.Mesosisaclusteroperatingsystemwhichpowerssomeoftheworldsbiggestdatacentersrunningseveralthousandsofnodes.
DC/OS
DC/OSistherecommendedwaytoinstallaclusterasiteasesmuchoftheprocesstoinstallaMesoscluster.YoucandeployitveryquicklyonavarietyofcloudhostersorsetupyourownDC/OSlocally.DC/OSisasetoftoolsbuiltontopofApacheMesos.ApacheMesosisasocalled"DistributedClusterOperationSystem"andthecoreofDC/OS.ApacheMesoshastheconceptofsocalledpersistentvolumeswhichmakeitperfectlysuitableforadatabase.
Installing
FirstprepareaDC/OSclusterbygoingtohttps://dcos.ioandfollowingtheinstructionsthere.
DC/OScomeswithitsownpackagemanagement.Packagescanbeinstalledfromthesocalled"Universe".AsanofficialDC/OSpartnerArangoDBcanbeinstalledfromtherestraightaway.
1. InstallingviaDC/OSUI
i. OpenyourbrowserandgototheDC/OSadmininterfaceii. Openthe"Universe"tabiii. Locatearangodbandhit"InstallPackage"iv. Press"InstallPackage"
2. InstallingviatheDC/OScommandline
i. Installthedcoscliii. Openaterminalandissuedcosinstallarangodb
Bothoptionsareessentiallydoingthesameinthebackground.BotharestartingArangoDBwithitsdefaultoptionsset.
Toreviewthedefaultoptionsusingthewebinterfacesimplyclick"AdvancedInstallation"inthewebinterface.Thereyouwillfindalistofoptionsincludingsomeexplanation.
ToreviewthedefaultoptionsusingtheCLIfirsttypedcospackagedescribe--configarangodb.Thiswillgiveyouaflatlistofdefaultsettings.
Togetanexplanationofthevariouscommandlineoptionspleasecheckthelatestoptionshere(choosethemostrecentnumberandhavealookatconfig.json):
https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/A/arangodb
AfterinstallationDC/OSwillstartdeployingtheArangoDBclusterontheDC/OScluster.YoucanwatchArangoDBstartingonthe"Services"tabinthewebinterface.OnceitislistedashealthyclickthelinknexttoitandyoushouldseetheArangoDBwebinterface.
ArangoDBMesosframework
AssoonasArangoDBwasdeployedMesoswillkeepyourclusterrunning.ThewebinterfacehasmanymonitoringfacilitiessobesuretomakeyourselffamiliarwiththeDC/OSwebinterface.AsafaulttolerantsystemMesoswilltakecareofmostfailurescenariosautomatically.MesosdoesthatbyrunningArangoDBasasocalled"framework".ThisframeworkhasbeenspecificallybuilttokeepArangoDBrunninginahealthyconditionontheMesoscluster.Fromtimetotimeataskmightfail.TheArangoDBframeworkwillthentakecareofreschedulingthefailedtask.Asitknowsabouttheveryspecificsofeachclustertaskanditsroleitwillautomaticallytakecareofmostfailurescenarios.
Toinspectwhattheframeworkisdoinggotohttp://web-interface-url/mesosinyourbrowser.Locatethetask"arangodb"andinspectstderrinthe"Sandbox".Thiscanbeofinterestforexamplewhenaslavegotlostandtheframeworkisreschedulingthetask.
Mesos,DC/OS
309
UsingArangoDB
TouseArangoDBasadatastoreinyourDC/OSclusteryoucanfacilitatetheservicediscoveryofDC/OS.AssumingyoudeployedastandardArangoDBclusterthemesosdnswillknowaboutarangodb.mesos.BydoingaSRVDNSrequest(checkthedocumentationofmesosdns)youcanfindouttheportwheretheinternalHAProxyofArangoDBisrunning.ThiswillofferaroundrobinloadbalancertoaccessallArangoDBcoordinators.
ScalingArangoDB
TochangethesettingsofyourArangoDBClusteraccesstheArangoDBUIandhit"Nodes".Onthescaletabyouwillhavetheabilitytoscaleyourclusterupanddown.
AfterchangingthesettingstheArangoDBframeworkwilltakecareoftherest.ScalingyourclusterupisgenerallyastraightforwardoperationasMesoswillsimplylaunchanothertaskandbedonewithit.Scalingdownisabitmorecomplicatedasthedatafirsthastobemovedtosomeotherplacesothatwillnaturallytakesomewhatlonger.
Pleasenotethatscalingoperationsmightnotalwayswork.ForexampleiftheunderlyingMesosclusteriscompletelysaturatedwithitsrunningtasksscalingupwillnotbepossible.ScalingdownmightalsofailduetotheclusternotbeingabletomoveallshardsofaDBServertoanewdestinationbecauseofsizelimitations.BesuretochecktheoutputoftheArangoDBframework.
Deinstallation
DeinstallingArangoDBisabitmoredifficultasthereismuchstatebeingkeptintheMesosclusterwhichisnotautomaticallycleanedup.Todeinstallfromthecommandlineusethefollowingoneliner:
dcosarangodbuninstall;dcospackageuninstallarangodb
Thiswillfirstcleanupthestateintheclusterandthenuninstallarangodb.
arangodb-cleanup-framework
Shouldyouforgettocleanupthestateyoucandosolaterbyusingthearangodb-cleanup-frameworkcontainer.Otherwiseyoumightnotbeabletodeployanewarangodbinstallation.
ThecleanupframeworkwillannounceitselfasanormalArangoDB.MesoswillrecognizethisandofferallpersistentvolumesitstillhasforArangoDBtothisframework.Thecleanupframeworkwillthenproperlyfreethepersistentvolumes.Finallyitwillcleanupanystateleftinzookeeper(thecentralconfigurationmanagerinaMesoscluster).
Todeploythecleanupframework,followtheinstructionsinthegithubrepository.AfterdeploymentwatchtheoutputinthesandboxoftheMesoswebinterface.Afterawhilethereshouldn'tbeanypersistentresourceoffersanymoreaseverythingwascleanedup.AfterthatyoucandeletethecleanupframeworkagainviaMarathon.
ApacheMesosandMarathon
YoucanalsoinstallArangoDBonabareApacheMesosclusterprovidedthatMarathonisrunningonit.
Doingsohasthefollowingdownsides:
1. ManualMesosclustersetup2. Youneedtoimplementyourownservicediscovery3. Youaremissingthedcoscli4. Installationanddeinstallationaretedious5. YouneedtosetupsomekindofproxytunneltoaccessArangoDBfromtheoutside6. Sparsemonitoringcapabilities
HoweverthesearethingswhichdonotinfluenceArangoDBitselfandoperatingyourclusterlikethisisfullysupported.
InstallingviaMarathon
ToinstallArangoDBviamarathonyouneedaproperconfigfile:
{
Mesos,DC/OS
310
"id":"arangodb",
"cpus":0.25,
"mem":256.0,
"ports":[0,0,0],
"instances":1,
"args":[
"framework",
"--framework_name=arangodb",
"--master=zk://172.17.0.2:2181/mesos",
"--zk=zk://172.17.0.2:2181/arangodb",
"--user=",
"--principal=pri",
"--role=arangodb",
"--mode=cluster",
"--async_replication=true",
"--minimal_resources_agent=mem(*):512;cpus(*):0.25;disk(*):512",
"--minimal_resources_dbserver=mem(*):512;cpus(*):0.25;disk(*):1024",
"--minimal_resources_secondary=mem(*):512;cpus(*):0.25;disk(*):1024",
"--minimal_resources_coordinator=mem(*):512;cpus(*):0.25;disk(*):1024",
"--nr_agents=1",
"--nr_dbservers=2",
"--nr_coordinators=2",
"--failover_timeout=86400",
"--arangodb_image=arangodb/arangodb-mesos:3.1",
"--secondaries_with_dbservers=false",
"--coordinators_with_dbservers=false"
],
"container":{
"type":"DOCKER",
"docker":{
"image":"arangodb/arangodb-mesos-framework:3.1",
"network":"HOST"
}
},
"healthChecks":[
{
"protocol":"HTTP",
"path":"/framework/v1/health.json",
"gracePeriodSeconds":3,
"intervalSeconds":10,
"portIndex":0,
"timeoutSeconds":10,
"maxConsecutiveFailures":0
}
]
}
Carefullyreviewthesettings(especiallytheIPsandtheresources).ThenyoucandeploytoMarathon:
curl-XPOST-H"Content-Type:application/json"http://url-of-marathon/v2/[email protected]
AlternativelyusethewebinterfaceofMarathontodeployArangoDB.IthasaJSONmodeandyoucanusetheaboveconfigurationfile.
DeinstallationviaMarathon
AswithDC/OSyoufirstneedtoproperlycleanupanystateleftovers.
TheeasiestistosimplydeleteArangoDBandthendeploythecleanup-framework(seesectionarangodb-cleanup-framework).
Configurationoptions
TheArangodbMesosframeworkhasatonofdifferentoptionswhicharelistedanddescribedhere:https://github.com/arangodb/arangodb-mesos-framework/tree/3.1
Mesos,DC/OS
311
AutomaticnativeClusters
SimilarlytohowtheMesosframeworkarangesanArangoDBclusterinaDC/OSenvironmentforyou,arangodbcandothisforyouinaplainenvironment.
Byinvokingthefirstarangodbyoulaunchaprimarynode.Itwillbindanetworkport,andoutputthecommandsyouneedtocut'n'pasteintotheothernodes.Let'sreviewtheprocessofsuchastartuponthreehostsnamedh01,h02,andh03:
arangodb@h01~>arangodb--ownAddressh01:4000
2017/06/1214:59:38Startingarangodbversion0.5.0+git,build5f97368
2017/06/1214:59:38ServingasmasterwithID'52698769'onh01:4000...
2017/06/1214:59:38Waitingfor3serverstoshowup.
2017/06/1214:59:38Usethefollowingcommandstostartotherservers:
arangodb--dataDir=./db2--joinh01:4000
arangodb--dataDir=./db3--joinh01:4000
2017/06/1214:59:38Listeningon0.0.0.0:4000(h01:4000)
Soyoucutthelinesarangodb--data.dir=./db2--starter.join127.0.0.1andexecutethemfortheothernodes.Ifyourunitonanothernodeonyournetwork,replacethe--starter.join127.0.0.1bythepublicIPofthefirsthost.
arangodbh02~>arangodb--dataDir=./db2--joinh01:4000
2017/06/1214:48:50Startingarangodbversion0.5.0+git,build5f97368
2017/06/1214:48:50Contactingmasterh01:4000...
2017/06/1214:48:50Waitingfor3serverstoshowup...
2017/06/1214:48:50Listeningon0.0.0.0:4000(:4000)
arangodbh03~>arangodb--dataDir=./db3--joinh01:4000
2017/06/1214:48:50Startingarangodbversion0.5.0+git,build5f97368
2017/06/1214:48:50Contactingmasterh01:4000...
2017/06/1214:48:50Waitingfor3serverstoshowup...
2017/06/1214:48:50Listeningon0.0.0.0:4000(:4000)
Oncethetwootherprocessesjoinedthecluster,andstartedtheirArangoDBserverprocesses(thismaytakeawhiledependingonyoursystem),itwillinformyouwheretoconnecttheClusterfromaBrowser,shelloryourprogramm:
...
2017/06/1214:55:21coordinatorupandrunning.
Atthispointyoumayaccessyourclusterateithercoordinatorendpoint,http://h01:4002/,http://h02:4002/orhttp://h03:4002/.
AutomaticnativelocaltestClusters
Ifyouonlywantalocaltestcluster,youcanrunasinglestarterwiththe--starter.localargument.Itwillstarta3"machine"clusteronyourlocalPC.
arangodb--starter.local
Note.AlocalclusterisintendedonlyfortestpurposessinceafailureofasinglePCwillbringdowntheentirecluster.
AutomaticDockerClusters
ArangoDBStartercanalsobeusedtolaunchclustersbasedondockercontainers.Itsabitmorecomplicated,sinceyouneedtoprovideinformationaboutyourenvironmentthatcan'tbeautodetected.
IntheDockerworldyouneedtotakecareaboutwherepersistantdataisstored,sincecontainersareintendedtobevolatile.Weuseavolumenamedarangodb1here:
Generic&Docker
312
dockervolumecreatearangodb1
(Youcanuseanytypeofdockervolumethatfitsyoursetupinstead.)
WethenneedtodeterminethetheIPofthedockerhostwhereyouintendtorunArangoDBstarteron.Dependingonyouroperatingsystemexecuteipaddr,ifconfigoripconfigtodetermineyourlocalipaddress.
192.168.1.0/24deveth0protokernelscopelinksrc192.168.1.32
SothisexampleusestheIP192.168.1.32:
dockerrun-it--name=adb1--rm-p8528:8528\
-varangodb1:/data\
-v/var/run/docker.sock:/var/run/docker.sock\
arangodb/arangodb-starter\
--starter.address=192.168.1.32
Itwillstartthemasterinstance,andcommandyoutostarttheslaveinstances:
Unabletofindimage'arangodb/arangodb-starter:latest'locally
latest:Pullingfromarangodb/arangodb-starter
Digest:sha256:b87d20c0b4757b7daa4cb7a9f55cb130c90a09ddfd0366a91970bcf31a7fd5a4
Status:Downloadednewerimageforarangodb/arangodb-starter:latest
2017/06/1213:26:14Startingarangodbversion0.7.1,buildf128884
2017/06/1213:26:14ServingasmasterwithID'46a2b40d'on192.168.1.32:8528...
2017/06/1213:26:14Waitingfor3serverstoshowup.
2017/06/1213:26:14Usethefollowingcommandstostartotherservers:
dockervolumecreatearangodb2&&\
dockerrun-it--name=adb2--rm-p8533:8528-varangodb2:/data\
-v/var/run/docker.sock:/var/run/docker.sockarangodb/arangodb-starter:0.7\
--starter.address=192.168.1.32--starter.join=192.168.1.32
dockervolumecreatearangodb3&&\
dockerrun-it--name=adb3--rm-p8538:8528-varangodb3:/data\
-v/var/run/docker.sock:/var/run/docker.sockarangodb/arangodb-starter:0.7\
--starter.address=192.168.1.32--starter.join=192.168.1.32
Onceyoustarttheotherinstances,itwillcontinuelikethis:
2017/05/1109:05:45Addedmaster'fc673b3b':192.168.1.32,portOffset:0
2017/05/1109:05:45Addednewpeer'e98ea757':192.168.1.32,portOffset:5
2017/05/1109:05:50Addednewpeer'eb01d0ef':192.168.1.32,portOffset:10
2017/05/1109:05:51Startingservice...
2017/05/1109:05:51Lookingforarunninginstanceofagentonport8531
2017/05/1109:05:51Startingagentonport8531
2017/05/1109:05:52Lookingforarunninginstanceofdbserveronport8530
2017/05/1109:05:52Startingdbserveronport8530
2017/05/1109:05:53Lookingforarunninginstanceofcoordinatoronport8529
2017/05/1109:05:53Startingcoordinatoronport8529
2017/05/1109:05:58agentupandrunning(version3.2.devel).
2017/05/1109:06:15dbserverupandrunning(version3.2.devel).
2017/05/1109:06:31coordinatorupandrunning(version3.2.devel).
Andatleastittellsyouwhereyoucanworkwithyourcluster:
2017/05/1109:06:31Yourclustercannowbeaccessedwithabrowserat`http://192.168.1.32:8529`or
2017/05/1109:06:31using`arangosh--server.endpointtcp://192.168.1.32:8529`.
Underthehood
Thefirstarangodbyouran(asshownabove)willbecomethemasterinyoursetup,the--starter.joinwillbetheslaves.
Generic&Docker
313
ThemasterdetermineswhichArangoDBserverprocessestolaunchonwhichslave,andhowtheyshouldcommunicate.Itwillthenlaunchtheserverprocessesandmonitorthem.Onceithasdetectedthatthesetupiscompleteyouwillgettheprompt.Themasterwillsavethesetupforsubsequentstarts.
MorecomplicatedsetupoptionscanbefoundinArangoDBStartersReadme.
Generic&Docker
314
AdvancedTopics
Incontrasttotheothertopicsinthischapterthatstrivetogetyousimplysetupinpreparedenvironments,Thefollowingchaptersdescribewhatsgoingonunderthehoodindetails,thecomponentsofArangoDBClusters,andhowthey'reputtogether:
StandaloneAgencyTestsetuponalocalmachineStartingprocessesondifferentmachinesLaunchinganArangoDBclusterusingDockercontainers
AdvancedTopics
315
LaunchingArangoDB'sstandalone"agency"
MultipleArangoDBinstancescanbedeployedasafault-tolerantdistributedstatemachine.
Whatisafault-tolerantstatemachineinthefirstplace?
Inmanyservicedeploymentsconsistingofarbitrarycomponentsdistributedovermultiplemachinesoneisfacedwiththechallengeofcreatingadependablecentralisedknowledgebaseorconfiguration.Implementationofsuchaserviceturnsouttobeoneofthemostfundamentalproblemsininformationengineering.Whileitmayseemasiftherealisationofsuchaserviceiseasilyconceivable,dependablityformulatesaparadoxononcomputernetworksperse.Ontheonehand,oneneedsadistributedsystemtoavoidasinglepointoffailure.Ontheotherhand,onehastoestablishconsensusamongthecomputersinvolved.
Consensusisthekeywordhereanditsrealisationonanetworkprovestobefarfromtrivial.Manypapersandconferenceproceedingshavediscussedandevaluatedthiskeychallenge.Twoalgorithms,historicallyfarapart,havebecomewidelypopular,namelyPaxosanditsderivativesandRaft.Discussingthemandtheirdifferences,althoughhighlyenjoyable,mustremainfarbeyondthescopeofthisdocument.Findthereferencestothemainpublicationsatthebottomofthispage.
AtArangoDB,wedecidedtoimplementRaftasitisarguablytheeasiertounderstandandthusimplement.Insimpleterms,Raftguaranteesthatalinearstreamoftransactions,isreplicatedinrealtimeamongagroupofmachinesthroughanelectedleader,whointurnmusthaveaccesstoandprojectleadershipuponanoverallmajorityofparticipatinginstances.InArangoDBweliketocalltheentiretyofthecomponentsofthereplicatedtransactionlog,thatisthemachinesandtheArangoDBinstances,whichconstitutethereplicatedlog,theagency.
Startup
Theagencymustconsistsofanoddnumberofagentsinordertobeabletoestablishanoverallmajorityandsomemeansfortheagentstobeabletofindoneanotheratstartup.
Themostobviouswaywouldbetoinformallagentsoftheaddressesandportsoftherest.Thishowever,ismoreinformationthanneeded.Forexample,itwouldsuffice,ifallagentswouldknowtheaddressandportofthenextagentinacyclicfashion.Anotherstraitforwardsolutionwouldbetoinformallagentsoftheaddressandportofsaythefirstagent.
Clearlyallcases,whichwouldformdisjunctsubsetsofagentswouldbreakorintheleastimpairthefunctionalityoftheagency.Fromthereontheagentswillgossipthemissinginformationabouttheirpeers.
Typically,oneachievesfairlyhighfault-tolerancewithlow,oddnumberofagentswhilekeepingthenecessarynetworktrafficataminimum.Itseemsthatthetypicalagencysizewillbeinrangeof3to7agents.
Thebelowcommandsstartupa3-hostagencyononephysical/logicalboxwithports8529,8530and8531fordemonstrationpurposes.Theadressofthefirstinstance,port8529,isknowntotheothertwo.Afteratmost2roundsofgossipping,thelast2agentswillhaveacompletepictureoftheirsurroundingandpersistitforthenextrestart.
./build/bin/arangod--agency.activatetrue--agency.size3--agency.my-addresstcp://localhost:8529--server.authenticationfal
se--server.endpointtcp://0.0.0.0:8529agency-8529
./build/bin/arangod--agency.activatetrue--agency.size3--agency.endpointtcp://localhost:8529--agency.my-addresstcp://loc
alhost:8530--server.authenticationfalse--server.endpointtcp://0.0.0.0:8530agency-8530
./build/bin/arangod--agency.activatetrue--agency.size3--agency.endpointtcp://localhost:8529--agency.my-addresstcp://loc
alhost:8531--server.authenticationfalse--server.endpointtcp://0.0.0.0:8531agency-8531
Theparameteragency.endpointisthekeyingredientforthesecondandthirdinstancestofindthefirstinstanceandthusformacompleteagency.Pleaserefertothetheshell-scriptscripts/startStandaloneAgency.shongithuborinthesourcedirectory.
Key-value-storeAPI
Theagencyshouldbeupandrunningwithinacoupleofseconds,duringwhichtheinstanceshavegossipedtheirwayintoknowingtheotheragentsandelectedaleader.ThepublicAPIcanbecheckedforthestateoftheconfiguration:
curl-slocalhost:8529/_api/agency/config
AdvancedTopics
316
{
"term":1,
"leaderId":"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98",
"lastCommitted":1,
"lastAcked":{
"ac129027-b440-4c4f-84e9-75c042942171":0.21,
"c54dbb8a-723d-4c82-98de-8c841a14a112":0.21,
"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98":0
},
"configuration":{
"pool":{
"ac129027-b440-4c4f-84e9-75c042942171":"tcp://localhost:8531",
"c54dbb8a-723d-4c82-98de-8c841a14a112":"tcp://localhost:8530",
"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98":"tcp://localhost:8529"
},
"active":[
"ac129027-b440-4c4f-84e9-75c042942171",
"c54dbb8a-723d-4c82-98de-8c841a14a112",
"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98"
],
"id":"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98",
"agencysize":3,
"poolsize":3,
"endpoint":"tcp://localhost:8529",
"minping":0.5,
"maxping":2.5,
"supervision":false,
"supervisionfrequency":5,
"compactionstepsize":1000,
"supervisiongraceperiod":120
}
}
Tohighlightsomedetailsintheaboveoutputlookfor"term"and"leaderId".BotharekeyinformationaboutthecurrentstateoftheRaftalgorithm.Youmayhavenotedthatthefirstelectiontermhasestablishedarandomleaderfortheagency,whoisinchargeofreplicationofthestatemachineandforallexternalreadandwriterequestsuntilsuchtimethattheprocessgetsisolatedfromtheothertwosubsequenctlylosingitsleadership.
ReadandWriteAPIs
Generally,allreadandwriteaccessesaretransactionsmoreoveranyreadandwriteaccessmayconsistofmultiplesuchtransactionsformulatedasarraysofarraysinJSONdocuments.
Readtransaction
Anagencystartedfromscratchwilldealwiththesimplestqueryasfollows:
curl-Llocalhost:8529/_api/agency/read-d'[["/"]]'
[{}]
Theaboverequestforanemptykeyvaluestorewillreturnwithanemptydocument.Theinnerarraybracketswillaggregatearesultfrommultiplesourcesinthekey-value-storewhiletheouterarraywilldelivermultiplesuchaggregatedresults.Alsonotethe-Lcurlflag,whichallowstherequesttofollowredirectstothecurrentleader.
Considerthefollowingkey-value-store:
{
"baz":12,
"corge":{
"e":2.718281828459045,
"pi":3.14159265359
},
"foo":{
"bar":"HelloWorld"
},
AdvancedTopics
317
"qux":{
"quux":"HelloWorld"
}
}
Thefollowingarrayofreadtransactionswillyield:
curl-Llocalhost:8529/_api/agency/read-d'[["/foo","/foo/bar","/baz"],["/qux"]]'
[
{
"baz":12,
"foo":{
"bar":"HelloWorld"
}
},
{
"qux":{
"quux":"HelloWorld"
}
}
]
Notethattheresultisanarrayoftworesultsforthefirstandsecondreadtransactionsfromaboveaccordingly.Alsonotethattheresultsfromthefirstreadtransactionareaggregatedinto
{
"baz":12,
"foo":{
"bar":"HelloWorld"
}
}
Theaggregationisperformedon2levels:
1. /foo/bariseliminatedasasubsetof/foo2. Theresultsfrom/fooand/bararejoined
Thewordtransactionmeansherethatitisguaranteedthatallaggregationshappeninquasi-realtimeandthatnowriteaccesscouldhavehappenedinbetween.
Btw,thesametransactiononthevirginkey-valuestorewouldproduce[{},{}]
WriteAPI:
ThewriteAPImustunfortunatelybealittlemorecomplex.MultipleroadsleadtoRome:
curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"bar"}}]]'
curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"baz"}}]]'
curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"qux"}}]]'
and
curl-Llocalhost:8529/_api/agency/write-d'[[{"foo":["bar","baz","qux"]}]]'
areequivalentforexampleandwillcreateandfillanarrayat/foo.Here,again,theoutermostarrayisthecontainerforthetransactionarrays.
WedocumententacompleteguideoftheAPIintheAPIsection.
AdvancedTopics
318
LaunchinganArangoDBclusterfortesting
AnArangoDBclusterconsistsofseveralrunningtasks(orserverprocesses)whichformthecluster.ArangoDBitselfwon'tstartormonitoranyofthesetasks.Soitwillneedsomekindofsupervisorwhichismonitoringandstartingthesetasks.ForproductionusagewerecommendusingApacheMesosastheclustersupervisor.
HoweverstartingaclustermanuallyispossibleandisaveryeasymethodtogetafirstimpressionofwhatanArangoDBclusterlookslike.
Theeasiestwaytostartalocalclusterfortestingpurposesistorunscripts/startLocalCluster.shfromacloneofthesourcerepositoryaftercompilingArangoDBfromsource(seeinstructionsinthefileREADME_maintainers.mdintherepository.Thiswillstart1Agency,2DBServersand1Coordinator.Tostoptheclusterissuescripts/stopLocalCluster.sh.
ThissectionwilldiscusstherequiredparametersforeveryroleinanArangoDBcluster.BesuretoreadtheArchitecturedocumentationtogetabasicunderstandingoftheunderlyingarchitectureandtheinvolvedrolesinanArangoDBcluster.
Inthefollowingsectionswewillgothroughtherelevantoptionsperrole.
Agency
Tostartupanagencyyoufirsthavetoactivateit.Thisisdonebyproviding--agency.activatetrue.
Tostartuptheagencyinitsfaulttolerantmodesetthe--agency.sizeto3.Youwillthenhavetoprovideatleast3agentsbeforetheagencywillstartoperation.
Duringinitializationtheagentshavetofindeachother.Todosoprovideatleastonecommon--agency.endpoint.Theagentswillthencoordinatestartupthemselves.Theywillannouncethemselveswiththeirexternaladdresswhichmaybespecifiedusing--agency.my-address.ThisisrequiredinbridgeddockersetupsorNATedenvironments.
Soinsummarythisiswhatyourstartupmightlooklike:
AdvancedTopics
319
arangod--server.endpointtcp://0.0.0.0:5001--agency.my-address=tcp://127.0.0.1:5001--server.authenticationfalse--agency.ac
tivatetrue--agency.size3--agency.endpointtcp://127.0.0.1:5001--agency.supervisiontrue--database.directoryagency1&
arangod--server.endpointtcp://0.0.0.0:5002--agency.my-address=tcp://127.0.0.1:5002--server.authenticationfalse--agency.ac
tivatetrue--agency.size3--agency.endpointtcp://127.0.0.1:5001--agency.supervisiontrue--database.directoryagency2&
arangod--server.endpointtcp://0.0.0.0:5003--agency.my-address=tcp://127.0.0.1:5003--server.authenticationfalse--agency.ac
tivatetrue--agency.size3--agency.endpointtcp://127.0.0.1:5001--agency.supervisiontrue--database.directoryagency3&
Ifyouarehappywithasingleagent,thensimplyuseasinglecommandlikethis:
arangod--server.endpointtcp://0.0.0.0:5001--server.authenticationfalse--agency.activatetrue--agency.size1--agency.endp
ointtcp://127.0.0.1:5001--agency.supervisiontrue--database-directoryagency1&
Furthermore,inthefollowingsectionswhen--cluster.agency-addressisusedmultipletimestospecifyallthreeagentaddresses,justuseasingleoption--cluster.agency.addresstcp://127.0.0.1:5001instead.
CoordinatorsandDBServers
Thesetworolesshareacommonsetofrelevantoptions.Firstyoushouldspecifytheroleusing--cluster.my-role.ThiscaneitherbePRIMARY(adatabaseserver)orCOORDINATOR.Furthermoreprovidetheexternalendpoint(IPandport)ofthetaskvia--cluster.my-address.
Thefollowingisafull-exampleofwhatitmightlooklike:
arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8529--cluster.my-addresstcp://127.0.0.1:8529--cluster.
my-rolePRIMARY--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.agency-
endpointtcp://127.0.0.1:5003--database.directoryprimary1&
arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8530--cluster.my-addresstcp://127.0.0.1:8530--cluster.
my-rolePRIMARY--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.agency-
endpointtcp://127.0.0.1:5003--database.directoryprimary2&
arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8531--cluster.my-addresstcp://127.0.0.1:8531--cluster.
my-roleCOORDINATOR--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.age
ncy-endpointtcp://127.0.0.1:5003--database.directorycoordinator&
Noteinparticularthattheendpointdescriptionsgivenunder--cluster.my-addressand--cluster.agency-endpointmustnotusetheIPaddress0.0.0.0becausetheymustcontainanactualaddressthatcanberoutedtothecorrespondingserver.The0.0.0.0in--server.endpointsimplymeansthattheserverbindsitselftoallavailablenetworkdeviceswithallavailableIPaddresses.
UponregisteringwiththeagencyduringstartuptheclusterwillassignanIDtoeveryserver.ThegeneratedIDwillbeprintedouttothelogorcanbeaccessedviathehttpAPIbycallinghttp://server-address/_admin/server/id.
YouhavenowlaunchedacompleteArangoDBclusterandcancontactitscoordinatorattheendpointtcp://127.0.0.1:8531,whichmeansthatyoucanreachthewebUIunderhttp://127.0.0.1:8531.
AdvancedTopics
320
LaunchinganArangoDBclusteronmultiplemachines
Essentially,onecanusethemethodfromtheprevioussectiontostartanArangoDBclusteronmultiplemachinesaswell.Theonlychangesarethatonehastoreplacealllocaladdresses127.0.0.1bytheactualIPaddressofthecorrespondingserver.
IfweassumethatyouwanttostartyouArangoDBclusteronthreedifferentmachineswithIPaddresses
192.168.1.1
192.168.1.2
192.168.1.3
thenthecommandsyouhavetouseare(youcanusehostnamesiftheycanberesolvedtoIPaddressesonallmachines):
On192.168.1.1:
sudoarangod--server.endpointtcp://0.0.0.0:5001--agency.my-addresstcp://192.168.1.1:5001--server.authenticationfalse--ag
ency.activatetrue--agency.size3--agency.supervisiontrue--database.directoryagency
On192.168.1.2:
sudoarangod--server.endpointtcp://0.0.0.0:5001--agency.my-addresstcp://192.168.1.2:5001--server.authenticationfalse--ag
ency.activatetrue--agency.size3--agency.supervisiontrue--database.directoryagency
On192.168.1.3:
sudoarangod--server.endpointtcp://0.0.0.0:5001--agency.my-addresstcp://192.168.1.3:5001--server.authenticationfalse--ag
ency.activatetrue--agency.size3--agency.endpointtcp://192.168.1.1:5001--agency.endpointtcp://192.168.1.2:5001--agency.e
ndpointtcp://192.168.1.3:5001--agency.supervisiontrue--database.directoryagency
On192.168.1.1:
sudoarangod--server.authentication=false--server.endpointtcp://0.0.0.0:8529--cluster.my-addresstcp://192.168.1.1:8529--c
luster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clus
ter.agency-endpointtcp://192.168.1.3:5001--database.directoryprimary1&
On192.168.1.2:
sudoarangod--server.authentication=false--server.endpointtcp://0.0.0.0:8530--cluster.my-addresstcp://192.168.1.2:8530--c
luster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clus
ter.agency-endpointtcp://192.168.1.3:5001--database.directoryprimary2&
On192.168.1.3:
arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8531--cluster.my-addresstcp://192.168.1.3:8531--cluste
r.my-roleCOORDINATOR--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clust
er.agency-endpointtcp://192.168.1.3:5001--database.directorycoordinator&
Obviously,itwouldnolongerbenecessarytousedifferentportnumbersondifferentservers.Wehavechosentokeepallportnumbersincomparisontothelocalsetuptominimizethenecessarychanges.
Afterhavingswallowedtheselongishcommands,wehopethatyouappreciatethesimplicityofthesetupwithApacheMesosandDC/OS.
AdvancedTopics
321
ArangoDBClusterandDocker
Networking
AbitofextracarehastobeinvestedduetothewayinwhichDockerisolatesitsnetwork.Bydefaultitfullyisolatesthenetworkandbydoingsoanendpointlike--server.endpointtcp://0.0.0.0:8529willonlybindtoallinterfacesinsidetheDockercontainerwhichdoesnotincludeanyexternalinterfaceonthehostmachine.ThismaybesufficientifyoujustwanttoaccessitlocallybutincaseyouwanttoexposeittotheoutsideyoumustfacilitateDockersportforwardingusingthe-pcommandlineoption.BesuretochecktheofficialDockerdocumentation.
Tosimplymakearangodbavailableonallhostinterfacesonport8529:
dockerrun-p8529:8529-eARANGO_NO_AUTH=1arangodb
AnotherpossibilityistostartDockervianetworkmodehost.Thisispossiblebutgenerallynotrecommended.TodoitanywaychecktheDockerdocumentationfordetails.
DockerandClustertasks
TostarttheclusterviaDockerisbasicallythesameasstartinglocallyoronmultiplemachines.Howeverjustlikewiththesinglenetworkingimagewewillfacenetworkingissues.Youcansimplyusethe-pflagtomaketheindividualtaskavailableonthehostmachineoryoucoulduseDocker'slinkstoenabletaskintercommunication.
PleasenotethattherearesomeflagsthatspecifyhowArangoDBcanreachataskfromtheoutside.Theseareveryimportantandbuiltforthisexactusecase.Anexampleconfigurationmightlooklikethis:
dockerrun-eARANGO_NO_AUTH=1-p192.168.1.1:10000:8529arangodb/arangodbarangod--server.endpointtcp://0.0.0.0:8529--clust
er.my-addresstcp://192.168.1.1:10000--cluster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agen
cy-endpointtcp://192.168.1.2:5002--cluster.agency-endpointtcp://192.168.1.3:5003
ThiswillstartaprimaryDBserverwithinaDockercontainerwithanisolatednetwork.WithintheDockercontaineritwillbindtoallinterfaces(thiswillbe127.0.0.1:8529andsomeinternalDockeriponport8529).Bysupplying-p192.168.1.1:10000:8529weareestablishingaportforwardingfromourlocalIP(192.168.1.1port10000inthisexample)toport8529insidethecontainer.Withinthecommandwearetellingarangodhowitcanbereachedfromtheoutside--cluster.my-addresstcp://192.168.1.1:10000.ThisinformationwillbeforwardedtotheagencysothattheothertasksinyourclustercanseehowthisparticularDBServermaybereached.
AdvancedTopics
322
Datacentertodatacenterreplication.
About
Atsomepointinthegrowsofadatabase,therecomesaneedforreplicatingitacrossmultipledatacenters.
Reasonsforthatcanbe:
Fallbackincaseofadisasterinonedatacenter.RegionalavailabilitySeparationofconcerns
Andmanymore.
ThistutorialdescribeswhattheArangoSyncdatacentertodatacenterreplicationsolution(ArangoSyncfromnowon)offers,whentouseit,whennottouseitandhowtoconfigure,operate,troubleshootit&keepitsafe.
Whatisit
ArangoSyncisasolutionthatenablesyoutoasynchronouslyreplicatetheentirestructureandcontentinanArangoDBclusterinoneplacetoaclusterinanotherplace.Typicallyitisusedfromonedatacentertoanother.Itisnotasolutionforreplicatingsingleserverinstances.
ThereplicationdonebyArangoSyncinasynchronous.Thatmeansthatwhenaclientiswritingdataintothesourcedatacenter,itwillconsidertherequestfinishedbeforethedatahasbeenreplicatedtotheotherdatacenter.Thetimeneededtocompletelyreplicatechangestotheotherdatacenteristypicallyintheorderofseconds,butthiscanvarysignificantlydependingonload,network&computercapacity.
ArangoSyncperformsreplicationinasingledirectiononly.ThatmeansthatyoucanreplicatedatafromclusterAtoclusterBorfromclusterBtoclusterA,butneveratthesametime.Datamodifiedinthedestinationclusterwillbelost!
Replicationisacompletelyautonomousprocess.Onceitisconfigureditisdesignedtorun24/7withoutfrequentmanualintervention.Thisdoesnotmeanthatitrequiresnomaintenanceorattentionatall.Aswithanydistributedsystemsomeattentionisneededtomonitoritsoperationandkeepitsecure(e.g.certificate&passwordrotation).
Onceconfigured,ArangoSyncwillreplicatebothstructureanddataofanentirecluster.Thismeansthatthereisnoneedtomakeadditionalconfigurationchangeswhenadding/removingdatabasesorcollections.Alsometadatasuchasusers,foxxapplication&jobsareautomaticallyreplicated.
Whentouseit...andwhennot
ArangoSyncisagoodsolutioninallcaseswhereyouwanttoreplicatedatafromoneclustertoanotherwithouttherequirementthatthedataisavailableimmediatelyintheothercluster.
ArangoSyncisnotagoodsolutionwhenoneofthefollowingapplies:
YouwanttoreplicatedatafromclusterAtoclusterBandfromclusterBtoclusterAatthesametime.Youneedsynchronousreplicationbetween2clusters.ThereisnonetworkconnectionbetweeclusterAandB.Youwantcompletecontroloverwhichdatabase,collection&documentsarereplicateandwhichnot.
Requirements
TouseArangoSyncyouneedthefollowing:
Twodatacenters,eachrunninganArangoDBEnterprisecluster,version3.3orhigher.
MultipleDatacenters
323
Anetworkconnectionbetweenbothdatacenterswithaccessibleendpointsforseveralcomponents(seeindividualcomponentsfordetails).TLScertificatesforArangoSyncmasterinstances(canbeself-signed).TLScertificatesforKafkabrokers(canbeself-signed).Optional(butrecommended)TLScertificatesforArangoDBclusters(canbeself-signed).ClientcertificatesCAforArangoSyncmasters(typicallyself-signed).ClientcertificatesforArangoSyncmasters(typicallyself-signed).Atleast2instancesoftheArangoSyncmasterineachdatacenter.OneinstancesoftheArangoSyncworkeroneverymachineineachdatacenter.
Note:Inseveralplacesyouwillneeda(x509)certificate.Thecertificatessectionbelowprovidesmoreguidanceforcreatingandrenewingthesecertificates.
Besidestheabovelist,youprobablywanttousethefollowing:
Anorchestratortokeepallcomponentsrunning.Inthistutorialwewillusesystemdasanexample.Alogfilecollectorforcentralizedcollection&accesstothelogsofallcomponents.Ametricscollector&viewingsolutionsuchasPrometheus+Grafana.
Deployment
Inthefollowingparagraphsyou'lllearnhowtodeployallthecomponentsneededfordatacentertodatacenterreplication.
ArangoDBcluster
ThereareseveralwaystostartanArangoDBcluster.InthistutorialwewillfocusonourrecommendedwaytostartArangoDB:theArangoDBstarter.
Datacentertodatacenterreplicationrequirestherocksdbstorageengine.Inthistutorialtheexamplesetupwillhaverocksdbenabled.Ifyouchoosetodeploywithadifferentstrategykeepinmindtosetthestorageengine.
AlsoseeotherpossibilitiestodeployanArangoDBcluster.
Thestartersimplifiesthingsfortheoperatorandwillcoordinateadistributedclusterstartupacrossseveralmachinesandassignclusterrolesautomatically.
Whenstartedonseveralmachinesandenoughmachineshavejoined,thestarterswillstartagents,coordinatorsanddbserversonthesemachines.
Whenrunningthestarterwillsuperviseitschildtasks(namelycoordinators,dbserversandagents)andrestartthemincaseoffailure.
Tostarttheclusterusingasystemdunitfileusethefollowing:
[Unit]
Description=RuntheArangoDBStarter
After=network.target
[Service]
Restart=on-failure
EnvironmentFile=/etc/arangodb.env
EnvironmentFile=/etc/arangodb.env.local
Environment=DATADIR=/var/lib/arangodb/cluster
ExecStartPre=/usr/bin/sh-c"mkdir-p${DATADIR}"
ExecStart=/usr/bin/arangodb\
--starter.address=${PRIVATEIP}\
--starter.data-dir=${DATADIR}\
--starter.join=${STARTERENDPOINTS}\
--server.storage-engine=rocksdb\
--auth.jwt-secret=${CLUSTERSECRETPATH}
TimeoutStopSec=60
[Install]
WantedBy=multi-user.target
Notethatwesetrocksdbintheunitservicefile.
MultipleDatacenters
324
Clusterauthentication
Thecommunicationbetweentheclusternodesuseatoken(JWT)toauthenticate.Thismustbesharedbetweenclusternodes.
Sharingsecretsisobviouslyaverydelicatetopic.Theaboveworkflowassumesthattheoperatorwillputasecretinafilenamed${CLUSTERSECRETPATH}.
WerecommendtouseadedicatedsystemformanagingsecretslikeHashiCorps'VaultorthesecretmanagementofDC/OS.
Requiredports
Assoonasenoughmachineshavejoined,thestarterwillbeginstartingagents,coordinatorsanddbservers.
Eachofthesetasksneedsaporttocommunicate.Pleasemakesurethatthefollowingportsareavailableonallmachines:
8529forcoordinators8530fordbservers8531foragents
Thestarteritselfwilluseport8528.
Kafka&Zookeeper
HowtodeployzookeeperHowtodeploykafkaAccessibleports
SyncMaster
TheSyncMasterisresponsibleformanagingallsynchronization,creatingtasksandassigningthosetoworkers.Atleast2instancesmutsbedeployedineachdatacenter.Oneinstancewillbethe"leader",theotherwillbeaninactiveslave.Whentheleaderisgoneforashortwhile,oneoftheotherinstanceswilltakeover.
Withclustersofasignificantsize,thesyncmasterwillrequireasignificantsetofresources.Thereforeitisrecommendedtodeploysyncmastersontheirownservers,equipedwithsufficientCPUpowerandmemorycapacity.
Tostartasyncmasterusingasystemdservice,useaunitlikethis:
[Unit]
Description=RunArangoSyncinmastermode
After=network.target
[Service]
Restart=on-failure
EnvironmentFile=/etc/arangodb.env
EnvironmentFile=/etc/arangodb.env.local
ExecStart=/usr/sbin/arangosyncrunmaster\
--log.level=debug\
--cluster.endpoint=${CLUSTERENDPOINTS}\
--cluster.jwtSecret=${CLUSTERSECRET}\
--server.keyfile=${CERTIFICATEDIR}/tls.keyfile\
--server.client-cafile=${CERTIFICATEDIR}/client-auth-ca.crt\
--server.endpoint=https://${PUBLICIP}:${MASTERPORT}\
--server.port=${MASTERPORT}\
--master.jwtSecret=${MASTERSECRET}\
--mq.type=kafka\
--mq.kafka-addr=${KAFKAENDPOINTS}\
--mq.kafka-client-keyfile=${CERTIFICATEDIR}/kafka-client.key\
--mq.kafka-cacert=${CERTIFICATEDIR}/tls-ca.crt\
TimeoutStopSec=60
[Install]
WantedBy=multi-user.target
ThesyncmasterneedsaTLSservercertificateandaIfyouwanttheservicetocreateaTLScertificate&clientauthenticationcertificate,forauthenticatingwithsyncmastersinanotherdatacenter,foreverystart,addthistotheServicesection.
MultipleDatacenters
325
ExecStartPre=/usr/bin/sh-c"mkdir-p${CERTIFICATEDIR}"
ExecStartPre=/usr/sbin/arangosynccreatetlskeyfile\
--cacert=${CERTIFICATEDIR}/tls-ca.crt\
--cakey=${CERTIFICATEDIR}/tls-ca.key\
--keyfile=${CERTIFICATEDIR}/tls.keyfile\
--host=${PUBLICIP}\
--host=${PRIVATEIP}\
--host=${HOST}
ExecStartPre=/usr/sbin/arangosynccreateclient-authkeyfile\
--cacert=${CERTIFICATEDIR}/tls-ca.crt\
--cakey=${CERTIFICATEDIR}/tls-ca.key\
--keyfile=${CERTIFICATEDIR}/kafka-client.key\
--host=${PUBLICIP}\
--host=${PRIVATEIP}\
--host=${HOST}
ThesyncmastermustbereachableonaTCPport${MASTERPORT}(usedwith--server.portoption).Thisportmustbereachablefrominsidethedatacenter(bysyncworkersandoperations)andfrominsideoftheotherdatacenter(bysyncmastersintheotherdatacenter).
SyncWorkers
TheSyncWorkerisresponsibleforexecutingsynchronizationtasks.Foroptimalperformanceatleast1workerinstancemustbeplacedoneverymachinethathasanArangoDBdbserverrunning.Thisensuresthattaskscanbeexecutedwithminimalnetworktrafficoutsideofthemachine.
SincesyncworkerswillautomaticallystoponcetheirTLSservercertificateexpires(whichissetto2yearsbydefault),itisrecommendedtorunatleast2instancesofaworkeroneverymachineinthedatacenter.Thatway,taskscanstillbeassignedinthemostoptimalway,evenwhenaworkerintemporarilydownforarestart.
Tostartasyncworkerusingasystemdservice,useaunitlikethis:
[Unit]
Description=RunArangoSyncinworkermode
After=network.target
[Service]
Restart=on-failure
EnvironmentFile=/etc/arangodb.env
EnvironmentFile=/etc/arangodb.env.local
Environment=PORT=8729
ExecStart=/usr/sbin/arangosyncrunworker\
--log.level=debug\
--server.port=${PORT}\
--server.endpoint=https://${PRIVATEIP}:${PORT}\
--master.endpoint=${MASTERENDPOINTS}\
--master.jwtSecret=${MASTERSECRET}
TimeoutStopSec=60
[Install]
WantedBy=multi-user.target
ThesyncworkermustbereachableonaTCPport${PORT}(usedwith--server.portoption).Thisportmustbereachablefrominsidethedatacenter(bysyncmasters).
Prometheus&Grafana(optional)
ArangoSyncprovidesmetricsinaformatsupportedbyPrometheus.WealsoprovideastandardsetofdashboardsforviewingthosemetricsinGrafana.
Ifyouwanttousethesetools,gototheirwebsitesforinstructionsonhowtodeploythem.
Afterdeployment,youmustconfigureprometheususingaconfigurationfilethatinstructsitaboutwhichtargetstoscrape.ForArangoSyncyoushouldconfigurescrapetargetsforallsyncmastersandallsyncworkers.Todoso,youcanuseaconfigurationsuchasthis:
global:
scrape_interval:10s#scrapetargetsevery10seconds.
MultipleDatacenters
326
scrape_configs:
#Scrapsyncmasters
-job_name:'sync_master'
scheme:'https'
bearer_token:"${MONITORINGTOKEN}"
tls_config:
insecure_skip_verify:true
static_configs:
-targets:
-"${IPMASTERA1}:8629"
-"${IPMASTERA2}:8629"
-"${IPMASTERB1}:8629"
-"${IPMASTERB2}:8629"
labels:
type:"master"
relabel_configs:
-source_labels:[__address__]
regex:${IPMASTERA1}\:8629|${IPMASTERA2}\:8629
target_label:dc
replacement:A
-source_labels:[__address__]
regex:${IPMASTERB1}\:8629|${IPMASTERB2}\:8629
target_label:dc
replacement:B
-source_labels:[__address__]
regex:${IPMASTERA1}\:8629|${IPMASTERB1}\:8629
target_label:instance
replacement:1
-source_labels:[__address__]
regex:${IPMASTERA2}\:8629|${IPMASTERB2}\:8629
target_label:instance
replacement:2
#Scrapsyncworkers
-job_name:'sync_worker'
scheme:'https'
bearer_token:"${MONITORINGTOKEN}"
tls_config:
insecure_skip_verify:true
static_configs:
-targets:
-"${IPWORKERA1}:8729"
-"${IPWORKERA2}:8729"
-"${IPWORKERB1}:8729"
-"${IPWORKERB2}:8729"
labels:
type:"worker"
relabel_configs:
-source_labels:[__address__]
regex:${IPWORKERA1}\:8729|${IPWORKERA2}\:8729
target_label:dc
replacement:A
-source_labels:[__address__]
regex:${IPWORKERB1}\:8729|${IPWORKERB2}\:8729
target_label:dc
replacement:B
-source_labels:[__address__]
regex:${IPWORKERA1}\:8729|${IPWORKERB1}\:8729
target_label:instance
replacement:1
-source_labels:[__address__]
regex:${IPWORKERA2}\:8729|${IPWORKERB2}\:8729
target_label:instance
replacement:2
Note:Theaboveexampleassumes2datacenters,with2syncmasters&2syncworkersperdatacenter.Youhavetoreplaceall${...}variablesintheaboveconfigurationwithapplicablevaluesfromyourenvironment.
Configuration
MultipleDatacenters
327
OnceallcomponentsoftheArangoSyncsolutionhavebeendeployedandarerunningproperly,ArangoSyncwillnotautomaticallyreplicatedatabasestructureandcontent.Forthat,itisisneededtoconfiguresynchronization.
Toconfiguresynchronization,youneedthefollowing:
Theendpointofthesyncmasterinthetargetdatacenter.Theendpointofthesyncmasterinthesourcedatacenter.Acertificate(inkeyfileformat)usedforclientauthenticationofthesyncmaster(withthesyncmasterinthesourcedatacenter).ACAcertificate(publickeyonly)forverifyingtheintegrityofthesyncmasters.Ausername+passwordpair(orclientcertificate)forauthenticatingtheconfigurerequirewiththesyncmaster(inthetargetdatacenter)
Withthatinformation,run:
arangosyncconfiguresync\
--master.endpoint=<endpointsofsyncmastersintargetdatacenter>\
--master.keyfile=<keyfileofofsyncmastersintargetdatacenter>\
--source.endpoint=<endpointsofsyncmastersinsourcedatacenter>\
--source.cacert=<publickeyofCAcertificateusedtoverifysyncmasterinsourcedatacenter>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>
Thecommandwillfinishquickly.Afterwardsitwilltakesometimeuntiltheclustersinbothdatacentersareinsync.
Usethefollowingcommandtoinspectthestatusofthesynchronizationofadatacenter:
arangosyncgetstatus\
--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>\
-v
Note:Invokingthiscommandonthetargetdatacenterwillreturndifferentresultsfrominvokingitonthesourcedatacenter.Youneedinsightinbothresultstogeta"completepicture".
Wherethegetstatuscommandgivesinsightinthestatusofsynchronization,therearemoredetailedcommandstogiveinsightintasks®isteredworkers.
Usethefollowingcommandtogetalistofallsynchronizationtasksinadatacenter:
arangosyncgettasks\
--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>\
-v
Usethefollowingcommandtogetalistofallmastersinadatacenterandknowwhichmasteristhecurrentleader:
arangosyncgetmasters\
--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>\
-v
Usethefollowingcommandtogetalistofallworkersinadatacenter:
arangosyncgetworkers\
--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>\
-v
Stopsynchronization
MultipleDatacenters
328
Ifyounolongerwanttosynchronizedatafromasourcetoatargetdatacenteryoumuststopit.Todoso,runthefollowingcommand:
arangosyncstopsync\
--master.endpoint=<endpointsofsyncmastersintargetdatacenter>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>
Thecommandwillwaituntilsynchronizationhascompletelystoppedbeforereturning.Ifthesynchronizationisnotcompletelystoppedwithinareasonableperiod(2minutesbydefault)thecommandwillfail.
Ifthesourcedatacenterisnolongeravailableitisnotpossibletostopsynchronizationinagracefulmanner.Ifthathappensabortthesynchronizationwiththefollowingcommand:
arangosyncabortsync\
--master.endpoint=<endpointsofsyncmastersintargetdatacenter>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>
Ifthesourcedatacenterrecoversafteranabortsynchasbeenexecuted,itisneededto"cleanup"ArangoSyncinthesourcedatacenter.Todoso,executethefollowingcommand:
arangosyncabortoutgoingsync\
--master.endpoint=<endpointsofsyncmastersinsourcedatacenter>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>
Reversingsynchronizationdirection
Ifyouwanttoreversethedirectionofsynchronization(e.g.afterafailureindatacenterAandyouswitchedtothedatacenterBforfallback),youmustfirststop(orabort)theoriginalsynchronization.
Oncethatisfinished(andcleanuphasbeenappliedincaseofabort),youmustnowconfigurethesynchronizationagain,butwithswappedsource&targetsettings.
Operations&Maintenance
ArangoSyncisadistributedsystemwithalotdifferentcomponents.Aswithanysuchsystem,itrequiressome,butnotalot,ofoperationalsupport.
Whatmeansareavailabletomonitorstatus
AllofthecomponentsofArangoSyncprovidemeanstomonitortheirstatus.Belowyou'llfindanoverviewpercomponent.
Syncmaster&workers:Thearangosyncserversrunningaseithermasterorworker,provide:
AstatusAPI,seearangosyncgetstatus.Makesurethatallstatusesreportrunning.Forevenmoredetailthefollowingcommandsarealsoavailable:arangosyncgettasks,arangosyncgetmasters&arangosyncgetworkers.Alogonthestandardoutput.Loglevelscanbeconfiguredusing--log.levelsettings.AmetricsAPIGET/metrics.ThisAPIiscompatiblewithPrometheus.SampleGrafanadashboardsforinspectingthesemetricsareavailable.
ArangoDBcluster:ThearangodserversthatmakeuptheArangoDBclusterprovide:
Alogfile.Thisisconfigurablewithsettingswithalog.prefix.E.g.--log.output=file://myLogFileor--log.level=info.AstatisticsAPIGET/_admin/statistics
Kafkacluster:Thekafkabrokersprovide:
Alogfile,seesettingswithlog.prefixinitsserver.propertiesconfigurationfile.Zookeeper:Thezookeeperagentsprovide:
Alogonstandardoutput.
MultipleDatacenters
329
Whattolookforwhilemonitoringstatus
TheveryfirstthingtodowhenmonitoringthestatusofArangoSyncistolookintothestatusprovidedbyarangosyncgetstatus...-v.Whennoteverythingisintherunningstate(onbothdatacenters),thisisanindicationthatsomethingmaybewrong.Incasethathappens,giveitsometime(incrementalsynchronizationmaytakequitesometimeforlargecollections)andlookatthestatusagain.Ifthestatusesdonotchange(orchange,butnotreachrunning)itistimetoinspectsthemetrics&logfiles.Whenthemetricsorlogsseemtoindicateaprobleminasyncmasterorworker,itissafetorestartit,aslongasonly1instanceisrestartedatatime.Giverestartedinstancessometimeto"catchup".
Whattodowhenproblemsremain
Whenaproblemremainsandrestartingmasters/workersdoesnotsolvetheproblem,contactsupport.Makesuretoincludeprovidesupportwiththefollowinginformation:
Outputofarangosyncgetversion...onbothdatacenters.Outputofarangosyncgetstatus...-vonbothdatacenters.Outputofarangosyncgettasks...-vonbothdatacenters.Outputofarangosyncgetmasters...-vonbothdatacenters.Outputofarangosyncgetworkers...-vonbothdatacenters.LogfilesofallcomponentsAcompletedescriptionoftheproblemyouobservedandwhatyoudidtoresolveit.
HowtomonitorstatusofArangoSync
HowtokeepitaliveWhattodoincaseoffailuresorbugs
Whattodowhenasourcedatacenterisdown
WhenyouuseArangoSyncforbackupofyourclusterfromonedatacentertoanotherandthesourcedatacenterhasacompleteoutage,youmayconsiderswitchingyourapplicationstothetarget(backup)datacenter.
Thisiswhatyoumustdointhatcase.
1. Stopconfigurationusingarangosyncstopsync....Whenthesourcedatacenteriscompletelyunresponsivethiswillnotsucceed.Inthatcaseusearangosyncabortsync....SeeConfigurationforhowtocleanupthesourcedatacenterwhenitbecomesavailableagain.
2. Verifythatconfigurationhascompletelystoppedusingarangosyncgetstatus...-v.3. Reconfigureyourapplicationstousethetarget(backup)datacenter.
Whentheoriginalsourcedatacenterisrestored,youmayswitchrolesandmakeitthetargetdatacenter.Todoso,usearangosyncconfiguresync...asdescribedinConfiguration.
Whattodoincaseofaplannednetworkoutage.
AllArangoSynctaskssendoutheartbeatmessagesouttotheotherdatacentertoindicate"itisstillalive".Theotherdatacenterassumestheconnectionis"outofsync"whenitdoesnotreceiveanymessagesforacertainperiodoftime.
Ifyou'replanningsomesortofmaintenancewhereyouknowtheconnectivitywillbelostforsometime(e.g.3hours),youcanprepareArangoSyncforthatsuchthatitwillholdofre-synchronizationforagivenperiodoftime.
Todoso,onbothdatacenters,run:
arangosyncsetmessagetimeout\
--master.endpoint=<endpointsofsyncmastersinthedatacenter>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>\
3h
ThelastargumentistheperiodthatArangoSyncshouldhold-ofresynchronizationfor.Thiscanbeminutes(e.g.15m)orhours(e.g.3h).
MultipleDatacenters
330
Ifmaintenanceistakinglongerthanexpected,youcanusethesamecommandtheextendtheholdofperiod(e.g.to4h).
Afterthemaintenance,usethesamecommandrestoretheholdofperiodtoitsdefaultof1h.
Whattodoincaseofadocumentthatexceedsthemessagequeuelimits.
Ifyouinsert/updateadocumentinacollectionandthesizeofthatdocumentislargerthanthemaximummessagesizeofyourmessagequeue,thecollectionwillnolongerbeabletosynchronize.Itwillgointoafailedstate.
Torecoverfromthat,firstremovethedocumentfromtheArangoDBclusterinthesourcedatacenter.Afterthat,foreachfailedshard,run:
arangosyncresetfailedshard\
--master.endpoint=<endpointsofsyncmastersinthedatacenter>\
--auth.user=<usernameusedforauthenticationofthiscommand>\
--auth.password=<passwordofauth.user>\
--database=<nameofthedatabase>\
--collection=<nameofthecollection>\
--shard=<indexoftheshard(startingat0)>
Afterthiscommand,anewsetoftaskswillbestartedtosynchronizetheshard.Itcantakesometimefortheshardtoreachrunningstate.
Metrics
ArangoSync(master&worker)providemetricsthatcanbeusedformonitoringtheArangoSyncsolution.ThesemetricsareavailableusingthefollowingHTTPSendpoints:
GET/metrics:ProvidesmetricsinaformatsupportedbyPrometheus.GET/metrics.json:ProvidesthesamemetricsinJSONformat.
Bothendpointsincludehelpinformationpermetrics.
Note:Bothendpointsrequireauthentication.Besidestheusualauthenticationmethodstheseendpointsarealsoaccessibleusingaspecialbearertokenspecifiedusingthe--monitoring.tokencommandlineoption.
ThePrometheusoutput(/metrics)lookslikethis:
...
#HELParangosync_master_worker_registrationsTotalnumberofregistrations
#TYPEarangosync_master_worker_registrationscounter
arangosync_master_worker_registrations2
#HELParangosync_master_worker_storageNumberoftimesworkerinfoisstored,loaded
#TYPEarangosync_master_worker_storagecounter
arangosync_master_worker_storage{kind="",op="save",result="success"}20
arangosync_master_worker_storage{kind="empty",op="load",result="success"}1
...
TheJSONoutput(/metrics.json)lookslikethis:
{
...
"arangosync_master_worker_registrations":{
"help":"Totalnumberofregistrations",
"type":"counter",
"samples":[
{
"value":2
}
]
},
"arangosync_master_worker_storage":{
"help":"Numberoftimesworkerinfoisstored,loaded",
"type":"counter",
"samples":[
{
"value":8,
MultipleDatacenters
331
"labels":{
"kind":"",
"op":"save",
"result":"success"
}
},
{
"value":1,
"labels":{
"kind":"empty",
"op":"load",
"result":"success"
}
}
]
}
...
}
Hint:Togetalistofametricsandtheirhelpinformation,run:
aliasjq='dockerrun--rm-irealguess/jqjq'
curl-sk-u"<user>:<password>"https://<syncmaster-IP>:8629/metrics.json|\
jq'with_entries({key:.key,value:.value.help})'
Security
Firewallsettings
ThecomponentsofArangoSyncuse(TCP)networkconnectionstocommunicatewitheachother.Belowyou'llfindanoverviewoftheseconnectionsandtheTCPportsthatshouldbeaccessible.
1. Thesyncmastersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:
ArangoDBagentsandcoordinators(defaultports:8531and8529)Kafkabrokers(defaultport9092)Syncworkers(defaultport8729)
Additionallythesyncmastersmustbeallowedtoconnecttothesyncmastersintheotherdatacenter.
Bydefaultthesyncmasterswilloperateonport8629.
2. Thesyncworkersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:
ArangoDBcoordinators(defaultport8529)Kafkabrokers(defaultport9092)Syncmasters(defaultport8629)
Bydefaultthesyncworkerswilloperateonport8729.
AdditionallythesyncworkersmustbeallowedtoconnecttotheKafkabrokersintheotherdatacenter.
3. Kafka
Thekafkabrokersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:
Otherkafkabrokers(defaultport9092)Zookeeper(defaultports2181,2888and3888)
Thedefaultportforkafkais9092.Thedefaultkafkainstallationwillalsoexposesomeprometheusmetricsonport7071.Togainmoreinsightintokafkaopenthisportforyourprometheusinstallation.
4. Zookeeper
Thezookeeperagentsmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:
OtherzookeeperagentsThesetuphereisabitspecialaszookeeperuses3portsfordifferentoperations.Allagentsneedtobeabletoconnecttoalloftheseports.
MultipleDatacenters
332
BydefaultZookeeperuses:
port2181forclientcommunicationport2888forfollowercommunicationport3888forleaderelections
Certificates
DigitalcertificatesareusedinmanyplacesinArangoSyncforbothencryptionandauthentication.
InArangoSyncallnetworkconnectionsareusingTransportLayerSecurity(TLS),asetofprotocolsthatensurethatallnetworktrafficisencrypted.ForthisTLScertificatesareused.TheserversideofthenetworkconnectionoffersaTLScertificate.Thiscertificateis(often)verifiedbytheclientsideofthenetworkconnection,toensurethatthecertificateissignedbyatrustedCertificateAuthority(CA).Thisensurestheintegrityoftheserver.Inseveralplacesadditionalcertificatesareusedforauthentication.Inthosecasestheclientsideoftheconnectionoffersaclientcertificate(ontopofanexistingTLSconnection).Theserversideoftheconnectionusestheclientcertificatetoauthenticatetheclientand(optionally)decideswhichrightsshouldbeassignedtotheclient.
Note:ArangoSyncdoesallowtheuseofcertificatessignedbyawellknowCA(eg.verisign)howeveritismoreconvenient(andcommon)touseyourownCA.
Formats
Allcertificatesarex509certificateswithapublickey,aprivatekeyandanoptionalchainofcertificatesusedtosignthecertificate(thischainistypicallyprovidedbytheCertificateAuthority(CA)).Dependingontheiruse,certificatesstoredinadifferentformat.
Thefollowingformatsareused:
Publickeyonly(.crt):Afilethatcontainsonlythepublickeyofacertificatewithanoptionalchainofparentcertificates(publickeysofcertificatesusedtosignedthecertificate).Sincethisformatcontainsonlypublickeys,itisnotaproblemifitscontentsareexposed.Itmuststillbestoreitinasafeplacetoavoidlosingit.Privatekeyonly(.key):Afilethatcontainsonlytheprivatekeyofacertificate.Itisvitaltoprotectthesefilesandstoretheminasafeplace.Keyfilewithpublic&privatekey(.keyfile):Afilethatcontainsthepublickeyofacertificate,anoptionalchainofparentcertificatesandaprivatekey.Sincethisformatalsocontainsaprivatekey,itisvitaltoprotectthesefilesandstoretheminasafeplace.Javakeystore(.jks):Afilecontainingasetofpublicandprivatekeys.Itispossibletoprotectaccesstothecontentofthisfileusingakeystorepassword.Sincethisformatcancontainprivatekeys,itisvitaltoprotectthesefilesandstoretheminasafeplace(evenwhenitscontentisprotectedwithakeystorepassword).
Creatingcertificates
ArangoSyncprovidescommandstocreateallcertificatesneeded.
TLSservercertificates
TocreateacertificateusedforTLSserversinthekeyfileformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses).Thenrun:
arangosynccreatetlskeyfile\
--cacert=my-tls-ca.crt--cakey=my-tls-ca.key\
--host=<hostname>\
--keyfile=my-tls-cert.keyfile
Makesuretostorethegeneratedkeyfile(my-tls-cert.keyfile)inasafeplace.
MultipleDatacenters
333
TocreateacertificateusedforTLSserversinthecrt&keyformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses).Thenrun:
arangosynccreatetlscertificate\
--cacert=my-tls-ca.crt--cakey=my-tls-ca.key\
--host=<hostname>\
--cert=my-tls-cert.crt\
--key=my-tls-cert.key\
Makesuretoprotectandstorethegeneratedfiles(my-tls-cert.crt&my-tls-cert.key)inasafeplace.
Clientauthenticationcertificates
Tocreateacertificateusedforclientauthenticationinthekeyfileformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses)oremailaddresses.Thenrun:
arangosynccreateclient-authkeyfile\
--cacert=my-client-auth-ca.crt--cakey=my-client-auth-ca.key\
[--host=<hostname>|--email=<emailaddress>]\
--keyfile=my-client-auth-cert.keyfile
Makesuretoprotectandstorethegeneratedkeyfile(my-client-auth-cert.keyfile)inasafeplace.
CAcertificates
TocreateaCAcertificateusedtosignTLScertificates,run:
arangosynccreatetlsca\
--cert=my-tls-ca.crt--key=my-tls-ca.key
Makesuretoprotectandstorebothgeneratedfiles(my-tls-ca.crt&my-tls-ca.key)inasafeplace.Note:CAcertificateshaveamuchlongerlifetimethannormalcertificates.Thereforeevenmorecareisneededtostorethemsafely.
TocreateaCAcertificateusedtosignclientauthenticationcertificates,run:
arangosynccreateclient-authca\
--cert=my-client-auth-ca.crt--key=my-client-auth-ca.key
Makesuretoprotectandstorebothgeneratedfiles(my-client-auth-ca.crt&my-client-auth-ca.key)inasafeplace.Note:CAcertificateshaveamuchlongerlifetimethannormalcertificates.Thereforeevenmorecareisneededtostorethemsafely.
Renewingcertificates
Allcertificateshavemetainformationinthemthelimittheiruseinfunction,target&lifetime.Acertificatecreatedforclientauthentication(function)cannotbeusedasaTLSservercertificate(sameistrueforthereverse).Acertificateforhostmyserver(target)cannotbeusedforhostanotherserver.AcertficiatethatisvaliduntilOctober2017(limetime)cannotbeusedafterOctober2017.
Ifanythingchangesinfunction,targetorlifetimeyouneedanewcertificate.
Theprocedureforcreatingarenewedcertificateisthesameasforcreatinga"first"certificate.Aftercreatingtherenewedcertificatetheprocess(es)usingthemhavetobeupdated.Thismeanrestartingthem.AllArangoSynccomponentsaredesignedtosupportstoppingandstartingsingleinstances,butdonotrestartmorethan1instanceatthesametime.Assoonas1instancehasbeenrestarted,giveitsometimeto"catchup"beforerestartingthenextinstance.
MultipleDatacenters
334
ArangoDBKubernetesOperatorTheArangoDBKubernetesOperator(kube-arangodb)isasetofoperatorsthatyoudeployinyourKubernetesclusterto:
ManagedeploymentsoftheArangoDBdatabaseProvidePersistentVolumesonlocalstorageofyournodesforoptimalstorageperformance.ConfigureArangoDBDatacentertoDatacenterreplication
Eachoftheseusesinvolvesadifferentcustomresource.
UseanArangoDeploymentresourcetocreateanArangoDBdatabasedeployment.UseanArangoLocalStorageresourcetoprovidelocalPersistentVolumesforoptimalI/Operformance.UseanArangoDeploymentReplicationresourcetoconfigureArangoDBDatacentertoDatacenterreplication.
ContinuewithUsingtheArangoDBKubernetesOperatortolearnhowtoinstalltheArangoDBKubernetesoperatorandcreateyourfirstdeployment.
Kubernetes
335
UsingtheArangoDBKubernetesOperator
Installation
TheArangoDBKubernetesOperatorneedstobeinstalledinyourKubernetesclusterfirst.
Todoso,run(replace<version>withtheversionoftheoperatorthatyouwanttoinstall):
exportURLPREFIX=https://raw.githubusercontent.com/arangodb/kube-arangodb/<version>/manifests
kubectlapply-f$URLPREFIX/crd.yaml
kubectlapply-f$URLPREFIX/arango-deployment.yaml
TouseArangoLocalStorageresources,alsorun:
kubectlapply-f$URLPREFIX/arango-storage.yaml
TouseArangoDeploymentReplicationresources,alsorun:
kubectlapply-f$URLPREFIX/arango-deployment-replication.yaml
YoucanfindthelatestreleaseoftheArangoDBKubernetesOperatorinthekube-arangodbrepository.
ArangoDBdeploymentcreationOncetheoperatorisrunning,youcancreateyourArangoDBdatabasedeploymentbycreatingaArangoDeploymentcustomresourceanddeployingitintoyourKubernetescluster.
Forexample(allexamplescanbefoundinthekube-arangodbrepository):
kubectlapply-fexamples/simple-cluster.yaml
Deploymentremoval
ToremoveanexistingArangoDBdeployment,deletethecustomresource.Theoperatorwillthendeleteallcreatedresources.
Forexample:
kubectldelete-fexamples/simple-cluster.yaml
NotethatthiswillalsodeletealldatainyourArangoDBdeployment!
Ifyouwanttokeepyourdata,makesuretocreateabackupbeforeremovingthedeployment.
Operatorremoval
ToremovetheentireArangoDBKubernetesOperator,removeallclustersfirstandthenremovetheoperatorbyrunning:
kubectldeletedeploymentarango-deployment-operator
#If`ArangoLocalStorage`operatorisinstalled
kubectldeletedeployment-nkube-systemarango-storage-operator
#If`ArangoDeploymentReplication`operatorisinstalled
kubectldeletedeploymentarango-deployment-replication-operator
UsingtheOperator
336
ArangoDeploymentCustomResourceTheArangoDBDeploymentOperatorcreatesandmaintainsArangoDBdeploymentsinaKubernetescluster,givenadeploymentspecification.ThisdeploymentspecificationisaCustomResourcefollowingaCustomResourceDefinitioncreatedbytheoperator.
ExampleminimaldeploymentdefinitionofanArangoDBdatabasecluster:
apiVersion:"database.arangodb.com/v1alpha"
kind:"ArangoDeployment"
metadata:
name:"example-arangodb-cluster"
spec:
mode:Cluster
Examplemoreelaboratedeploymentdefinition:
apiVersion:"database.arangodb.com/v1alpha"
kind:"ArangoDeployment"
metadata:
name:"example-arangodb-cluster"
spec:
mode:Cluster
environment:Production
agents:
count:3
args:
---log.level=debug
resources:
requests:
storage:8Gi
storageClassName:ssd
dbservers:
count:5
resources:
requests:
storage:80Gi
storageClassName:ssd
coordinators:
count:3
image:"arangodb/arangodb:3.3.4"
Specificationreference
Belowyou'llfindallsettingsoftheArangoDeploymentcustomresource.Severalsettingsareforvariousgroupsofservers.Theseareindicatedwith<group>where<group>canbeanyof:
agentsforallagentsofaClusterorActiveFailoverpair.dbserversforalldbserversofaCluster.coordinatorsforallcoordinatorsofaCluster.singleforallsingleserversofaSingleinstanceorActiveFailoverpair.syncmastersforallsyncmastersofaCluster.syncworkersforallsyncworkersofaCluster.
spec.mode:string
Thissettingspecifiesthetypeofdeploymentyouwanttocreate.Possiblevaluesare:
Cluster(default)Fullcluster.Defaultsto3agents,3dbservers&3coordinators.ActiveFailoverActive-failoversinglepair.Defaultsto3agentsand2singleservers.SingleSingleserveronly(notethisdoesnotprovidehighavailabilityorreliability).
Thissettingcannotbechangedafterthedeploymenthasbeencreated.
DeploymentResourceReference
338
spec.environment:string
Thissettingspecifiesthetypeofenvironmentinwhichthedeploymentiscreated.Possiblevaluesare:
Development(default)Thisvalueoptimizesthedeploymentfordevelopmentuse.Itispossibletorunadeploymentonasmallnumberofnodes(e.g.minikube).ProductionThisvalueoptimizesthedeploymentforproductionuse.Itputsrequiredaffinityconstraintsonallpodstoavoidagents&dbserversfromrunningonthesamemachine.
spec.image:string
ThissettingspecifiesthedockerimagetouseforallArangoDBservers.Inadevelopmentenvironmentthissettingdefaultstoarangodb/arangodb:latest.Forproductionenvironmentsthisisarequiredsettingwithoutadefaultvalue.Itishighlyrecommendtouseexplicitversion(notlatest)forproductionenvironments.
spec.imagePullPolicy:string
ThissettingspecifiesthepullpolicyforthedockerimagetouseforallArangoDBservers.Possiblevaluesare:
IfNotPresent(default)topullonlywhentheimageisnotfoundonthenode.Alwaystoalwayspulltheimagebeforeusingit.
spec.storageEngine:string
Thissettingspecifiesthetypeofstorageengineusedforallserversinthecluster.Possiblevaluesare:
MMFilesTousetheMMFilesstorageengine.RocksDB(default)TousetheRocksDBstorageengine.
Thissettingcannotbechangedaftertheclusterhasbeencreated.
spec.downtimeAllowed:bool
ThissettingisusedtoallowautomaticreconciliationactionsthatyieldsomedowntimeoftheArangoDBdeployment.Whenthissettingissettofalse(thedefault),noautomaticactionthatmayresultindowntimeisallowed.Iftheneedforsuchanactionisdetected,aneventisaddedtotheArangoDeployment.
Oncethissettingissettotrue,theautomaticactionisexecuted.
Operationsthatmayresultindowntimeare:
RotatingTLSCAcertificate
Note:ItisstillpossiblethatthereissomedowntimewhentheKubernetesclusterisdown,orinabadstate,irrespectiveofthevalueofthissetting.
spec.rocksdb.encryption.keySecretName
ThissettingspecifiesthenameofakubernetesSecretthatcontainsanencryptionkeyusedforencryptingalldatastoredbyArangoDBservers.Whenanencryptionkeyisused,encryptionofthedataintheclusterisenabled,withoutitencryptionisdisabled.Thedefaultvalueisempty.
ThisrequirestheEnterpriseversion.
Theencryptionkeycannotbechangedaftertheclusterhasbeencreated.
Thesecretspecifiedbythissetting,musthaveadatafieldnamed'key'containinganencryptionkeythatisexactly32byteslong.
spec.externalAccess.type:string
ThissettingspecifiesthetypeofServicethatwillbecreatedtoprovideaccesstotheArangoDBdeploymentfromoutsidetheKubernetescluster.Possiblevaluesare:
DeploymentResourceReference
339
NoneTolimitaccesstoapplicationrunninginsidetheKubernetescluster.LoadBalancerTocreateaServiceoftypeLoadBalancerfortheArangoDBdeployment.NodePortTocreateaServiceoftypeNodePortfortheArangoDBdeployment.Auto(default)TocreateaServiceoftypeLoadBalancerandfallbacktoaServiceortypeNodePortwhentheLoadBalancerisnotassignedanIPaddress.
spec.externalAccess.loadBalancerIP:string
ThissettingspecifiestheIPusedtofortheLoadBalancertoexposetheArangoDBdeploymenton.Thissettingisusedwhenspec.externalAccess.typeissettoLoadBalancerorAuto.
Ifyoudonotspecifythissetting,anIPwillbechosenautomaticallybytheload-balancerprovisioner.
spec.externalAccess.nodePort:int
ThissettingspecifiestheportusedtoexposetheArangoDBdeploymenton.Thissettingisusedwhenspec.externalAccess.typeissettoNodePortorAuto.
Ifyoudonotspecifythissetting,arandomportwillbechosenautomatically.
spec.auth.jwtSecretName:string
ThissettingspecifiesthenameofakubernetesSecretthatcontainstheJWTtokenusedforaccessingallArangoDBservers.Whennonameisspecified,itdefaultsto<deployment-name>-jwt.Todisableauthentication,setthisvaluetoNone.
IfyouspecifyanameofaSecret,thatsecretmusthavethetokeninadatafieldnamedtoken.
IfyouspecifyanameofaSecretthatdoesnotexist,arandomtokeniscreatedandstoredinaSecretwithgivenname.
ChangingaJWTtokenresultsinstoppingtheentireclusterandrestartingit.
spec.tls.caSecretName:string
ThissettingspecifiesthenameofakubernetesSecretthatcontainsastandardCAcertificate+privatekeyusedtosigncertificatesforindividualArangoDBservers.Whennonameisspecified,itdefaultsto<deployment-name>-ca.Todisableauthentication,setthisvaluetoNone.
IfyouspecifyanameofaSecretthatdoesnotexist,aself-signedCAcertificate+keyiscreatedandstoredinaSecretwithgivenname.
ThespecifiedSecret,mustcontainthefollowingdatafields:
ca.crtPEMencodedpublickeyoftheCAcertificateca.keyPEMencodedprivatekeyoftheCAcertificate
spec.tls.altNames:[]string
Thissettingspecifiesalistofalternatenamesthatwillbeaddedtoallgeneratedcertificates.ThesenamescanbeDNSnamesoremailaddresses.Thedefaultvalueisempty.
spec.tls.ttl:duration
Thissettingspecifiesthetimetoliveofallgeneratedservercertificates.Thedefaultvalueis2160h(about3month).
Whentheservercertificateisabouttoexpire,itwillbeautomaticallyreplacedbyanewoneandtheaffectedserverwillberestarted.
Note:ThetimetoliveoftheCAcertificate(whencreatedautomatically)willbesetto10years.
spec.sync.enabled:bool
Thissettingenables/disablessupportfordatacenter2datacenterreplicationinthecluster.Whenenabled,theclusterwillcontainanumberofsyncmaster&syncworkerservers.Thedefaultvalueisfalse.
DeploymentResourceReference
340
spec.sync.externalAccess.type:string
ThissettingspecifiesthetypeofServicethatwillbecreatedtoprovideaccesstotheArangoSyncsyncMastersfromoutsidetheKubernetescluster.Possiblevaluesare:
NoneTolimitaccesstoapplicationsrunninginsidetheKubernetescluster.LoadBalancerTocreateaServiceoftypeLoadBalancerfortheArangoSyncSyncMasters.NodePortTocreateaServiceoftypeNodePortfortheArangoSyncSyncMasters.Auto(default)TocreateaServiceoftypeLoadBalancerandfallbacktoaServiceortypeNodePortwhentheLoadBalancerisnotassignedanIPaddress.
NotethatwhenyouspecifyavalueofNone,aServicewillstillbecreated,butoftypeClusterIP.
spec.sync.externalAccess.loadBalancerIP:string
ThissettingspecifiestheIPusedfortheLoadBalancertoexposetheArangoSyncSyncMasterson.Thissettingisusedwhenspec.sync.externalAccess.typeissettoLoadBalancerorAuto.
Ifyoudonotspecifythissetting,anIPwillbechosenautomaticallybytheload-balancerprovisioner.
spec.sync.externalAccess.nodePort:int
ThissettingspecifiestheportusedtoexposetheArangoSyncSyncMasterson.Thissettingisusedwhenspec.sync.externalAccess.typeissettoNodePortorAuto.
Ifyoudonotspecifythissetting,arandomportwillbechosenautomatically.
spec.sync.externalAccess.masterEndpoint:[]string
Thissettingspecifiesthemasterendpoint(s)advertisedbytheArangoSyncSyncMasters.Ifnotset,thissettingdefaultsto:
Ifspec.sync.externalAccess.loadBalancerIPisset,itdefaultstohttps://<load-balancer-ip>:<8629>.Otherwiseitdefaultstohttps://<sync-service-dns-name>:<8629>.
spec.sync.externalAccess.accessPackageSecretNames:[]string
ThissettingspecifiesthenamesofzeroofmoreSecretsthatwillbecreatedbythedeploymentoperatorcontaining"accesspackages".AnaccesspackagecontainsthoseSecretsthatareneededtoaccesstheSyncMastersofthisArangoDeployment.
Byremovinganamefromthissetting,thecorrespondingSecretisalsodeleted.Notethattoremoveallaccesspackages,leaveanemptyarrayinplace([]).Completelyremovingthesettingresultsinnotmodifyingthelist.
SeetheArangoDeploymentReplicationspecificationformoreinformationonaccesspackages.
spec.sync.auth.jwtSecretName:string
ThissettingspecifiesthenameofakubernetesSecretthatcontainstheJWTtokenusedforaccessingallArangoSyncmasterservers.Whennotspecified,thespec.auth.jwtSecretNamevalueisused.
IfyouspecifyanameofaSecretthatdoesnotexist,arandomtokeniscreatedandstoredinaSecretwithgivenname.
spec.sync.auth.clientCASecretName:string
ThissettingspecifiesthenameofakubernetesSecretthatcontainsaPEMencodedCAcertificateusedforclientcertificateverificationinallArangoSyncmasterservers.Thisisarequiredsettingwhenspec.sync.enabledistrue.Thedefaultvalueisempty.
spec.sync.mq.type:string
ThissettingsetsthetypeofmessagequeueusedbyArangoSync.Possiblevaluesare:
Direct(default)fordirectHTTPconnectionsbetweenthe2datacenters.
DeploymentResourceReference
341
spec.sync.tls.caSecretName:string
ThissettingspecifiesthenameofakubernetesSecretthatcontainsastandardCAcertificate+privatekeyusedtosigncertificatesforindividualArangoSyncmasterservers.
Whennonameisspecified,itdefaultsto<deployment-name>-sync-ca.
IfyouspecifyanameofaSecretthatdoesnotexist,aself-signedCAcertificate+keyiscreatedandstoredinaSecretwithgivenname.
ThespecifiedSecret,mustcontainthefollowingdatafields:
ca.crtPEMencodedpublickeyoftheCAcertificateca.keyPEMencodedprivatekeyoftheCAcertificate
spec.sync.tls.altNames:[]string
Thissettingspecifiesalistofalternatenamesthatwillbeaddedtoallgeneratedcertificates.ThesenamescanbeDNSnamesoremailaddresses.Thedefaultvalueisempty.
spec.sync.monitoring.tokenSecretName:string
ThissettingspecifiesthenameofakubernetesSecretthatcontainsthebearertokenusedforaccessingallmonitoringendpointsofallArangoSyncservers.Whennotspecified,nomonitoringtokenisused.Thedefaultvalueisempty.
spec.ipv6.forbidden:bool
ThissettingpreventstheuseofIPv6addressesbyArangoDBservers.Thedefaultisfalse.
spec.<group>.count:number
Thissettingspecifiesthenumberofserverstostartforthegivengroup.Fortheagentgroup,thisvaluemustbeapositive,oddnumber.Thedefaultvalueis3forallgroupsexceptsingle(therethedefaultis1forspec.mode:Singleand2forspec.mode:ActiveFailover).
Forthesyncworkersgroup,itishighlyrecommendedtousethesamenumberasforthedbserversgroup.
spec.<group>.args:[string]
Thissettingspecifiesadditionalcommandlineargumentspassedtoallserversofthisgroup.Thedefaultvalueisanemptyarray.
spec.<group>.resources.requests.cpu:cpuUnit
ThissettingspecifiestheamountofCPUrequestedbyserverofthisgroup.
Seehttps://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/fordetails.
spec.<group>.resources.requests.memory:memoryUnit
Thissettingspecifiestheamountofmemoryrequestedbyserverofthisgroup.
Seehttps://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/fordetails.
spec.<group>.resources.requests.storage:storageUnit
Thissettingspecifiestheamountofstoragerequiredforeachserverofthisgroup.Thedefaultvalueis8Gi.
Thissettingisnotavailableforgroupcoordinators,syncmasters&syncworkersbecauseserversinthesegroupsdonotneedpersistentstorage.
spec.<group>.serviceAccountName:string
DeploymentResourceReference
342
ThissettingspecifiestheserviceAccountNameforthePodscreatedforeachserverofthisgroup.
UsinganalternativeServiceAccountistypicallyusedtoseparateaccessrights.TheArangoDBdeploymentsdonotrequireanyspecialrights.
spec.<group>.storageClassName:string
ThissettingspecifiesthestorageClassforthePersistentVolumescreatedforeachserverofthisgroup.
Thissettingisnotavailableforgroupcoordinators,syncmasters&syncworkersbecauseserversinthesegroupsdonotneedpersistentstorage.
spec.<group>.tolerations:[Toleration]
ThissettingspecifiesthetolerationsforthePodscreatedforeachserverofthisgroup.
Bydefault,suitabletolerationsaresetforthefollowingkeyswiththeNoExecuteeffect:
node.kubernetes.io/not-ready
node.kubernetes.io/unreachable
node.alpha.kubernetes.io/unreachable(willberemovedinfutureversion)
Formoreinformationontolerations,consulttheKubernetesdocumentation.
DeploymentResourceReference
343
ConfiguringyourdriverforArangoDBaccessInthischapteryou'lllearnhowtoconfigureadriverforaccessinganArangoDBdeploymentinKubernetes.
Theexactmethodstoconfigureadriverarespecifictothatdriver.
Databaseendpoint(s)
Theendpoint(s)(orURLs)tocommunicatewithisthemostimportantparameteryourneedtoconfigureinyourdriver.
FindingtherightendpointsdependonwetheryourclientapplicationisrunninginthesameKubernetesclusterastheArangoDBdeploymentornot.
ClientapplicationinsameKubernetescluster
IfyourclientapplicationisrunninginthesameKubernetesclusterastheArangoDBdeployment,youshouldconfigureyourdrivertousethefollowingendpoint:
https://<deployment-name>.<namespace>.svc:8529
Onlyifyourdeploymenthassetspec.tls.caSecretNametoNone,shouldyouusehttpinsteadofhttps.
ClientapplicationoutsideKubernetescluster
IfyourclientapplicationisrunningoutsidetheKubernetesclusterinwhichtheArangoDBdeploymentisrunning,yourdriverendpointdependsontheexternal-accessconfigurationofyourArangoDBdeployment.
Iftheexternal-accessoftheArangoDBdeploymentisoftypeLoadBalancer,thenusetheIPaddressofthatLoadBalancerlikethis:
https://<load-balancer-ip>:8529
Iftheexternal-accessoftheArangoDBdeploymentisoftypeNodePort,thenusetheIPaddress(es)oftheNodesoftheKubernetescluster,combinedwiththeNodePortthatisusedbytheexternal-accessservice.
Forexample:
https://<kubernetes-node-1-ip>:30123
Youcanfindthetypeofexternal-accessbyinspectingtheexternal-accessService.Todoso,runthefollowingcommand:
kubectlgetservice-n<namespace-of-deployment><deployment-name>-ea
Theoutputlookslikethis:
NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGESELECTOR
example-simple-cluster-eaLoadBalancer10.106.175.38192.168.10.2088529:31890/TCP1sapp=arangodb,arango_de
ployment=example-simple-cluster,role=coordinator
Inthiscasetheexternal-accessisoftypeLoadBalancerwithaload-balancerIPaddressof192.168.10.208.Thisresultsinanendpointofhttps://192.168.10.208:8529.
TLSsettingsAsmentionedbeforetheArangoDBdeploymentmanagedbytheArangoDBoperatorwilluseasecure(TLS)connectionunlessyousetspec.tls.caSecretNametoNoneinyourArangoDeployment.
DriverConfiguration
344
Whenusingasecureconnection,youcanchoosetoverifytheservercertificatesprovidesbytheArangoDBserversornot.
Ifyouwanttoverifythesecertificates,configureyourdriverwiththeCAcertificatefoundinaKubernetesSecretfoundinthesamenamespaceastheArangoDeployment.
ThenameofthisSecretisstoredinthespec.tls.caSecretNamesettingoftheArangoDeployment.Ifyoudon'tsetthissettingexplicitly,itwillbesetautomatically.
ThenfetchtheCAsecretusingthefollowingcommand(oruseaKubernetesclientlibrarytofetchit):
kubectlgetsecret-n<namespace><secret-name>--template='{{index.data"ca.crt"}}'|base64-D>ca.crt
Thisresultsinafilecalledca.crtcontainingaPEMencoded,x509CAcertificate.
QueryrequestsFormostclientrequestsmadebyadriver,itdoesnotmatterifthereisanykindofload-balancerbetweenyourclientapplicationandtheArangoDBdeployment.
NotethatevenasimpleServiceoftypeClusterIPalreadybehavesasaload-balancer.
TheexceptiontothisiscursorrelatedrequestsmadetoanArangoDBClusterdeployment.Thecoordinatorthathandlesaninitialqueryrequest(thatresultsinaCursor)willsavesomein-memorystateinthatcoordinator,iftheresultofthequeryistoobigtobetransferbackintheresponseoftheinitialrequest.
Follow-uprequestshavetobemadetofetchtheremainingdata.Thesefollow-uprequestsmustbehandledbythesamecoordinatortowhichtheinitialrequestwasmade.
Assoonasthereisaload-balancerbetweenyourclientapplicationandtheArangoDBcluster,itisuncertainwhichcoordinatorwillactuallyhandlethefollow-uprequest.
Toresolvethisuncertainty,makesuretorunyourclientapplicationinthesameKubernetesclusterandsynchronizeyourendpointsbeforemakingtheinitialqueryrequest.Thiswillresultintheuse(bythedriver)ofinternalDNSnamesofallcoordinators.Afollow-uprequestcanthenbesenttoexactlythesamecoordinator.
IfyourclientapplicationisrunningoutsidetheKubernetesclusterthisismuchhardertosolve.Theeasiestwaytoworkaroundit,isbymakingsurethatthequeryresultsaresmallenough.Whenthatisnotfeasible,itisalsopossibletoresolvethiswhentheinternalDNSnamesofyourKubernetesclusterareexposedtoyourclientapplicationandtheresulingIPaddressesarerouteablefromyourclientapplication.ToexposeinternalDNSnamesofyourKubernetescluster,yourcanuseCoreDNS.
DriverConfiguration
345
AuthenticationTheArangoDBKubernetesOperatorwillbydefaultcreateArangoDBdeploymentsthatrequireauthenticationtoaccessthedatabase.
ItusesasingleJWTsecret(storedinaKubernetessecret)toprovidesuper-useraccessbetweenallserversofthedeploymentaswellasaccessfromtheArangoDBOperatortothedeployment.
Todisableauthentication,setspec.auth.jwtSecretNametoNone.
Initiallythedeploymentisaccessiblethroughthewebuser-interfaceandAPI's,usingtheuserrootwithanemptypassword.Makesuretochangethispasswordimmediatelyafterstartingthedeployment!
Seealso
Secureconnections(TLS)
Authentication
346
ScalingTheArangoDBKubernetesOperatorsupportsupanddownscalingofthenumberofdbservers&coordinators.
Currentlyitisnotpossibletochangethenumberofagentsofacluster.
Thescaleupordown,changethenumberofserversinthecustomresource.
E.g.changespec.dbservers.countfrom3to4.
Thenapplytheupdatedresourceusing:
kubectlapply-fyourCustomResourceFile.yaml
Inspectthestatusofthecustomresourcetomonitortheprogressofthescalingoperation.
Scaling
347
UpgradingTheArangoDBKubernetesOperatorsupportsupgradinganArangoDBfromoneversiontothenext.
UpgradeanArangoDBdeployment
Toupgradeacluster,changetheversionbychangingthespec.imagesettingandtheapplytheupdatedcustomresourceusing:
kubectlapply-fyourCustomResourceFile.yaml
TheArangoDBoperatorwillperformansequentialupgradeofallserversinyourdeployment.Onlyoneserverisupgradedatatime.
Forpatchlevelupgrades(e.g.3.3.9to3.3.10)eachserverisstoppedandrestartedwiththenewversion.
Forminorlevelupgrades(e.g.3.3.9to3.4.0)eachserverisstopped,thenthenewversionisstartedwith--database.auto-upgradeandoncethatisfinishthenewversionisstartedwiththenormalarguments.
Theprocessformajorlevelupgradesdependsonthespecificversion.
UpgradetheoperatoritselfToupdatetheArangoDBKubernetesOperatoritselftoanewversion,updatetheimageversionofthedeploymentresourceandapplyitusing:
kubectlapply-fexamples/yourUpdatedDeployment.yaml
Seealso
Scaling
Upgrading
348
Configuration&secretsAnArangoDBclusterhaslotsofconfigurationoptions.SomewillbesupporteddirectlyintheArangoDBOperator,otherswillhavetospecifiedseparately.
Built-inoptions
Allbuilt-inoptionsarepassedtoArangoDBserversviacommandlineargumentsconfiguredinthePod-spec.
Otherconfigurationoptions
Allcommandlineoptionsofarangod(andarangosync)areavailablebyaddingoptionstothespec.<group>.argslistofagroupofservers.
Theseargumentsareaddedtothcommandlinecreatedfortheseservers.
Secrets
TheArangoDBclusterneedsseveralsecretssuchasJWTtokensTLScertificatesandsoon.
AllthesesecretsarestoredasKubernetesSecretsandpassedtotheapplicablePodsasfiles,mappedintothePodsfilesystem.
Thenameofthesecretisspecifiedinthecustomresource.Forexample:
apiVersion:"cluster.arangodb.com/v1alpha"
kind:"Cluster"
metadata:
name:"example-arangodb-cluster"
spec:
mode:Cluster
auth:
jwtSecretName:<name-of-JWT-token-secret>
ArangoDBConfiguration&Secrets
349
MetricsTheArangoDBKubernetesOperator(kube-arangodb)exposesmetricsofitsoperationsinaformatthatiscompatiblewithPrometheus.
ThemetricsareexposedthroughHTTPSonport8528underpath/metrics.
Lookatexamples/metricsforexamplesofServicesandServiceMonitorsyoucanusetointegratewithPrometheusthroughthePrometheus-OperatorbyCoreOS.
Metrics
350
ServicesandloadbalancerTheArangoDBKubernetesOperatorwillcreateservicesthatcanbeusedtoreachtheArangoDBserversfrominsidetheKubernetescluster.
Bydefault,theArangoDBKubernetesOperatorwillalsocreateanadditionalservicetoreachtheArangoDBdeploymentfromoutsidetheKubernetescluster.
ForexposingtheArangoDBdeploymenttotheoutside,thereare2options:
UsingaNodePortservice.Thiswillexposethedeploymentonaspecificport(above30.000)onallnodesoftheKubernetescluster.UsingaLoadBalancerservice.Thiswillexposethedeploymentonaload-balancerthatisprovisionedbytheKubernetescluster.
TheLoadBalanceroptionisthemostconvenient,butnotallKubernetesclustersareabletoprovisionaload-balancer.Thereforeweofferathird(anddefault)option:Auto.Inthisoption,theArangoDBKubernetesOperatortriestocreateaLoadBalancerservice.ItthenwaitsforuptoaminutefortheKubernetesclustertoprovisionaload-balancerforit.Ifthathasnothappenedafteraminute,theserviceisreplacedbyaserviceoftypeNodePort.
Toinspectthecreatedservice,run:
kubectlgetservices<deployment-name>-ea
TousetheArangoDBserversfromoutsidetheKubernetesclusteryouhavetoaddanotherserviceasexplainedbelow.
ServicesIfyoudonotwanttheArangoDBKubernetesOperatortocreateanexternal-accessserviceforyou,setspec.externalAccess.TypetoNone.
Ifyouwanttocreateexternalaccessservicesmanually,followtheinstructionsbelow.
Singleserver
Forasingleserverdeployment,theoperatorcreatesasingleServicenamed<deployment-name>.ThisservicehasanormalclusterIPaddress.
Fullcluster
Forafullclusterdeployment,theoperatorcreatestwoServices.
<deployment-name>-intaheadlessServiceintendedtoprovideDNSnamesforallpodscreatedbytheoperator.ItselectsallArangoDB&ArangoSyncserversinthecluster.
<deployment-name>anormalServicethatselectsonlythecoordinatorsofthecluster.ThisServiceisconfiguredwithClientIPsessionaffinity.Thisisneededforcursorrequests,sincetheyareboundtoaspecificcoordinator.
Whenthecoordinatorsareaskedtoprovideendpointsofthecluster(e.g.whencallingclient.SynchronizeEndpoints()inthegodriver)theDNSnamesoftheindividualPodswillbereturned(<pod>.<deployment-name>-int.<namespace>.svc)
FullclusterwithDC2DC
Forafullclusterwithdatacenterreplicationdeployment,thesameServicesarecreatedasforaFullcluster,withthefollowingadditions:
<deployment-name>-syncanormalServicethatselectsonlythesyncmastersofthecluster.
Loadbalancer
Services&Loadbalancer
351
IfyouwantfullcontroloftheServicesneededtoaccesstheArangoDBdeploymentfromoutsideyourKubernetescluster,setspec.externalAccess.TypeoftheArangoDeploymenttoNoneandcreateaServiceasspecifiedbelow.
CreateaServiceoftypeLoadBalancerorNodePort,dependingonyourKubernetesdeployment.
Thisserviceshouldselect:
arango_deployment:<deployment-name>
role:coordinator
ThefollowingexampleyieldsaserviceoftypeLoadBalancerwithaspecificloadbalancerIPaddress.Withthisservice,theArangoDBclustercannowbereachedonhttps://1.2.3.4:8529.
kind:Service
apiVersion:v1
metadata:
name:arangodb-cluster-exposed
spec:
selector:
arango_deployment:arangodb-cluster
role:coordinator
type:LoadBalancer
loadBalancerIP:1.2.3.4
ports:
-protocol:TCP
port:8529
targetPort:8529
ThefollowingexampleyieldsaserviceoftypeNodePortwiththeArangoDBclusterexposedonport30529ofallnodesoftheKubernetescluster.
kind:Service
apiVersion:v1
metadata:
name:arangodb-cluster-exposed
spec:
selector:
arango_deployment:arangodb-cluster
role:coordinator
type:NodePort
ports:
-protocol:TCP
port:8529
targetPort:8529
nodePort:30529
Services&Loadbalancer
352
ArangoDeploymentReplicationCustomResourceTheArangoDBReplicationOperatorcreatesandmaintainsArangoDBarangosyncconfigurationsinaKubernetescluster,givenareplicationspecification.ThisreplicationspecificationisaCustomResourcefollowingaCustomResourceDefinitioncreatedbytheoperator.
Exampleminimalreplicationdefinitionfor2ArangoDBclusterwithsyncinthesameKubernetescluster:
apiVersion:"replication.database.arangodb.com/v1alpha"
kind:"ArangoDeploymentReplication"
metadata:
name:"replication-from-a-to-b"
spec:
source:
deploymentName:cluster-a
auth:
keyfileSecretName:cluster-a-sync-auth
destination:
deploymentName:cluster-b
Thisdefinitionresultsin:
thearangosyncSyncMasterindeploymentcluster-biscalledtoconfigureasynchronizationfromthesyncmastersincluster-atothesyncmastersincluster-b,usingtheclientauthenticationcertificatestoredinSecretcluster-a-sync-auth.Toaccesscluster-a,theJWTsecretfoundinthedeploymentofcluster-aisused.Toaccesscluster-b,theJWTsecretfoundinthedeploymentofcluster-bisused.
ExamplereplicationdefinitionforreplicatingfromasourcethatisoutsidethecurrentKubernetesclustertoadestinationthatisinthesameKubernetescluster:
apiVersion:"replication.database.arangodb.com/v1alpha"
kind:"ArangoDeploymentReplication"
metadata:
name:"replication-from-a-to-b"
spec:
source:
masterEndpoint:["https://163.172.149.229:31888","https://51.15.225.110:31888","https://51.15.229.133:31888"]
auth:
keyfileSecretName:cluster-a-sync-auth
tls:
caSecretName:cluster-a-sync-ca
destination:
deploymentName:cluster-b
Thisdefinitionresultsin:
thearangosyncSyncMasterindeploymentcluster-biscalledtoconfigureasynchronizationfromthesyncmasterslocatedatthegivenlistofendpointURL'stothesyncmasterscluster-b,usingtheclientauthenticationcertificatestoredinSecretcluster-a-sync-auth.Toaccesscluster-a,thekeyfile(containingaclientauthenticationcertificate)isused.Toaccesscluster-b,theJWTsecretfoundinthedeploymentofcluster-bisused.
Specificationreference
Belowyou'llfindallsettingsoftheArangoDeploymentReplicationcustomresource.
spec.source.deploymentName:string
ThissettingspecifiesthenameofanArangoDeploymentresourcethatrunsaclusterwithsyncenabled.
Thisclusterconfiguredasthereplicationsource.
DeploymentReplicationResourceReference
353
spec.source.masterEndpoint:[]string
ThissettingspecifieszeroormoremasterendpointURL'softhesourcecluster.
UsethissettingifthesourceclusterisnotrunninginsideaKubernetesclusterthatisreachablefromtheKubernetesclustertheArangoDeploymentReplicationresourceisdeployedin.
Specifyingthissettingandspec.source.deploymentNameatthesametimeisnotallowed.
spec.source.auth.keyfileSecretName:string
ThissettingspecifiesthenameofaSecretcontainingaclientauthenticationcertificatecalledtls.keyfileusedtoauthenticatewiththeSyncMasteratthespecifiedsource.
Ifspec.source.auth.userSecretNamehasnotbeenset,theclientauthenticationcertificatefoundinthesecretwiththisnameisalsousedtoconfigurethesynchronizationandfetchthesynchronizationstatus.
Thissettingisrequired.
spec.source.auth.userSecretName:string
ThissettingspecifiesthenameofaSecretcontainingausername&passwordusedtoauthenticatewiththeSyncMasteratthespecifiedsourceinordertoconfiguresynchronizationandfetchsynchronizationstatus.
Theuseridentifiedbytheusernamemusthavewriteaccessinthe_systemdatabaseofthesourceArangoDBcluster.
spec.source.tls.caSecretName:string
ThissettingspecifiesthenameofaSecretcontainingaTLSCAcertificateca.crtusedtoverifytheTLSconnectioncreatedbytheSyncMasteratthespecifiedsource.
Thissettingisrequired,unlessspec.source.deploymentNamehasbeenset.
spec.destination.deploymentName:string
ThissettingspecifiesthenameofanArangoDeploymentresourcethatrunsaclusterwithsyncenabled.
Thisclusterconfiguredasthereplicationdestination.
spec.destination.masterEndpoint:[]string
ThissettingspecifieszeroormoremasterendpointURL'softhedestinationcluster.
UsethissettingifthedestinationclusterisnotrunninginsideaKubernetesclusterthatisreachablefromtheKubernetesclustertheArangoDeploymentReplicationresourceisdeployedin.
Specifyingthissettingandspec.destination.deploymentNameatthesametimeisnotallowed.
spec.destination.auth.keyfileSecretName:string
ThissettingspecifiesthenameofaSecretcontainingaclientauthenticationcertificatecalledtls.keyfileusedtoauthenticatewiththeSyncMasteratthespecifieddestination.
Ifspec.destination.auth.userSecretNamehasnotbeenset,theclientauthenticationcertificatefoundinthesecretwiththisnameisalsousedtoconfigurethesynchronizationandfetchthesynchronizationstatus.
Thissettingisrequired,unlessspec.destination.deploymentNameorspec.destination.auth.userSecretNamehasbeenset.
Specifyingthissettingandspec.destination.userSecretNameatthesametimeisnotallowed.
spec.destination.auth.userSecretName:string
DeploymentReplicationResourceReference
354
ThissettingspecifiesthenameofaSecretcontainingausername&passwordusedtoauthenticatewiththeSyncMasteratthespecifieddestinationinordertoconfiguresynchronizationandfetchsynchronizationstatus.
Theuseridentifiedbytheusernamemusthavewriteaccessinthe_systemdatabaseofthedestinationArangoDBcluster.
Specifyingthissettingandspec.destination.keyfileSecretNameatthesametimeisnotallowed.
spec.destination.tls.caSecretName:string
ThissettingspecifiesthenameofaSecretcontainingaTLSCAcertificateca.crtusedtoverifytheTLSconnectioncreatedbytheSyncMasteratthespecifieddestination.
Thissettingisrequired,unlessspec.destination.deploymentNamehasbeenset.
Authenticationdetails
TheauthenticationsettingsinaArangoDeploymentReplicationresourceareusedfortwodistinctpurposes.
Thefirstuseistheauthenticationofthesyncmastersatthedestinationwiththesyncmastersatthesource.Thisisalwaysdoneusingaclientauthenticationcertificatewhichisfoundinatls.keyfilefieldinasecretidentifiedbyspec.source.auth.keyfileSecretName.
TheseconduseistheauthenticationoftheArangoDBReplicationoperatorwiththesyncmastersatthesourceordestination.Theseconnectionsaremadetoconfiguresynchronization,stopconfigurationandfetchthestatusoftheconfiguration.Themethodusedforthisauthenticationisderivedasfollows(whereXiseithersourceordestination):
Ifspec.X.userSecretNameisset,theusername+passwordfoundintheSecretidentifiedbythisnameisused.Ifspec.X.keyfileSecretNameisset,theclientauthenticationcertificate(keyfile)foundintheSecretidentifierbythisnameisused.Ifspec.X.deploymentNameisset,theJWTsecretfoundinthedeploymentisused.
Creatingclientauthenticationcertificatekeyfiles
TheclientauthenticationcertificatesneededfortheSecretsidentifiedbyspec.source.auth.keyfileSecretName&spec.destination.auth.keyfileSecretNamearenormalArangoDBkeyfilesthatcanbecreatedbythearangosynccreateclient-authkeyfilecommand.Inordertodoso,youmusthaveaccesstotheclientauthenticationCAofthesource/destination.
IftheclientauthenticationCAatthesource/destinationalsocontainsaprivatekey(ca.key),theArangoDeploymentoperatorcanbeusedtocreatesuchakeyfileforyou,withouttheneedtohavearangosyncinstalledlocally.Readthefollowingparagraphsforinstructionsonhowtodothat.
CreatingandusingaccesspackagesAnaccesspackageisaYAMLfilethatcontains:
Aclientauthenticationcertificate,wrappedinaSecretinatls.keyfiledatafield.ATLScertificateauthoritypublickey,wrappedinaSecretinaca.crtdatafield.
TheformatoftheaccesspackageissuchthatitcanbeinsertedintoaKubernetesclusterusingthestandardkubectltool.
TocreateanaccesspackagethatcanbeusedtoauthenticatewiththeArangoDBSyncMastersofanArangoDeployment,addanameofanon-existingSecrettothespec.sync.externalAccess.accessPackageSecretNamesfieldoftheArangoDeployment.Inresponse,aSecretiscreatedinthatKubernetescluster,withthegivenname,thatcontainsaaccessPackage.yamldatafieldthatcontainsaKubernetesresourcespecificationthatcanbeinsertedintotheotherKubernetescluster.
Theprocessforcreatingandusinganaccesspackageforauthenticationatthesourceclusterisasfollows:
EdittheArangoDeploymentresourceofthesourcecluster,setspec.sync.externalAccess.accessPackageSecretNamesto["my-access-package"]
WaitfortheArangoDeploymentoperatortocreateaSecretnamedmy-access-package.ExtracttheaccesspackagefromtheKubernetessourceclusterusing:
DeploymentReplicationResourceReference
355
kubectlgetsecretmy-access-package--template='{{index.data"accessPackage.yaml"}}'|base64-D>accessPackage.yaml
InsertthesecretsfoundintheaccesspackageintheKubernetesdestinationclusterusing:
kubectlapply-faccessPackage.yaml
Asaresult,thedestinationKubernetesclusterwillhave2additionalSecrets.Onecontainsaclientauthenticationcertificateformattedasakeyfile.AnothercontainsthepublickeyoftheTLSCAcertificateofthesourcecluster.
DeploymentReplicationResourceReference
356
StorageAnArangoDBclusterreliesheavilyonfastpersistentstorage.TheArangoDBKubernetesOperatorusesPersistentVolumeClaimstodeliverthestoragetoPodsthatneedthem.
Storageconfiguration
IntheArangoDeploymentresource,onecanspecifythetypeofstorageusedbygroupsofserversusingthespec.<group>.storageClassNamesetting.
ThisisanexampleofaClusterdeploymentthatstoresitsagent&dbserverdataonPersistentVolumesthatusethemy-local-ssdStorageClass
apiVersion:"database.arangodb.com/v1alpha"
kind:"ArangoDeployment"
metadata:
name:"cluster-using-local-ssh"
spec:
mode:Cluster
agents:
storageClassName:my-local-ssd
dbservers:
storageClassName:my-local-ssd
Theamountofstorageneededisconfiguredusingthespec.<group>.resources.requests.storagesetting.
Notethatconfiguringstorageisdonepergroupofservers.Itisnotpossibletoconfigurestorageperindividualserver.
ThisisanexampleofaClusterdeploymentthatrequestsvolumesof80GBforeverydbserver,resultinginatotalstoragecapacityof240GB(with3dbservers).
apiVersion:"database.arangodb.com/v1alpha"
kind:"ArangoDeployment"
metadata:
name:"cluster-using-local-ssh"
spec:
mode:Cluster
dbservers:
resources:
requests:
storage:80Gi
Localstorage
Foroptimalperformance,ArangoDBshouldbeconfiguredwithlocallyattachedSSDstorage.
TheeasiestwaytoaccomplishthisistodeployanArangoLocalStorageresource.TheArangoDBStorageOperatorwilluseittoprovidePersistentVolumesforyou.
ThisisanexampleofanArangoLocalStorageresourcethatwillresultinPersistentVolumescreatedonanynodeoftheKubernetesclusterunderthedirectory/mnt/big-ssd-disk.
apiVersion:"storage.arangodb.com/v1alpha"
kind:"ArangoLocalStorage"
metadata:
name:"example-arangodb-storage"
spec:
storageClass:
name:my-local-ssd
localPath:
-/mnt/big-ssd-disk
Storage
357
NotethatusinglocalstoragerequiredVolumeSchedulingtobeenabledinyourKubernetescluster.ONKubernetes1.10thisisenabledbydefault,onversion1.9youhavetoenableitwitha--feature-gatesetting.
ManuallycreatingPersistentVolumes
ThealternativeistocreatePersistentVolumesmanually,forallserversthatneedpersistentstorage(single,agents&dbservers).E.g.foraClusterwith3agentsand5dbservers,youmustcreate8volumes.
Notethateachvolumemusthaveacapacitythatisequaltoorhigherthanthecapacityneededforeachserver.
Toselectthecorrectnode,addarequirednode-affinityannotationasshownintheexamplebelow.
apiVersion:v1
kind:PersistentVolume
metadata:
name:volume-agent-1
annotations:
"volume.alpha.kubernetes.io/node-affinity":'{
"requiredDuringSchedulingIgnoredDuringExecution":{
"nodeSelectorTerms":[
{"matchExpressions":[
{"key":"kubernetes.io/hostname",
"operator":"In",
"values":["node-1"]
}
]}
]}
}'
spec:
capacity:
storage:100Gi
accessModes:
-ReadWriteOnce
persistentVolumeReclaimPolicy:Delete
storageClassName:local-ssd
local:
path:/mnt/disks/ssd1
ForKubernetes1.9andup,youshouldcreateaStorageClasswhichisconfiguredtobindvolumesontheirfirstuseasshownintheexamplebelow.ThisensuresthattheKubernetesschedulertakesallconstraintsonaPodthatintoconsiderationbeforebindingthevolumetoaclaim.
kind:StorageClass
apiVersion:storage.k8s.io/v1
metadata:
name:local-ssd
provisioner:kubernetes.io/no-provisioner
volumeBindingMode:WaitForFirstConsumer
Storage
358
ArangoLocalStorageCustomResourceTheArangoDBStorageOperatorcreatesandmaintainsArangoDBstorageresourcesinaKubernetescluster,givenastoragespecification.ThisstoragespecificationisaCustomResourcefollowingaCustomResourceDefinitioncreatedbytheoperator.
Exampleminimalstoragedefinition:
apiVersion:"storage.arangodb.com/v1alpha"
kind:"ArangoLocalStorage"
metadata:
name:"example-arangodb-storage"
spec:
storageClass:
name:my-local-ssd
localPath:
-/mnt/big-ssd-disk
Thisdefinitionresultsin:
aStorageClasscalledmy-local-ssdthedynamicprovisioningofPersistentVolume'swithalocalvolumeonanodewherethelocalvolumestartsinasub-directoryof/mnt/big-ssd-disk.thedynamiccleanupofPersistentVolume's(createdbytheoperator)afteroneisreleased.
Theprovisionedvolumeswillhaveacapacitythatmatchestherequestedcapacityofvolumeclaims.
SpecificationreferenceBelowyou'llfindallsettingsoftheArangoLocalStoragecustomresource.
spec.storageClass.name:string
ThissettingspecifiesthenameofthestorageclassthatcreatedPersistentVolumewilluse.
Ifempty,thisfielddefaultstothenameoftheArangoLocalStorageobject.
IfaStorageClasswithgivennamedoesnotyetexist,itwillbecreated.
spec.storageClass.isDefault:bool
ThissettingspecifiesifthecreatedStorageClasswillbemarkedasdefaultstorageclass.(defaultisfalse)
spec.localPath:stringList
Thissettingspecifiesoneofmorelocaldirectories(onthenodes)usedtocreatepersistentvolumesin.
spec.nodeSelector:nodeSelector
Thissettingspecifieswhichnodestheoperatorwillprovisionpersistentvolumeson.
StorageResource
359
Secureconnections(TLS)TheArangoDBKubernetesOperatorwillbydefaultcreateArangoDBdeploymentsthatusesecureTLSconnections.
ItusesasingleCAcertificate(storedinaKubernetessecret)andonecertificateperArangoDBserver(storedinaKubernetessecretperserver).
TodisableTLS,setspec.tls.caSecretNametoNone.
InstallCAcertificateIftheCAcertificateisself-signed,itwillnotbetrustedbybrowsers,untilyouinstallitinthelocaloperatingsystemorbrowser.Thisprocessdiffersperoperatingsystem.
Todoso,youfirsthavetofetchtheCAcertificatefromitsKubernetessecret.
kubectlgetsecret<deploy-name>-ca--template='{{index.data"ca.crt"}}'|base64-D>ca.crt
Windows
ToinstallaCAcertificateinWindows,followtheproceduredescribedhere.
MacOS
ToinstallaCAcertificateinMacOS,run:
sudo/usr/bin/securityadd-trusted-cert-d-rtrustRoot-k/Library/Keychains/System.keychainca.crt
TouninstallaCAcertificateinMacOS,run:
sudo/usr/bin/securityremove-trusted-cert-dca.crt
Linux
ToinstallaCAcertificateinLinux,onUbuntu,run:
sudocpca.crt/usr/local/share/ca-certificates/<some-name>.crt
sudoupdate-ca-certificates
Seealso
Authentication
TLS
360
TroubleshootingWhileKubernetesandtheArangoDBKubernetesoperatorwillautomaticallyresolvealotofissues,therearealwayscaseswherehumanattentionisneeded.
Thischaptergivesyourtips&trickstohelpyoutroubleshootdeployments.
Wheretolook
InKubernetesallresourcescanbeinspectedusingkubectlusingeitherthegetordescribecommand.
Togetalldetailsoftheresource(bothspecification&status),runthefollowingcommand:
kubectlget<resource-type><resource-name>-n<namespace>-oyaml
Forexample,togettheentirespecificationandstatusofanArangoDeploymentresourcenamedmy-arangodbinthedefaultnamespace,run:
kubectlgetArangoDeploymentmy-arango-ndefault-oyaml
#orshorter
kubectlgetarangomy-arango-oyaml
Severaltypesofresources(includingallArangoDBcustomresources)supportevents.Theseeventsshowwhathappenedtotheresourceovertime.
Toshowtheevents(andmostimportantresourcedata)ofaresource,runthefollowingcommand:
kubectldescribe<resource-type><resource-name>-n<namespace>
GettinglogsAnotherinvaluablesourceofinformationisthelogofcontainersbeingruninKubernetes.TheselogsareaccessiblethroughthePodsthatgroupthesecontainers.
TofetchthelogsofthedefaultcontainerrunninginaPod,run:
kubectllogs<pod-name>-n<namespace>
#orwithfollowoptiontokeepinspectinglogswhiletheyarewritten
kubectllogs<pod-name>-n<namespace>-f
ToinspectthelogsofaspecificcontainerinPod,add-c<container-name>.YoucanfindthenamesofthecontainersinthePod,usingkubectldescribepod....
NotethattheArangoDBoperatorsarebeingdeployedthemselvesasaKubernetesDeploymentwith2replicas.Thismeansthatyouwillhavetofetchthelogsof2Podsrunningthosereplicas.
Whatif
ThePodsofadeploymentstayinPendingstate
Therearetwocommoncausesforthis.
Troubleshooting
361
1)ThePodscannotbescheduledbecausetherearenotenoughnodesavailable.Thisisusallyonlythecasewithaspec.environmentsettingthathasavalueofProduction.
Solution:Addmorenodes.
1)TherearenoPersistentVolumesavailabletobeboundtothePersistentVolumeClaimscreatedbytheoperator.
Solution:
UsekubectlgetpersistentvolumestoinspecttheavailablePersistentVolumesandifneeded,usetheArangoLocalStorageoperatortoprovisionPersistentVolumes.
WhenrestartingaNode,thePodsscheduledonthatnoderemaininTerminatingstate
WhenaNodenolongermakesregularcallstotheKubernetesAPIserver,itismarkedasnotavailable.DependingonspecificsettingsinyourPods,KuberneteswillatsomepointdecidetoterminatethePod.AslongastheNodeisnotcompletelyremovedfromtheKubernetesAPIserver,KuberneteswilltrytousetheNodeitselftoterminatethePod.
TheArangoDeploymentoperatorrecognizesthisconditionandwilltrytoreplacethosePodswithPodsondifferentnodes.Theexactbehaviordifferspertypeofserver.
WhathappenswhenaNodewithlocaldataisbroken
WhenaNodewithPersistentVolumeshostedonthatNodeisbrokenandcannotberepaired,thedatainthosePersistentVolumesislost.
IfanArangoDeploymentoftypeSinglewasusingoneofthosePersistentVolumesthedatabaseislostandmustberestoredfromabackup.
IfanArangoDeploymentoftypeActiveFailoverorClusterwasusingoneofthosePersistentVolumes,itdependsonthetypeofserverthatwasusingthevolume.
IfanAgentwasusingthevolume,itcanberepairedaslongas2otheragentsarestillhealthy.IfaDBServerwasusingthevolume,andthereplicationfactorofalldatabasecollectionsis2orhigher,andtheremainingdbserversarestillhealthy,theclusterwillduplicatetheremainingreplicastobringthenumberofreplicasesbacktotheoriginalnumber.IfaDBServerwasusingthevolume,andthereplicationfactorofadatabasecollectionis1andhappenstobestoredonthatdbserver,thedataislost.IfasingleserverofanActiveFailoverdeploymentwasusingthevolume,andtheothersingleserverisstillhealthy,theothersingleserverwillbecomeleader.Afterreplacingthefailedsingleserver,thenewfollowerwillsynchronizewiththeleader.
Troubleshooting
362
AdministrationMostadministrationcanbemanagedusingthearangosh.
FilesystemsAsonewouldexpectforadatabase,werecommendalocallymountedfilesystems.
NFSorsimilarnetworkfilesystemswillnotwork.
OnLinuxwerecommendtheuseofext4fs,onWindowsNTFSandonMacOSHFS+.
WerecommendtonotuseBTRFSonLinux.ItisknowntonotworkwellinconjunctionwithArangoDB.WeexperiencedthatArangoDBfaceslatencyissuesonaccessingitsdatabasefilesonBTRFSpartitions.InconjunctionwithBTRFSandAUFSwealsosawdatalossonrestart.
Administration
363
WebInterfaceArangoDBcomeswithabuilt-inwebinterfaceforadministration.Theinterfacediffersforstandaloneinstancesandclustersetups.
Standalone:
Cluster:
WebInterface
364
DashboardTheDashboardtabprovidesstatisticswhicharepolledregularlyfromtheArangoDBserver.
RequestsStatistics:
RequestspersecondRequesttypesNumberofclientconnectionsTransfersizeTransfersize(distribution)AveragerequesttimeAveragerequesttime(distribution)
SystemResources:
NumberofthreadsMemoryVirtualsizeMajorpagefaultsUsedCPUtime
Replication:
ReplicationstateTotalsTicksProgress
Dashboard
366
ClusterTheclustersectiondisplaysstatisticsaboutthegeneralclusterperformance.
Statistics:
AvailableandmissingcoordinatorsAvailableandmissingdatabaseserversMemoryusage(percent)CurrentconnectionsData(bytes)HTTP(bytes)Averagerequesttime(seconds)
Nodes
Overview
Theoverviewshowsavailableandmissingcoordinatorsanddatabaseservers.
Cluster
367
Functions:
CoordinatorDashboard:ClickonaCoordinatorwillopenastatisticsdashboard.
Information(Coordinator/Databaseservers):
NameEndpointLastHeartbeatStatusHealth
Shards
Theshardsectiondisplaysallavailableshardedcollections.
Cluster
368
Functions:
MoveShardLeader:Clickonaleaderdatabaseofashardserverwillopenamovesharddialog.Shardscanbetransferredtoallavailabledatabasservers,excepttheleadingdatabaseserveroranavailablefollower.MoveShardFollower:Clickonafollowerdatabaseofashardwillopenamovesharddialog.Shardscanbetransferredtoallavailabledatabasservers,excepttheleadingdatabaseserveroranavailablefollower.RebalanceShards:Anewdatabaseserverwillnothaveanyshards.Withtherebalancefunctionalitytheclusterwillstarttorebalanceshardsincludingemptydatabaseservers.
Information(collection):
ShardLeader(greenstate:synciscomplete)Followers
Cluster
369
CollectionsThecollectionssectiondisplaysallavailablecollections.Fromhereyoucancreatenewcollectionsandjumpintoacollectionfordetails(clickonacollectiontile).
Functions:
A:TogglefilterpropertiesB:SearchcollectionbynameD:CreatecollectionC:FilterpropertiesH:Showcollectiondetails(clicktile)
Information:
E:CollectiontypeF:Collectionstate(unloaded,loaded,...)G:Collectionname
Collection
Collections
370
Therearefourviewcategories:
1. Content:
CreateadocumentDeleteadocumentFilterdocumentsDownloaddocumentsUploaddocuments
2. Indices:
CreateindicesDeleteindices
3. Info:
Detailedcollectioninformationandstatistics4. Settings:
Configurename,journalsize,indexbuckets,waitforsyncDeletecollectionTruncatecollectionUnload/LoadcollectionSavemodifedproperties(name,journalsize,indexbuckets,waitforsync)
Additionalinformation:
Uploadformat:
I.Line-wise
{"_key":"key1",...}
{"_key":"key2",...}
II.JSONdocumentsinalist
[
{"_key":"key1",...},
{"_key":"key2",...}
Collections
371
DocumentThedocumentsectionoffersaeditorwhichletyoueditdocumentsandedgesofacollection.
Functions:
EditdocumentSavedocumentDeletedocmentSwitchbetweenTree/Code-ModeCreateanewdocument
Information:
Displays:_id,_rev,_keyproperties
Document
373
QueryViewThequeryviewoffersyouthreedifferentsubviews:
EditorRunningQueriesSlowQueryHistory
AQLQueryEditor
ThewebinterfaceoffersaAQLQueryEditor:
Theeditorissplitintotwoparts,thequeryeditorpaneandthebindparameterpane.
Theleftpaneisyourregularqueryinputfield,whereyoucaneditandthenexecuteorexplainyourqueries.Bydefault,theenteredbindparameterwillautomaticallyberecognizedandshowninthebindparametertableintherightpane,whereyoucaneasilyeditthem.
Theinputfieldsareequippedwithtypedetection.Thismeansyoudon'thavetousequotemarksaroundstring,justwritethemas-is.Numberswillbetreatedasnumbers,trueandfalseasbooleans,nullasnull-typevalue.Squarebracketscanbeusedtodefinearrays,andcurlybracesforobjects(keysandvalueshavetobesurroundedbydoublequotes).Thiswillmostlybewhatyouwant.Butifyouwanttoforcesomethingtobetreatedasstring,usequotationmarksforthevalue:
123//interpretedasnumber
"123"//interpretedasstring
["foo","bar",123,true]//interpretedasarray
['foo','bar',123,true]//interpretedasstring
Queries
374
IfyouareusedtoworkwithJSON,youmaywanttoswitchthebindparametereditortoJSONmodebyclickingontheupperrighttogglebutton.YoucantheneditthebindparametersinrawJSONformat.
CustomQueries
TosavethecurrentqueryusetheSavebuttoninthetopleftcorneroftheeditororusetheshortcut(seebelow).
BypressingtheQueriesbuttoninthetopleftcorneroftheeditoryouactivatethecustomqueriesview.Hereyoucanselectapreviouslystoredcustomqueryoroneofourqueryexamples.
Clickonaquerytitletogetacodepreview.Inaddition,thereareactionbuttonsto:
CopytoeditorExplainqueryRunqueryDeletequery
Forthebuilt-inexamplequeries,thereisonlyCopytoeditoravailable.
ToexportorimportqueriestoandfromJSONyoucanusethebuttonsontheright-handside.
Result
Queries
375
Eachqueryyouexecuteorexplainopensupanewresultbox,soyouareabletofireupmultiplequeriesandviewtheirresultsatthesametime.Everyqueryresultboxgivesyoudetailedqueryinformationandofcoursethequeryresultitself.Theresultboxescanbedismissedindividually,oraltogetherusingtheRemoveresultsbutton.ThetogglebuttoninthetoprightcornerofeachboxswitchesbackandforthbetweentheResultandAQLquerywithbindparameters.
Spotlight
Thespotlightfeatureopensupamodalview.ThereyoucanfindallAQLkeywords,AQLfunctionsandcollections(filteredbytheirtype)tohelpyoutobemoreproductiveinwritingyourqueries.Spotlightcanbeopenedbythemagicwandiconinthetoolbarorviashortcut(seebelow).
AQLEditorShortcuts
Queries
376
Ctrl/Cmd+ReturntoexecuteaqueryCtrl/Cmd+Shift+ReturntoexplainaqueryCtrl/Cmd+Shift+StosavethecurrentqueryCtrl/Cmd+Shift+CtotogglecommentsCtrl+SpacetoopenupthespotlightsearchCtrl+Cmd+ZtoundolastchangeCtrl+Cmd+Shift+Ztoredolastchange
RunningQueries
TheRunningQueriestabgivesyouacompactoverviewofallrunningqueries.Byclickingtheredminusbutton,youcanaborttheexecutionofarunningquery.
SlowQueryHistory
Queries
377
GraphsTheGraphstabprovidesaviewerfacilityforgraphdatastoredinArangoDB.ItallowsbrowsingArangoDBgraphsstoredinthe_graphssystemcollectionoragraphconsistingofanarbitraryvertexandedgecollection.
Pleasenotethatthegraphviewerrequirescanvas(optional:webgl)supportinyourbrowser.EspeciallyInternetExplorerbrowsersolderthanversion9arelikelytonotsupportthis.
GraphViewer
Graphs
379
TopToolbarFunctions:
Loadfullgraph(Alsonodeswithoutconnectionswillbedrawn.Usefulduringgraphmodelingsetup)TakeagraphscreenshotStartfullscreenmodeOpengraphoptionsmenu
DefaultContextMenu(mouse-clickbackground):
AddanewnodeClosevisiblecontextmenu(s)
NodeContextMenu(mouse-clicknode):
DeletenodeEditnodeExpandnode(Showallboundedges)Drawedge(Connectwithanothernode)Setasstartnode(TheGraphwillrerenderstartingtheselectednodeandgivenoptions(graphoptionsmenu))
EdgeContextMenu(mouse-clickedge):
EditedgeDeleteedge
EdgeHighlighting(right-mouse-clicknode):
Highlightalledgesconnectedtothenode(right-clickatthebackgroundwillremovehighlighting)
Graphs
380
GraphViewerOptionsGraphOptionsMenu:
Startnode(string-validnodeidorspaceseperatedlistofid's):Heartofyourgraph.Renderingandtraversingwillstartfromhere.Emptyvaluemeans:arandomstartingpointwillbeused.Layout:Differentgraphlayoutingalgoritms.Nooverlap(optimal:biggraph),forcelayout(optimal:mediumgraph),fruchtermann(optimal:littletomediumgraph).Renderer:Canvasmodeallowsediting.WebGLcurrentlyoffersonlydisplaymode(alotfasterwithmuchnodes/edges).Searchdepth(number):Searchdepthwhichisstartingfromyourstartnode.Limit(number):Limitnodescount.Ifemptyorzero,nolimitisset.
NodesOptionsMenu:
Label(string):Nodeswillbelabeledbythisattribute.Ifnodeattributeisnotfound,nolabelwillbedisplayed.AddCollectionName:Thisappendsthecollectionnametothelabel,ifitexists.ColorByCollections:Shouldnodesbecolorizedbytheircollection?Ifenabled,nodecolorandnodecolorattributewillbeignored.Color:Defaultnodecolor.ColorAttribute(string):Ifanattributeisgiven,nodeswillthenbecolorizedbytheattribute.Thissettingignoresdefaultnodecolorifset.SizeByConnections:Shouldnodesbesizedbytheiredgescount?Ifenabled,nodesizingattributewillbeignored.SizingAttribute(number):Defaultnodesize.Numericvalue>0.
EdgesOptionsMenu:
Label(string):Edgeswillbelabeledbythisattribute.Ifedgeattributeisnotfound,nolabelwillbedisplayed.AddCollectionName:Thisappendsthecollectionnametothelabel,ifitexists.ColorByCollections:Shouldedgesbecolorizedbytheircollection?Ifenabled,edgecolorandedgecolorattributewillbeignored.Color:Defaultedgecolor.ColorAttribute(string):Ifanattributeisgiven,edgeswillthenbecolorizedbytheattribute.Thissettingignoresdefaultnodecolorifset.Type:Therendereroffersmultipletypesofrendering.Theyonlydifferintheirdisplaystyle,exceptforthetype'curved'.Thecurvedtypeallowstodisplaymorethanoneedgesbetweentwonodes.
Graphs
381
ServicesTheservicessectiondisplaysallinstalledfoxxapplications.Youcancreatenewservicesorgointoadetailedviewofachoosenservice.
CreateService
Therearefourdifferentpossibilitiestocreateanewservice:
1. Createserviceviazipfile2. Createserviceviagithubrepository3. CreateserviceviaofficialArangoDBstore4. Createablankservicefromscratch
Services
383
ServiceView
Thissectionoffersseveralinformationaboutaspecificservice.
Therearefourviewcategories:
1. Info:
Displaysname,shortdescription,license,version,mode(production,development)Offersabuttontogototheservicesinterface(ifavailable)
Services
384
2. Api:
DisplayAPIasSwaggerUIDisplayAPIasRAWJSON
3. Readme:
Displaystheservicesmanual(ifavailable)4. Settings:
DownloadserviceaszipfileRunservicetests(ifavailable)Runservicescripts(ifavailable)Configuredependencies(ifavailable)Changeserviceparameters(ifavailable)Changemode(production,development)ReplacetheserviceDeletetheservice
Services
385
ManagingUsersintheWebInterfaceArangoDBusersaregloballystoredinthe_systemdatabaseandcanonlybemanangedwhileloggedontothisdatabase.ThereyoucanfindtheUserssection:
General
SelectausertobringuptheGeneraltabwiththeusername,nameandactivestatus,aswellasoptionstodeletetheuserorchangethepassword.
Users
386
Permissions
SelectauserandgotothePermissionstab.Youwillseealistofdatabasesandtheircorrespondingdatabaseaccesslevelforthatuser.
Users
387
Pleasenotethatserveraccesslevelfollowsfromtheaccesslevelonthedatabase_system.Furthermore,thedefaultdatabaseaccesslevelforthisuserappearintheartificialrowwiththedatabasename*.
Belowthistableisanotheroneforthecollectioncategoryaccesslevels.Atfirst,itshowsthelistofdatabases,too.Ifyouclickonadatabase,thelistofcollectionsinthatdatabasewillbeopenandyoucanseethedefinedcollectionaccesslevelsforeachcollectionofthatdatabase(whichcanbeallunselectedwhichmeansthatnothingisexplicitlyset).Thedefaultaccesslevelsforthisuseranddatabaseappearintheartificialrowwiththecollectionname*.
AlsoseeManagingUsersaboutaccesslevels.
Users
388
LogsThelogssectiondisplaysallavailablelogentries.Logentriesarefilterablebytheirlogleveltypes.
Functions:
Filterlogentriesbyloglevel(all,info,error,warning,debug)
Information:
LoglevelDateMessage
Logs
389
ArangoDBShellIntroductionTheArangoDBshell(arangosh)isacommand-linetoolthatcanbeusedforadministrationofArangoDB,includingrunningad-hocqueries.
ThearangoshbinaryisshippedwithArangoDB.ItoffersaJavaScriptshellenvironmentprovidingaccesstotheArangoDBserver.Arangoshcanbeinvokedlikethis:
unix>arangosh
BydefaultarangoshwilltrytoconnecttoanArangoDBserverrunningonserverlocalhostonport8529.Itwillusetheusernamerootandanemptypasswordbydefault.Additionallyitwillconnecttothedefaultdatabase(_system).Allthesedefaultscanbechangedusingthefollowingcommand-lineoptions:
--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:databaseusername--server.password:passwordtousewhenconnecting--server.authentication:whetherornottouseauthentication
Forexample,toconnecttoanArangoDBserveronIP192.168.173.13onport8530withtheuserfooandusingthedatabasetest,use:
unix>arangosh\
--server.endpointtcp://192.168.173.13:8530\
--server.usernamefoo\
--server.databasetest\
--server.authenticationtrue
arangoshwillthendisplayapasswordpromptandtrytoconnecttotheserverafterthepasswordwasentered.
Tochangethecurrentdatabaseaftertheconnectionhasbeenmade,youcanusethedb._useDatabase()commandinarangosh:
arangosh>db._createDatabase("myapp");
true
arangosh>db._useDatabase("myapp");
true
arangosh>db._useDatabase("_system");
true
arangosh>db._dropDatabase("myapp");
true
Togetalistofavailablecommands,arangoshprovidesahelp()function.Callingitwilldisplayhelpfulinformation.
arangoshalsoprovidesauto-completion.Additionalinformationonavailablecommandsandmethodsisthusprovidedbytypingthefirstfewlettersofavariableandthenpressingthetabkey.Itisrecommendtotrythiswithenteringdb.(withoutpressingreturn)andthenpressingtab.
Bytheway,arangoshprovidesthedbobjectbydefault,andthisobjectcanbeusedforswitchingtoadifferentdatabaseandmanagingcollectionsinsidethecurrentdatabase.
Foralistofavailablemethodsforthedbobject,type
arangosh>db._help();
showexecutionresultsyoucanpastemultiplelinesintoarangosh,giventhefirstlineendswithanopeningbrace:
arangosh>for(vari=0;i<10;i++){
ArangoDBShell
390
........>require("@arangodb").print("Helloworld"+i+"!\n");
........>}
showexecutionresultsToloadyourownJavaScriptcodeintothecurrentJavaScriptinterpretercontext,usetheloadcommand:
require("internal").load("/tmp/test.js")//<-Linux/MacOS
require("internal").load("c:\\tmp\\test.js")//<-Windows
Exitingarangoshcanbedoneusingthekeycombination<CTRL>+Dorbytypingquit<CR>
Escaping
InAQL,escapingisdonetraditionallywiththebackslashcharacter:\.Asseenabove,thisleadstodoublebackslasheswhenspecifyingWindowspaths.Arangoshrequiresanotherlevelofescaping,alsowiththebackslashcharacter.ItaddsuptofourbackslashesthatneedtobewritteninArangoshforasingleliteralbackslash(c:\tmp\test.js):
db._query('RETURN"c:\\\\tmp\\\\test.js"')
Youcanusebindvariablestomitigatethis:
varsomepath="c:\\tmp\\test.js"
db._query(aql`RETURN${somepath}`)
ArangoDBShell
391
ArangoDBShellOutputTheArangoDBshellwillprinttheoutputofthelastevaluatedexpressionbydefault:
arangosh>42*23
966
Inordertopreventprintingtheresultofthelastevaluatedexpression,theexpressionresultcanbecapturedinavariable,e.g.
arangosh>varcalculationResult=42*23
ThereisalsotheprintfunctiontoexplicitlyprintoutvaluesintheArangoDBshell:
arangosh>print({a:"123",b:[1,2,3],c:"test"});
showexecutionresultsBydefault,theArangoDBshellusesaprettyprinterwhenJSONdocumentsareprinted.Thisensuresdocumentsareprintedinahuman-readableway:
arangosh>db._create("five")
arangosh>for(i=0;i<5;i++)db.five.save({value:i})
arangosh>db.five.toArray()
showexecutionresultsWhilethepretty-printerproducesnicelookingresults,itwillneedalotofscreenspaceforeachdocument.Sometimesamoredenseoutputmightbebetter.Inthiscase,theprettyprintercanbeturnedoffusingthecommandstop_pretty_print().
Toturnonprettyprintingagain,usethestart_pretty_print()command.
ShellOutput
392
ArangoDBShellConfigurationarangoshwilllookforauser-definedstartupscriptnamed.arangosh.rcintheuser'shomedirectoryonstartup.Thehomedirectorywilllikelybe/home/<username>/onUnix/Linux,andisdeterminedonWindowsbypeekingintotheenvironmentvariables%HOMEDRIVE%and%HOMEPATH%.
Ifthefile.arangosh.rcispresentinthehomedirectory,arangoshwillexecutethecontentsofthisfileinsidetheglobalscope.
Youcanusethistodefineyourownextravariablesandfunctionsthatyouneedoften.Forexample,youcouldputthefollowingintothe.arangosh.rcfileinyourhomedirectory:
//"var"keywordavoidedintentionally...
//otherwise"timed"wouldnotsurvivethescopeofthisscript
global.timed=function(cb){
console.time("callback");
cb();
console.timeEnd("callback");
};
Thiswillmakeafunctionnamedtimedavailableinarangoshintheglobalscope.
Youcannowstartarangoshandinvokethefunctionlikethis:
timed(function(){
for(vari=0;i<1000;++i){
db.test.save({value:i});
}
});
Pleasekeepinmindthat,ifpresent,the.arangosh.rcfileneedstocontainvalidJavaScriptcode.Ifyouwantanyvariablesintheglobalscopetosurviveyouneedtoomitthevarkeywordforthem.Otherwisethevariableswillonlybevisibleinsidethescriptitself,butnotoutside.
Configuration
393
DetailsabouttheArangoDBShellAftertheserverhasbeenstarted,youcanusetheArangoDBshell(arangosh)toadministratetheserver.Withoutanyarguments,theArangoDBshellwilltrytocontacttheserveronport8529onthelocalhost.FormoreinformationseetheArangoDBShelldocumentation.Youmightneedtosetadditionaloptions(endpoint,usernameandpassword)whenconnecting:
unix>./arangosh--server.endpointtcp://127.0.0.1:8529--server.usernameroot
TheshellwillprintitsownversionnumberandifsuccessfullyconnectedtoaservertheversionnumberoftheArangoDBserver.
Command-LineOptionsUse--helptogetalistofcommand-lineoptions:
unix>./arangosh--help
STANDARDoptions:
--audit-log<string>auditlogfiletosavecommandsandresultsto
--configuration<string>readconfigurationfile
--helphelpmessage
--max-upload-size<uint64>maximumsizeofimportchunks(inbytes)(default:500000)
--no-auto-completedisableautocompletion
--no-colorsdeactivatecolorsupport
--pager<string>outputpager(default:"less-X-R-F-L")
--pretty-printprettyprintvalues
--quietnobanner
--temp.path<string>pathfortemporaryfiles(default:"/tmp/arangodb")
--use-pagerusepager
JAVASCRIPToptions:
--javascript.check<string>syntaxcheckcodeJavaScriptcodefromfile
--javascript.execute<string>executeJavaScriptcodefromfile
--javascript.execute-string<string>executeJavaScriptcodefromstring
--javascript.startup-directory<string>startuppathscontainingtheJavaScriptfiles
--javascript.unit-tests<string>donotstartasshell,rununittestsinstead
--jslint<string>donotstartasshell,runjslintinstead
LOGGINGoptions:
--log.level<string>loglevel(default:"info")
CLIENToptions:
--server.connect-timeout<double>connecttimeoutinseconds(default:3)
--server.authentication<bool>whetherornottouseauthentication(default:true)
--server.endpoint<string>endpointtoconnectto,use'none'tostartwithoutaserver(default:"tcp://127.0
.0.1:8529")
--server.password<string>passwordtousewhenconnecting(leaveemptyforprompt)
--server.request-timeout<double>requesttimeoutinseconds(default:300)
--server.username<string>usernametousewhenconnecting(default:"root")
DatabaseWrappersThedbobjectisavailableinarangoshaswellasonarangodi.e.ifyou'reusingFoxx.Whileitsinterfaceispersistantbetweenthearangoshandthearangodimplementations,itsunderpinningisnot.ThearangodimplementationareJavaScriptwrappersaroundArangoDB'snativeC++implementation,whereasthearangoshimplementationwrapsHTTPaccessestoArangoDB'sRESTfullAPI.
Sowhilethiscodemayproducesimilarresultswhenexecutedinarangoshandarangod,thecpuusageandtimerequiredwillbereallydifferent:
for(i=0;i<100000;i++){
db.test.save({name:{first:"Jan"},count:i});
}
Sincethearangoshversionwillbedoingaround100kHTTPrequests,andthearangodversionwilldirectlywritetothedatabase.
Details
394
Using arangoshviaunixshebangmechanisms
Inunixoperatingsystemsyoucanstartscriptsbyspecifyingtheinterpreterinthefirstlineofthescript.Thisiscommonlycalledshebangorhashbang.Youcanalsodothatwitharangosh,i.e.create~/test.js:
#!/usr/bin/arangosh--javascript.execute
require("internal").print("helloworld")
db._query("FORxINtestRETURNx").toArray()
Notethatthefirstlinehastoendwithablankinordertomakeitwork.MarkitexecutabletotheOS:
#>chmoda+x~/test.js
andfinalytryitout:
#>~/test.js
Details
395
ArangoimpThismanualdescribestheArangoDBimporterarangoimp,whichcanbeusedforbulkimports.
ThemostconvenientmethodtoimportalotofdataintoArangoDBistousethearangoimpcommand-linetool.Itallowsyoutoimportdatarecordsfromafileintoanexistingdatabasecollection.
Itispossibletoimportdocumentkeyswiththedocumentsusingthe_keyattribute.Whenimportingintoanedgecollection,itismandatorythatallimporteddocumentshavethe_fromand_toattributes,andthattheycontainvalidreferences.
Let'sassumeforthefollowingexamplesyouwanttoimportuserdataintoanexistingcollectionnamed"users"ontheserver.
ImportingDataintoanArangoDBDatabase
ImportingJSON-encodedData
Let'sfurtherassumetheimportathandisencodedinJSON.We'llbeusingtheseexampleuserrecordstoimport:
{"name":{"first":"John","last":"Connor"},"active":true,"age":25,"likes":["swimming"]}
{"name":{"first":"Jim","last":"O'Brady"},"age":19,"likes":["hiking","singing"]}
{"name":{"first":"Lisa","last":"Jones"},"dob":"1981-04-09","likes":["running"]}
Toimporttheserecords,allyouneedtodoistoputthemintoafile(withonelineforeachrecordtoimport)andrunthefollowingcommand:
>arangoimp--file"data.json"--typejsonl--collection"users"
Thiswilltransferthedatatotheserver,importtherecords,andprintastatussummary.Toshowtheintermediateprogressduringtheimportprocess,theoption--progresscanbeadded.Thisoptionwillshowthepercentageoftheinputfilethathasbeensenttotheserver.Thiswillonlybeusefulforbigimportfiles.
>arangoimp--file"data.json"--typejson--collectionusers--progresstrue
Itisalsopossibletousetheoutputofanothercommandasaninputforarangoimp.Forexample,thefollowingshellcommandcanbeusedtopipedatafromthecatprocesstoarangoimp:
>catdata.json|arangoimp--file---typejson--collectionusers
Notethatyouhavetouse--file-ifyouwanttouseanothercommandasinputforarangoimp.Noprogresscanbereportedforsuchimportsasthesizeoftheinputwillbeunknowntoarangoimp.
Bydefault,theendpointtcp://127.0.0.1:8529willbeused.Ifyouwanttospecifyadifferentendpoint,youcanusethe--server.endpointoption.Youprobablywanttospecifyadatabaseuserandpasswordaswell.Youcandosobyusingtheoptions--server.usernameand--server.password.Ifyoudonotspecifyapassword,youwillbepromptedforone.
>arangoimp--server.endpointtcp://127.0.0.1:8529--server.usernameroot--file"data.json"--typejson--collection"users"
Notethatthecollection(usersinthiscase)mustalreadyexistortheimportwillfail.Ifyouwanttocreateanewcollectionwiththeimportdata,youneedtospecifythe--create-collectionoption.Notethatbydefaultitwillcreateadocumentcollectionandnoedecollection.
>arangoimp--file"data.json"--typejson--collection"users"--create-collectiontrue
Tocreateanedgecollectioninstead,usethe--create-collection-typeoptionandsetittoedge:
>arangoimp--file"data.json"--collection"myedges"--create-collectiontrue--create-collection-typeedge
Arangoimp
396
Whenimportingdataintoanexistingcollectionitisoftenconvenienttofirstremovealldatafromthecollectionandthenstarttheimport.Thiscanbeachievedbypassingthe--overwriteparametertoarangoimp.Ifitissettotrue,anyexistingdatainthecollectionwillberemovedpriortotheimport.Notethatanyexistingindexdefinitionsforthecollectionwillbepreservedevenif--overwriteissettotrue.
>arangoimp--file"data.json"--typejson--collection"users"--overwritetrue
AstheimportfilealreadycontainsthedatainJSONformat,attributenamesanddatatypesarefullypreserved.Ascanbeseenintheexampledata,thereisnoneedforalldatarecordstohavethesameattributenamesortypes.Recordscanbeinhomogeneous.
Pleasenotethatbydefault,arangoimpwillimportdataintothespecifiedcollectioninthedefaultdatabase(_system).Tospecifyadifferentdatabase,usethe--server.databaseoptionwheninvokingarangoimp.
Thetoolalsosupportsparallelimports,withmultiplethreads.Usingmultiplethreadsmayprovideaspeedup,especiallywhenusingtheRocksDBstorageengine.Tospecifythenumberofparallelthreadsusethe--threadsoption:
>arangoimp--threads4--file"data.json"--typejson--collection"users"
Notethatusingmultiplethreadsmayleadtoanon-sequentialimportoftheinputdata.Datathatappearslaterintheinputfilemaybeimportedearlierthandatathatappearsearlierintheinputfile.Thisisnormallynotaproblembutmaycauseissueswhenwhentherearedatadependenciesorduplicatesintheimportdata.Inthiscase,thenumberofthreadsshouldbesetto1.
JSONinputfileformats
Note:arangoimpsupportstwoformatswhenimportingJSONdatafromafile.Thefirstformatthatwealsousedaboveiscommonlyknownasjsonl).However,incontrasttotheJSONLspecificationitrequirestheinputfiletocontainonecompleteJSONdocumentineachline,e.g.
{"_key":"one","value":1}
{"_key":"two","value":2}
{"_key":"foo","value":"bar"}
...
SoonecouldarguethatthisisonlyasubsetofJSONL.
Theaboveformatcanbeimportedsequentiallybyarangoimp.Itwillreaddatafromtheinputfileinchunksandsenditinbatchestotheserver.Eachbatchwillbeaboutasbigasspecifiedinthecommand-lineparameter--batch-size.
AnalternativeistoputonebigJSONdocumentintotheinputfilelikethis:
[
{"_key":"one","value":1},
{"_key":"two","value":2},
{"_key":"foo","value":"bar"},
...
]
Thisformatallowslinebreakswithintheinputfileasrequired.Thedownsideisthatthewholeinputfilewillneedtobereadbyarangoimpbeforeitcansendthefirstbatch.Thismightbeaproblemiftheinputfileisbig.Bydefault,arangoimpwillallowimportingsuchfilesuptoasizeofabout16MB.
Ifyouwanttoallowyourarangoimpinstancetousemorememory,youmaywanttoincreasethemaximumfilesizebyspecifyingthecommand-lineoption--batch-size.Forexample,tosetthebatchsizeto32MB,usethefollowingcommand:
>arangoimp--file"data.json"--typejson--collection"users"--batch-size33554432
Pleasealsonotethatyoumayneedtoincreasethevalueof--batch-sizeifasingledocumentinsidetheinputfileisbiggerthanthevalueof--batch-size.
ImportingCSVData
Arangoimp
397
arangoimpalsooffersthepossibilitytoimportdatafromCSVfiles.ThiscomeshandywhenthedataathandisinCSVformatalreadyandyoudon'twanttospendtimeconvertingthemtoJSONfortheimport.
ToimportdatafromaCSVfile,makesureyourfilecontainstheattributenamesinthefirstrow.Allthefollowinglinesinthefilewillbeinterpretedasdatarecordsandwillbeimported.
TheCSVimportrequiresthedatatohaveahomogeneousstructure.Allrecordsmusthaveexactlythesameamountofcolumnsasthereareheaders.Bydefault,lineswithadifferentnumberofvalueswillnotbeimportedandtherewillbewarningsforthem.Tostillimportlineswithlessvaluesthanintheheader,thereisthe--ignore-missingoption.Ifsettotrue,linesthathaveadifferentamountoffieldswillbeimported.Inthiscaseonlythoseattributeswillbepopulatedforwhichtherearevalues.Attributesforwhichtherearenovaluespresentwillsilentlybediscarded.
Example:
"first","last","age","active","dob"
"John","Connor",25,true
"Jim","O'Brady"
With--ignore-missingthiswillproducethefollowingdocuments:
{"first":"John","last":"Connor","active":true,"age":25}
{"first":"Jim","last":"O'Brady"}
Thecellvaluescanhavedifferentdatatypesthough.Ifacelldoesnothaveanyvalue,itcanbeleftemptyinthefile.Thesevalueswillnotbeimportedsotheattributeswillnot"bethere"indocumentcreated.Valuesenclosedinquoteswillbeimportedasstrings,sotoimportnumericvalues,booleanvaluesorthenullvalue,don'tenclosethevalueinquotesinyourfile.
We'llbeusingthefollowingimportfortheCSVimport:
"first","last","age","active","dob"
"John","Connor",25,true,
"Jim","O'Brady",19,,
"Lisa","Jones",,,"1981-04-09"
Hans,dosSantos,0123,,
Wayne,Brewer,,false,
Thecommandlinetoexecutetheimportis:
>arangoimp--file"data.csv"--typecsv--collection"users"
Theabovedatawillbeimportedinto5documentswhichwilllookasfollows:
{"first":"John","last":"Connor","active":true,"age":25}
{"first":"Jim","last":"O'Brady","age":19}
{"first":"Lisa","last":"Jones","dob":"1981-04-09"}
{"first":"Hans","last":"dosSantos","age":123}
{"first":"Wayne","last":"Brewer","active":false}
Ascanbeseen,valuesleftcompletelyemptyintheinputfilewillbetreatedasabsent.Numericvaluesnotenclosedinquoteswillbetreatedasnumbers.Notethatleadingzerosinnumericvalueswillberemoved.Toimportnumberswithleadingzeros,pleaseusestrings.Theliteralstrueandfalsewillbetreatedasbooleansiftheyarenotenclosedinquotes.Othervaluesnotenclosedinquoteswillbetreatedasstrings.Anyvaluesenclosedinquoteswillbetreatedasstrings,too.
Stringvaluescontainingthequotecharacterortheseparatormustbeenclosedwithquotecharacters.Withinastring,thequotecharacteritselfmustbeescapedwithanotherquotecharacter(orwithabackslashifthe--backslash-escapeoptionisused).
Notethatthequoteandseparatorcharacterscanbeadjustedviathe--quoteand--separatorargumentswheninvokingarangoimp.Thequotecharacterdefaultstothedoublequote(").Tousealiteralquoteinastring,youcanusetwoquotecharacters.Tousebackslashforescapingquotecharacters,pleasesettheoption--backslash-escapetotrue.
TheimportersupportsWindows(CRLF)andUnix(LF)linebreaks.Linebreaksmightalsooccurinsidevaluesthatareenclosedwiththequotecharacter.
Arangoimp
398
Here'sanexampleforusingliteralquotesandnewlinesinsidevalues:
"name","password"
"Foo","r4ndom""123!"
"Bar","wow!
thisisa
multinepassword!"
"Bartholomew""Bart""Simpson","Milhouse"
Extrawhitespaceattheendofeachlinewillbeignored.Whitespaceatthestartoflinesorbetweenfieldvalueswillnotbeignored,sopleasemakesurethatthereisnoextrawhitespaceinfrontofvaluesorbetweenthem.
ImportingTSVData
Youmayalsoimporttab-separatedvalues(TSV)fromafile.Thisformatisverysimple:everylineinthefilerepresentsadatarecord.Thereisnoquotingorescaping.Thatalsomeansthattheseparatorcharacter(whichdefaultstothetabstopsymbol)mustnotbeusedanywhereintheactualdata.
AswithCSV,thefirstlineintheTSVfilemustcontaintheattributenames,andalllinesmusthaveanidenticalnumberofvalues.
Ifadifferentseparatorcharacterorstringshouldbeused,itcanbespecifiedwiththe--separatorargument.
AnexamplecommandlinetoexecutetheTSVimportis:
>arangoimp--file"data.tsv"--typetsv--collection"users"
AttributeNameTranslation
FortheCSVandTSVinputformats,attributenamescanbetranslatedautomatically.ThisisusefulincasetheimportfilehasdifferentattributenamesthanthosethatshouldbeusedinArangoDB.
Acommonusecaseistorenamean"id"columnfromtheinputfileinto"_key"asitisexpectedbyArangoDB.Todothis,specifythefollowingtranslationwheninvokingarangoimp:
>arangoimp--file"data.csv"--typecsv--translate"id=_key"
Othercommoncasesaretorenamecolumnsintheinputfileto_fromand_to:
>arangoimp--file"data.csv"--typecsv--translate"from=_from"--translate"to=_to"
Thetranslateoptioncanbespecifiedmultipletypes.Thesourceattributenameandthetargetattributemustbeseparatedwitha=.
IgnoringAttributes
FortheCSVandTSVinputformats,certainattributenamescanbeignoredonimports.InanArangoDBclustertherearecaseswherethiscancomeinhandy,whenyourdocumentsalreadycontaina_keyattributeandyourcollectionhasashardingattributeotherthan_key:Intheclusterthisconfigurationisnotsupported,becauseArangoDBneedstoguaranteetheuniquenessofthe_keyattributeinallshardsofthecollection.
>arangoimp--file"data.csv"--typecsv--remove-attribute"_key"
Thesamethingwouldapplyifyourdatacontainsan_idattribute:
>arangoimp--file"data.csv"--typecsv--remove-attribute"_id"
ImportingintoanEdgeCollection
Arangoimp
399
arangoimpcanalsobeusedtoimportdataintoanexistingedgecollection.Theimportdatamust,foreachedgetoimport,containatleastthe_fromand_toattributes.Theseindicatewhichothertwodocumentstheedgeshouldconnect.Itisnecessarythattheseattributesaresetforallrecords,andpointtovaliddocumentidsinexistingcollections.
Examples
{"_from":"users/1234","_to":"users/4321","desc":"1234isconnectedto4321"}
Note:Theedgecollectionmustalreadyexistwhentheimportisstarted.Usingthe--create-collectionflagwillnotworkbecausearangoimpwillalwaystrytocreatearegulardocumentcollectionifthetargetcollectiondoesnotexist.
Updatingexistingdocuments
Bydefault,arangoimpwilltrytoinsertalldocumentsfromtheimportfileintothespecifiedcollection.Incasetheimportfilecontainsdocumentsthatarealreadypresentinthetargetcollection(matchingisdoneviathe_keyattributes),thenadefaultarangoimprunwillnotimportthesedocumentsandcomplainaboutuniquekeyconstraintviolations.
However,arangoimpcanbeusedtoupdateorreplaceexistingdocumentsincasetheyalreadyexistinthetargetcollection.Itprovidesthecommand-lineoption--on-duplicatetocontrolthebehaviorincaseadocumentisalreadypresentinthedatabase.
Thedefaultvalueof--on-duplicateiserror.Thismeansthatwhentheimportfilecontainsadocumentthatispresentinthetargetcollectionalready,thentryingtore-insertadocumentwiththesame_keyvalueisconsideredanerror,andthedocumentinthedatabasewillnotbemodified.
Otherpossiblevaluesfor--on-duplicateare:
update:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbeupdatedbyarangoimp.updatewillperformapartialupdateoftheexistingdocument,modifyingonlytheattributesthatarepresentintheimportfileandleavingallotherattributesuntouched.
Thevaluesofsystemattributes_id,_key,_rev,_fromand_tocannotbeupdatedorreplacedinexistingdocuments.
replace:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbereplacebyarangoimp.replacewillreplacetheexistingdocumententirely,resultinginadocumentwithonlytheattributesspecifiedintheimportfile.
Thevaluesofsystemattributes_id,_key,_rev,_fromand_tocannotbeupdatedorreplacedinexistingdocuments.
ignore:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbeignoredandnotmodifiedinthetargetcollection.
When--on-duplicateissettoeitherupdateorreplace,arangoimpwillreturnthenumberofdocumentsupdated/replacedintheupdatedreturnvalue.Whensettoanothervalue,thevalueofupdatedwillalwaysbezero.When--on-duplicateissettoignore,arangoimpwillreturnthenumberofignoreddocumentsintheignoredreturnvalue.Whensettoanothervalue,ignoredwillalwaysbezero.
Itispossibletoperformacombinationofinsertsandupdates/replaceswithasinglearangoimprun.When--on-duplicateissettoupdateorreplace,alldocumentspresentintheimportfilewillbeinsertedintothetargetcollectionprovidedtheyarevalidanddonotalreadyexistwiththespecified_key.Documentsthatarealreadypresentinthetargetcollection(identifiedby_keyattribute)willinsteadbeupdated/replaced.
Arangoimpresultoutput
Anarangoimpimportrunwillprintoutthefinalresultsonthecommandline.Itwillshowthe
numberofdocumentscreated(created)numberofdocumentsupdated/replaced(updated/replaced,onlynon-zeroif--on-duplicatewassettoupdateorreplace,seebelow)numberofwarningsorerrorsthatoccurredontheserverside(warnings/errors)numberofignoreddocuments(onlynon-zeroif--on-duplicatewassettoignore).
Example
created:2
warnings/errors:0
updated/replaced:0
ignored:0
Arangoimp
400
ForCSVandTSVimports,thetotalnumberofinputfilelinesreadwillalsobeprinted(linesread).
arangoimpwillalsoprintoutdetailsaboutwarningsanderrorsthathappenedontheserver-side(ifany).
AttributeNamingandSpecialAttributes
AttributeswhosenamesstartwithanunderscorearetreatedinaspecialwaybyArangoDB:
theoptional_keyattributecontainsthedocument'skey.Ifspecified,thevaluemustbeformallyvalid(e.g.mustbeastringandconformtothenamingconventions).Additionally,thekeyvaluemustbeuniquewithinthecollectiontheimportisrunfor._from:whenimportingintoanedgecollection,thisattributecontainstheidofoneofthedocumentsconnectedbytheedge.Thevalueof_frommustbeasyntacticallyvaliddocumentidandthereferredcollectionmustexist._to:whenimportingintoanedgecollection,thisattributecontainstheidoftheotherdocumentconnectedbytheedge.Thevalueof_tomustbeasyntacticallyvaliddocumentidandthereferredcollectionmustexist._rev:thisattributecontainstherevisionnumberofadocument.However,therevisionnumbersaremanagedbyArangoDBandcannotbespecifiedonimport.Thusanyvalueinthisattributeisignoredonimport.
Ifyouimportvaluesinto_key,youshouldmakesuretheyarevalidandunique.
Whenimportingdataintoanedgecollection,youshouldmakesurethatallimportdocumentscan_fromand_toandthattheirvaluespointtoexistingdocuments.
Toavoidspecifyingcompletedocumentids(consistingofcollectionnamesanddocumentkeys)for_fromand_tovalues,therearetheoptions--from-collection-prefixand--to-collection-prefix.Ifspecified,thesevalueswillbeautomaticallyprependedtoeachvaluein_from(or_toresp.).Thisallowsspecifyingonlydocumentkeysinside_fromand/or_to.
Example
>arangoimp--from-collection-prefixusers--to-collection-prefixproducts...
Importingthefollowingdocumentwillthencreateanedgebetweenusers/1234andproducts/4321:
{"_from":"1234","_to":"4321","desc":"users/1234isconnectedtoproducts/4321"}
Arangoimp
401
DumpingDatafromanArangoDBdatabaseTodumpdatafromanArangoDBserverinstance,youwillneedtoinvokearangodump.Dumpscanbere-importedwitharangorestore.arangodumpcanbeinvokedbyexecutingthefollowingcommand:
unix>arangodump--output-directory"dump"
ThiswillconnecttoanArangoDBserveranddumpallnon-systemcollectionsfromthedefaultdatabase(_system)intoanoutputdirectorynameddump.Invokingarangodumpwillfailiftheoutputdirectoryalreadyexists.Thisisanintentionalsecuritymeasuretopreventyoufromaccidentallyoverwritingalreadydumpeddata.Ifyouarepositivethatyouwanttooverwritedataintheoutputdirectory,youcanusetheparameter--overwritetruetoconfirmthis:
unix>arangodump--output-directory"dump"--overwritetrue
arangodumpwillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:
--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:username--server.password:passwordtouse(omitthisandyou'llbepromptedforthepassword)--server.authentication:whetherornottouseauthentication
Here'sanexampleofdumpingdatafromanon-standardendpoint,usingadedicateddatabasename:
unix>arangodump--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--output-directory
"dump"
Whenfinished,arangodumpwillprintoutasummarylinewithsomeaggregatestatisticsaboutwhatitdid,e.g.:
Processed43collection(s),wrote408173500byte(s)intodatafiles,sent88batch(es)
Bydefault,arangodumpwilldumpbothstructuralinformationanddocumentsfromallnon-systemcollections.Toadjustthis,therearethefollowingcommand-linearguments:
--dump-data:settotruetoincludedocumentsinthedump.Settofalsetoexcludedocuments.Thedefaultvalueistrue.--include-system-collections:whetherornottoincludesystemcollectionsinthedump.Thedefaultvalueisfalse.
Forexample,toonlydumpstructuralinformationofallcollections(includingsystemcollections),use:
unix>arangodump--dump-datafalse--include-system-collectionstrue--output-directory"dump"
Torestrictthedumptojustspecificcollections,thereisisthe--collectionoption.Itcanbespecifiedmultipletimesifrequired:
unix>arangodump--collectionmyusers--collectionmyvalues--output-directory"dump"
Structuralinformationforacollectionwillbesavedinfileswithnamepattern.structure.json.EachstructurefilewillcontainsaJSONobjectwiththeseattributes:
parameters:containsthecollectionpropertiesindexes:containsthecollectionindexes
Documentdataforacollectionwillbesavedinfileswithnamepattern.data.json.Eachlineinadatafileisadocumentinsertion/updateordeletionmarker,alongsidewithsomemetadata.
StartingwithVersion2.1ofArangoDB,thearangodumptoolalsosupportssharding.Simplypointittooneofthecoordinatorsanditwillbehaveexactlyasdescribedabove,workingonshardedcollectionsinthecluster.
Arangodump
402
However,asopposedtothesingleinstancesituation,thisoperationdoesnotguaranteetodumpaconsistentsnapshotifwriteoperationshappenduringthedumpoperation.Itisthereforerecommendednottoperformanydata-modifcationoperationsontheclusterwhilstarangodumpisrunning.
Asabove,theoutputwillbeonestructuredescriptionfileandonedatafilepershardedcollection.Notethatthedatainthedatafileissortedfirstbyshardsandwithineachshardbyascendingtimestamp.Thestructuralinformationofthecollectioncontainsthenumberofshardsandtheshardkeys.
NotethattheversionofthearangodumpclienttoolneedstomatchtheversionoftheArangoDBserveritconnectsto.
Advancedclusteroptions
Startingwithversion3.1.17,collectionsmaybecreatedwithsharddistributionidenticaltoanexistingprototypicalcollection;i.e.shardsaredistributedintheverysamepatternasintheprototypecollection.Suchcollectionscannotbedumpedwithoutthereferencecollectionorarangodumpwithyieldanerror.
unix>arangodump--collectionclonedCollection--output-directory"dump"
ERRORCollectionclonedCollection'ssharddistributionisbasedonathatofcollectionprototypeCollection,whichisnotdumpe
dalong.Youmaydumpthecollectionregardlessofthemissingprototypecollectionbyusingthe--ignore-distribute-shards-lik
e-errorsparameter.
Therearetwowaystoapproachthatproblem:Solveit,i.e.dumptheprototypecollectionalong:
unix>arangodump--collectionclonedCollection--collectionprototypeCollection--output-directory"dump"
Processed2collection(s),wrote81920byte(s)intodatafiles,sent1batch(es)
Oroverridethatbehaviourtobeabletodumpthecollectionindividually.
unix>arangodump--collectionBclonedCollection--output-directory"dump"--ignore-distribute-shards-like-errors
Processed1collection(s),wrote34217byte(s)intodatafiles,sent1batch(es)
Nothatinconsequence,restoringsuchacollectionwithoutitsprototypeisaffected.arangorestore
Encryption
IntheArangoDBEnterpriseEditiontherearetheadditionalparameters:
Encryptionkeystoredinfile
--encryption.keyfilepath-of-keyfile
Thefilepath-to-keyfilemustcontaintheencryptionkey.Thisfilemustbesecured,sothatonlyarangodcanaccessit.Youshouldalsoensurethatincasesome-onestealsthehardware,hewillnotbeabletoreadthefile.Forexample,byencryption/mytmpfsorcreatingain-memoryfile-systemunder/mytmpfs.
Encryptionkeygeneratedbyaprogram
--encryption.key-generatorpath-to-my-generator
Theprogrampath-to-my-generatormustoutputtheencryptiononstandardoutputandexit.
Creatingkeys
Theencryptionkeyfilemustcontain32bytesofrandomdata.
Youcancreateitwithacommandlinethis.
ddif=/dev/randombs=1count=32of=yourSecretKeyFile
Arangodump
403
Forsecurity,itisbesttocreatethesekeysoffline(awayfromyourdatabaseservers)anddirectlystoretheminyousecretmanagementtool.
Arangodump
404
ArangorestoreToreloaddatafromadumppreviouslycreatedwitharangodump,ArangoDBprovidesthearangorestoretool.
Pleasenotethatarangorestoremustnotbeusedtocreateseveralsimilardatabaseinstancesinoneinstallation.
Thismeansifyouhaveanarangodumpoutputofdatabasea,andyoucreateaseconddatabasebonthesameinstanceofArangoDB,andrestorethedumpofaintob-dataintegritycannotbeguaranteed.
ReloadingDataintoanArangoDBdatabase
Invokingarangorestore
arangorestorecanbeinvokedfromthecommand-lineasfollows:
unix>arangorestore--input-directory"dump"
ThiswillconnecttoanArangoDBserverandreloadstructuralinformationanddocumentsfoundintheinputdirectorydump.Pleasenotethattheinputdirectorymusthavebeencreatedbyrunningarangodumpbefore.
arangorestorewillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:
--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:username--server.password:passwordtouse(omitthisandyou'llbepromptedforthepassword)--server.authentication:whetherornottouseauthentication
Sinceversion2.6arangorestoreprovidestheoption--create-database.Settingthisoptiontotruewillcreatethetargetdatabaseifitdoesnotexist.Whencreatingthetargetdatabase,theusernameandpasswordspassedtoarangorestore(inoptions--server.usernameand--server.password)willbeusedtocreateaninitialuserforthenewdatabase.
Theoption--force-same-databaseallowsrestrictingarangorestoreoperationstoadatabasewiththesamenameasinthesourcedump's"dump.json"file.Itcanthusbeusedtopreventrestoringdataintoa"wrong"databasebyaccident.
Forexample,ifadumpwastakenfromdatabasea,andtherestoreisattemptedintodatabaseb,thenwiththe--force-same-databaseoptionsettotrue,arangorestorewillabortinstantly.
The--force-same-databaseoptionissettofalsebydefaulttoensurebackwards-compatibility.
Here'sanexampleofreloadingdatatoanon-standardendpoint,usingadedicateddatabasename:
unix>arangorestore--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--input-directo
ry"dump"
Tocreatethetargetdatabasewherestoring,useacommandlikethis:
unix>arangorestore--server.usernamebackup--server.databasenewdb--create-databasetrue--input-directory"dump"
arangorestorewillprintoutitsprogresswhilerunning,andwillendwithalineshowingsomeaggregatestatistics:
Processed2collection(s),read2256byte(s)fromdatafiles,sent2batch(es)
Bydefault,arangorestorewillre-createallnon-systemcollectionsfoundintheinputdirectoryandloaddataintothem.Ifthetargetdatabasealreadycontainscollectionswhicharealsopresentintheinputdirectory,theexistingcollectionsinthedatabasewillbedroppedandre-createdwiththedatafoundintheinputdirectory.
Arangorestore
405
Thefollowingparametersareavailabletoadjustthisbehavior:
--create-collection:settotruetocreatecollectionsinthetargetdatabase.Ifthetargetdatabasealreadycontainsacollectionwiththesamename,itwillbedroppedfirstandthenre-createdwiththepropertiesfoundintheinputdirectory.Settofalsetokeepexistingcollectionsinthetargetdatabase.Ifsettofalseandarangorestoreencountersacollectionthatispresentinboththetargetdatabaseandtheinputdirectory,itwillabort.Thedefaultvalueistrue.--import-data:settotruetoloaddocumentdataintothecollectionsinthetargetdatabase.Settofalsetonotloadanydocumentdata.Thedefaultvalueistrue.--include-system-collections:whetherornottoincludesystemcollectionswhenre-creatingcollectionsorreloadingdata.Thedefaultvalueisfalse.
Forexample,to(re-)createallnon-systemcollectionsandloaddocumentdataintothem,use:
unix>arangorestore--create-collectiontrue--import-datatrue--input-directory"dump"
Thiswilldroppotentiallyexistingcollectionsinthetargetdatabasethatarealsopresentintheinputdirectory.
Toincludesystemcollectionstoo,use--include-system-collectionstrue:
unix>arangorestore--create-collectiontrue--import-datatrue--include-system-collectionstrue--input-directory"dump"
To(re-)createallnon-systemcollectionswithoutloadingdocumentdata,use:
unix>arangorestore--create-collectiontrue--import-datafalse--input-directory"dump"
Thiswillalsodropexistingcollectionsinthetargetdatabasethatarealsopresentintheinputdirectory.
Tojustloaddocumentdataintoallnon-systemcollections,use:
unix>arangorestore--create-collectionfalse--import-datatrue--input-directory"dump"
Torestrictreloadingtojustspecificcollections,thereisisthe--collectionoption.Itcanbespecifiedmultipletimesifrequired:
unix>arangorestore--collectionmyusers--collectionmyvalues--input-directory"dump"
Collectionswillbeprocessedbyinalphabeticalorderbyarangorestore,withalldocumentcollectionsbeingprocessedbeforealledgecollections.Thisistoensurethatreloadingdataintoedgecollectionswillhavethedocumentcollectionslinkedinedges(_fromand_toattributes)loaded.
Encryption
Seearangodumpfordetails.
RestoringRevisionIdsandCollectionIds
arangorestorewillreloaddocumentandedgesdatawiththeexactsame_key,_fromand_tovaluesfoundintheinputdirectory.However,whenloadingdocumentdata,itwillassignitsownvaluesforthe_revattributeofthereloadeddocuments.Thoughthisdifferenceisintentional(normally,everyservershouldcreateitsown_revvalues)theremightbesituationswhenitisrequiredtore-usetheexactsame_revvaluesforthereloadeddata.Thiscanbeachievedbysettingthe--recycle-idsparametertotrue:
unix>arangorestore--collectionmyusers--collectionmyvalues--input-directory"dump"
Notethatsetting--recycle-idstotruewillalsocausecollectionstobe(re-)createdinthetargetdatabasewiththeexactsamecollectionidasintheinputdirectory.Anypotentiallyexistingcollectioninthetargetdatabasewiththesamecollectionidwillthenbedropped.
ReloadingDataintoadifferentCollection
Arangorestore
406
Withsomecreativityyoucanusearangodumpandarangorestoretotransferdatafromonecollectionintoanother(eitheronthesameserverornot).Forexample,tocopydatafromacollectionmyvaluesindatabasemydbintoacollectionmycopyvaluesindatabasemycopy,youcanstartwiththefollowingcommand:
unix>arangodump--collectionmyvalues--server.databasemydb--output-directory"dump"
Thiswillcreatetwofiles,myvalues.structure.jsonandmyvalues.data.json,intheoutputdirectory.Toloaddatafromthedatafileintoanexistingcollectionmycopyvaluesindatabasemycopy,renamethefilestomycopyvalues.structure.jsonandmycopyvalues.data.json.Afterthat,runthefollowingcommand:
unix>arangorestore--collectionmycopyvalues--server.databasemycopy--input-directory"dump"
Usingarangorestorewithsharding
AsofVersion2.1thearangorestoretoolsupportssharding.Simplypointittooneofthecoordinatorsinyourclusteranditwillworkasusualbutonshardedcollectionsinthecluster.
Ifarangorestoreisaskedtodropandre-createacollection,itwillusethesamenumberofshardsandthesameshardkeysaswhenthecollectionwasdumped.Thedistributionoftheshardstotheserverswillalsobethesameasatthetimeofthedump.ThismeansinparticularthatDBserverswiththesameIDsasbeforemustbepresentintheclusterattimeoftherestore.
Ifacollectionwasdumpedfromasingleinstance,onecanmanuallyaddthestructuraldescriptionfortheshardkeysandthenumberanddistributionoftheshardsandthentherestoreintoaclusterwillwork.
IfyourestoreacollectionthatwasdumpedfromaclusterintoasingleArangoDBinstance,thenumberofshardsandtheshardkeyswillsilentlybeignored.
Notethatinacluster,everynewlycreatedcollectionwillhaveanewID,itisnotpossibletoreusetheIDfromtheoriginallydumpedcollection.ThisisforsafetyreasonstoensureconsistencyofIDs.
Restoringcollectionswithshardingprototypes
arangorestorewillyieldanerror,whiletryingtorestoreacollection,whosesharddistributionfollowsacollection,whichdoesnotexistintheclusterandwhichwasnotdumpedalong:
unix>arangorestore--collectionclonedCollection--server.databasemydb--input-directory"dump"
ERRORgoterrorfromserver:HTTP500(InternalServerError):ArangoError1486:mustnothaveadistributeShardsLikeattribute
pointingtoanunknowncollection
Processed0collection(s),read0byte(s)fromdatafiles,sent0batch(es)
Thecollectioncanberestoredbyoverridingtheerrormessageasfollows:
unix>arangorestore--collectionclonedCollection--server.databasemydb--input-directory"dump"--ignore-distribute-shards-li
ke-errors
RestoreintoanauthenticationenabledArangoDB
OfcourseyoucanrestoredataintoapasswordprotectedArangoDBaswell.Howeverthisrequirescertainuserrightsfortheuserusedintherestoreprocess.TherightsaredescribedindetailintheManagingUserschapter.Forrestorethisshortoverviewissufficient:
Whenimportingintoanexistingdatabase,thegivenuserneedsAdministrateaccessonthisdatabase.WhencreatinganewDatabaseduringrestore,thegivenuserneedsAdministrateaccesson_system.TheuserwillbepromotedwithAdministrateaccessonthenewlycreateddatabase.
Arangorestore
407
ExportingDatafromanArangoDBdatabaseToexportdatafromanArangoDBserverinstance,youwillneedtoinvokearangoexport.arangoexportcanbeinvokedbyexecutingthefollowingcommand:
unix>arangoexport--collectiontest--output-directory"dump"
Thisexportsthecollectionstestintothedirectorydumpasonebigjsonarray.Everyentryinthisarrayisonedocumentfromthecollectionwithoutaspecificorder.Toexportmorethanonecollectionatatimespecifymultiple--collectionoptions.
Thedefaultoutputdirectoryisexport.
arangoexportwillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:
--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:username--server.password:passwordtouse(omitthisandyou'llbepromptedforthepassword)--server.authentication:whetherornottouseauthentication
Here'sanexampleofexportingdatafromanon-standardendpoint,usingadedicateddatabasename:
unix>arangoexport--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--collectiontes
t--output-directory"my-export"
Whenfinished,arangoexportwillprintoutasummarylinewithsomeaggregatestatisticsaboutwhatitdid,e.g.:
Processed2collection(s),wrote9031763Byte(s),78HTTPrequest(s)
ExportJSON
unix>arangoexport--typejson--collectiontest
Thisexportsthecollectiontestintotheoutputdirectoryexportasonejsonarray.Everyarrayentryisonedocumentfromthecollectiontest
ExportJSONL
unix>arangoexport--typejsonl--collectiontest
Thisexportsthecollectiontestintotheoutputdirectoryexportasjsonl.Everylineintheexportisonedocumentfromthecollectiontestasjson.
ExportCSV
unix>arangoexport--typecsv--collectiontest--fields_key,_id,_rev
ThisexportsthecollectiontestintotheoutputdirectoryexportasCSV.Thefirstlinecontainstheheaderwithallfieldnames.EachlineisonedocumentrepresentedasCSVandseparatedwithacomma.ObjectsandArraysarerepresentedasaJSONstring.
Arangoexport
408
ExportXML
unix>arangoexport--typexml--collectiontest
ThisexportsthecollectiontestintotheoutputdirectoryexportasgenericXML.TherootelementofthegeneratedXMLfileisnamedcollection.EachdocumentinthecollectionisexportedinadocXMLattribute.Eachdocumentattributeisexportinagenericattelement,whichhasatypeattributeindicatingtheattributevalue,andavalueattributecontainingtheattribute'svalue.
ExportXGMMLXGMMLisanXMLapplicationbasedonGML.ToviewtheXGMMLfileyoucanuseforexampleCytoscape.
importantnoteIfyouexportallattributes(--xgmml-label-onlyfalse)keepinmindthataatrributenamestypehavetobethesametypeforalldocuments.Itwontworkifyouhaveaattributenamedrankthatisinonedocumentastringandinanotherdocumentainteger.
Bad
//doc1
{
"rank":1
}
//doc2
{
"rank":"2"
}
Good
//doc1
{
"rank":1
}
//doc2
{
"rank":2
}
XGMMLspecificoptions--xgmml-label-attributespecifythenameoftheattributethatwillbecomethelabelinthexgmmlfile.
--xgmml-label-onlysettotruewillonlyexportthelabelwithoutanyattributesinedgesornodes.
exportbasedoncollections
unix>arangoexport--typexgmml--graph-namemygraph--collectionvertex--collectionedge
Thisexportstheaunnamedgraphwithvertexcollectionvertexandedgecollectionedgeintothexgmmlfilemygraph.xgmml.
exportbasedonanamedgraph
unix>arangoexport--typexgmml--graph-namemygraph
Arangoexport
409
Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmml.
exportXGMMLwithoutattributes
unix>arangoexport--typexgmml--graph-namemygraph--xgmml-label-onlytrue
Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmmlwithoutthe<att>taginnodesandedges.
exportXGMMLwithaspecificlabel
unix>arangoexport--typexgmml--graph-namemygraph--xgmml-label-attributename
Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmmlwithalabelfromdocumentsattributenameinsteadofthedefaultattributelabel.
ExportviaAQLquery
unix>arangoexport--typejsonl--query"forbookinbooksfilterbook.sells>100returnbook"
Exportviaanaqlqueryallowsyoutoexportthereturneddataasthetypespecifiedwith--type.Theexampleexportsallbooksasjsonlthataresoldmorethan100times.
Arangoexport
410
ManagingUsersTheusermanagementinArangoDB3issimilartotheonesfoundinMySQL,PostgreSQL,orotherdatabasesystems.
Usermanagementispossibleinthewebinterfaceandinarangoshwhileloggedontothe_systemdatabase.
Notethattheonlyusernamesmustnotstartwith:role:.
ActionsandAccessLevelsAnArangoDBservercontainsalistofusers.Italsodefinesvariousaccesslevelsthatcanbeassignedtoauser(fordetails,seebelow)andthatareneededtoperformcertainactions.Theseactionscanbegroupedintothreecategories:
serveractionsdatabaseactionscollectionactions
Theserveractionsare
createuser:allowstocreateanewuser.
updateuser:allowstochangetheaccesslevelsanddetailsofanexistinguser.
dropuser:allowstodeleteanexistinguser.
createdatabase:allowstocreateanewdatabase.
dropdatabase:allowstodeleteanexistingdatabase.
shutdownserver:removeserverfromclusterandshutdown
Thedatabaseactionsaretiedtoagivendatabase,andaccesslevelsmustbesetforeachdatabaseindividually.Foragivendatabasetheactionsare
createcollection:allowstocreateanewcollectioninthegivendatabase.
updatecollection:allowstoupdatepropertiesofanexistingcollection.
dropcollection:allowstodeleteanexistingcollection.
createindex:allowstocreateanindexforanexistingcollectioninthegivendatabase.
dropindex:allowstodeleteanindexofanexistingcollectioninthegivendatabase.
Thecollectionactionsaretiedtoagivencollectionofagivendatabase,andaccesslevelsmustbesetforeachcollectionindividually.Foragivencollectiontheactionsare
readdocument:readadocumentofthegivencollection.
createdocument:createsanewdocumentinthegivencollection.
modifydocument:modifiesanexistingdocumentofthegivencollection,thiscanbeanupdateorreplaceoperation.
dropdocument:deletesanexistingdocumentofthegivencollection.
truncatecollection:deletesalldocumentsofagivencollection.
Toperformactionsontheserverleveltheuserneedsatleastthefollowingaccesslevels.TheaccesslevelsareAdministrateandNoaccess:
serveraction serverlevel
createadatabase Administrate
dropadatabase Administrate
createauser Administrate
ManagingUsers
411
updateauser Administrate
updateuseraccesslevel Administrate
dropauser Administrate
shutdownserver Administrate
Toperformactionsinaspecificdatabase(likecreatingordroppingcollections),auserneedsatleastthefollowingaccesslevel.ThepossibleaccesslevelsfordatabasesareAdministrate,AccessandNoaccess.TheaccesslevelsforcollectionsareRead/Write,ReadOnlyandNoAccess.
databaseaction databaselevel collectionlevel
createcollection Administrate Read/Write
listcollections Access ReadOnly
renamecollection Administrate Read/Write
modifycollectionproperties Administrate Read/Write
readproperties Access ReadOnly
dropcollection Administrate Read/Write
createanindex Administrate Read/Write
dropanindex Administrate Read/Write
seeindexdefinition Access ReadOnly
NotethattheaccesslevelAccessforadatabaseisalwaysrequiredtoperformanyactiononacollectioninthatdatabase.
Forcollectionsauserneedsthefollowingaccesslevelstothegivendatabaseandthegivencollection.TheaccesslevelsforthedatabaseareAdministrate,AccessandNoaccess.TheaccesslevelsforthecollectionareRead/Write,ReadOnlyandNoAccess.
action collectionlevel databaselevel
readadocument Read/WriteorReadOnly AdministrateorAccess
createadocument Read/Write AdministrateorAccess
modifyadocument Read/Write AdministrateorAccess
dropadocument Read/Write AdministrateorAccess
truncateacollection Read/Write AdministrateorAccess
Example
Forexample,given
adatabaseexampleacollectiondatainthedatabaseexampleauserJohnSmith
IftheuserJohnSmithisassignedtheaccesslevelAccessforthedatabaseexampleandthelevelRead/Writeforthecollectiondata,thentheuserisallowedtoread,create,modifyordeletedocumentsinthecollectiondata.Buttheuseris,forexample,notallowedtocreateindexesforthecollectiondatanorcreatenewcollectionsinthedatabaseexample.
GrantingAccessLevels
Accesslevelscanbemanagedviathewebinterfaceorinarangosh.
Inordertograntanaccessleveltoauser,youcanassignoneofthreeaccesslevelsforeachdatabaseandoneofthreelevelsforeachcollectioninadatabase.Theserveraccesslevelfortheuserfollowsfromthedatabaseaccesslevelinthe_systemdatabase,itisAdministrateifandonlyifthedatabaseaccesslevelisAdministrate.NotethatthismeansthatdatabaseaccesslevelAccessdoesnotgrantauserserveraccesslevelAdministrate.
ManagingUsers
412
InitialAccessLevels
WhenausercreatesadatabasetheaccessleveloftheuserforthatdatabaseissettoAdministrate.Thesameistrueforcreatingacollection,inthiscasetheusergetRead/Writeaccesstothecollection.
WildcardDatabaseAccessLevel
Withtheabovedefinition,onemustdefinethedatabaseaccesslevelforalldatabase/userpairsintheserver,whichwouldbeverytedious.Inordertosimplifythisprocess,itispossibletodefine,forauser,awildcarddatabaseaccesslevel.Thiswildcardisusedifthedatabaseaccesslevelisnotexplicitlydefinedforacertaindatabase.EachnewcreateduserhasaninitialdatabasewildcardofNoAccess.
Changingthewildcarddatabaseaccesslevelforauserwillchangetheaccesslevelforalldatabasesthathavenoexplicitlydefinedaccesslevel.Notethatthisincludesdatabaseswhichwillbecreatedinthefutureandforwhichnoexplicitaccesslevelsaresetforthatuser!
Ifyoudeletethewildcard,thedefaultaccesslevelisdefinedasNoAccess.
TherootuserhasaninitialdatabasewildcardofAdministrate.
Example
AssumeuserJohnSmithhasthefollowingdatabaseaccesslevels:
accesslevel
database* Access
databaseshop1 Administrate
databaseshop2 NoAccess
ThiswillgivetheuserJohnSmiththefollowingdatabaselevelaccess:
databaseshop1:Administratedatabaseshop2:NoAccessdatabasesomething:Access
Ifthewildcard*ischangedfromAccesstoNoAccessthenthepermissionswillchangeasfollows:
databaseshop1:Administratedatabaseshop2:NoAccessdatabasesomething:NoAccess
WildcardCollectionAccessLevel
Foreachuseranddatabasethereisawildcardcollectionaccesslevel.Thislevelisusedforallcollectionspairswithoutanexplicitlydefinedcollectionaccesslevel.Notethatthisincludescollectionswhichwillbecreatedinthefutureandforwhichnoexplicitaccesslevelsaresetforathatuser!EachnewcreateduserhasaninitialcollectionwildcardofNoAccess.
Ifyoudeletethewildcard,thesystemdefaultstoNoAccess.
TherootuserhasaninitialcollectionwildcardofRead/Writeineverydatabase.
Whencreatingauserthroughdb._createDatabase(name,options,users)theaccessleveloftheuserforthisdatabasewillbesettoAdministrateandthewildcardforallcollectionswithinthisdatabasewillbesettoRead/Write.
Example
AssumeuserJohnSmithhasthefollowingdatabaseaccesslevels:
accesslevel
database* Access
andcollectionaccesslevels:
accesslevel
ManagingUsers
413
database*,collection* Read/Write
databaseshop1,collectionproducts Read-Only
databaseshop1,collection* NoAccess
databaseshop2,collection* Read-Only
Thentheuserdoewillgetthefollowingcollectionaccesslevels:
databaseshop1,collectionproducts:Read-Onlydatabaseshop1,collectioncustomers:NoAccessdatabaseshop2,collectionreviews:Read-Onlydatabasesomething,collectionelse:Read/Write
Explanation:
Databaseshop1,collectionproductsdirectlymatchesadefinedaccesslevel.ThislevelisdefinedasRead-Only.
Databaseshop1,collectioncustomersdoesnotmatchadefinedaccesslevel.However,databaseshop1matchesandthewildcardinthisdatabaseforcollectionlevelisNoAccess.
Databaseshop2,collectionreviewsdoesnotmatchadefinedaccesslevel.However,databaseshop2matchesandthewildcardinthisdatabaseforcollectionlevelisRead-Only.
Databasesomehing,collectionelsedoesnotmatchadefinedaccesslevel.Thedatabasesomethingalsodoeshaveadirectmatches.Thereforethewildcardisselected.ThelevelisRead/Write.
PermissionResolution
Theaccesslevelsfordatabasesandcollectionsareresolvedinthefollowingway:
Foradatabase"foo":
1. Checkifthereisaspecificdatabasegrantforfoo,ifyesusethegrantedaccesslevel2. Chooseathehigheraccesslevelof:
Awildcarddatabasegrant(forexamplegrantDatabase('user','*','rw')Adatabasegrantonthe_systemdatabase
Foracollectionnamed"bar":
1. Checkifthereisaspecificdatabasegrantforbar,ifyesusethegrantedaccesslevel2. Chooseathehigheraccesslevelof:
Anywildcardaccessgrantinthesamedatabase,oron"/"(inthisexamplegrantCollection('user','foo','*','rw'))Theaccesslevelforthecurrentdatabase(inthisexamplegrantDatabase('user','foo','rw')Theaccesslevelforthe_systemdatabase
Anexceptiontothisaresystemcollections,hereonlytheaccesslevelforthedatabaseisused.
SystemCollections
Theaccesslevelforsystemcollectionscannotbechanged.Theyfollowdifferentrulesthanuserdefinedcollectionsandmaychangewithoutfurthernotice.Currentlythesystemcollectionsfollowtheserules:
collection accesslevel
_users(in_system) NoAccess
_queues Read-Only
_frontend Read/Write
* sameasdb
AllothersystemcollectionshaveaccesslevelRead/WriteiftheuserhasAdministrateaccesstothedatabase.TheyhaveaccesslevelRead/OnlyiftheuserhasAccesstothedatabase.
ManagingUsers
414
TomodifythesesystemcollectionsyoushouldalwaysusethespecializedAPIsprovidedbyArangoDB.Forexamplenouserhasaccesstothe_userscollectioninthe_systemdatabase.Allchangestotheaccesslevelsmustbedoneusingthe@arangodb/usersmodule,the/_users/APIorthewebinterface.
LDAPUsers
ThisfeatureisonlyavailableintheEnterpriseEdition.
ArangoDBsupportsLDAPasanexternalauthenticationsystem.FordetailedinformationpleasehavelookintotheLDAPconfigurationguide.
ThereareafewdifferencestonormalArangoDBusers:
ArangoDBdoesnot"know"LDAPusersbeforetheyfirstauthenticate,callstovariousAPI'susingendpointsin_api/users/*willfailuntiltheuserfirstlogs-inAccesslevelsofeachuserareperiodicallyupdated,thiswillhappenbydefaultevery5minutesItisnotpossibletochangepermissionsonLDAPusersdirectly,onlyonrolesLDAPuserscannotstoreconfigurationdataperuser(affectsforexamplecustomsettingsinthegraphviewer)
TograntaccessforanLDAPuseryouwillneedtocreateroleswithintheArangoDBserver.Aroleisjustauserwiththe":role:"prefixinitsname.Roleuserscannotloginasdatabaseusers,the":role:"prefixensuresthis.YourLDAPuserswillneedtohaveatleastonerole,oncetheuserlogsinhewillbeautomaticallygrantedtheunionofallaccessrightsofallhisroles.Notethatalowerrightgrantinonerolewillbeoverwrittenbyahigheraccessgrantinadifferentrole.
ManagingUsers
415
ManagingUsersintheArangoDBShellPleasenote,thatforbackwardcompatibilitytheserveraccesslevelsfollowfromthedatabaseaccesslevelonthedatabase_system.
Alsonotethattheserveranddatabaseaccesslevelsarerepresentedas
rw:forAdministratero:forAccessnone:forNoaccess
Thisisagainforbackwardcompatibility.
Example
Fireuparangoshandrequiretheusersmodule.Useittocreateanewuser:
arangosh>varusers=require('@arangodb/users');
arangosh>users.save('JohnSmith','mypassword');
CreatesausercalledJohnSmith.Thisuserwillhavenoaccessatall.
arangosh>users.grantDatabase('JohnSmith','testdb','rw');
ThisgrantstheuserAdministrateaccesstothedatabasetestdb.revokeDatabasewillrevokethisaccesslevelsetting.
Note:Beawarethatfrom3.2onwardsthegrantDatabasewillnotautomaticallygrantuserstheaccessleveltowriteorreadcollectionsinadatabase.IfyougrantaccesstoadatabasetestdbyouwilladditionallyneedtoexplicitlygrantaccesslevelstoindividualcollectionsviagrantCollection.
Theupgradeprocedurefrom3.1to3.2setsthewildcarddatabaseaccesslevelforalluserstoAdministrateandsetsthewildcardcollectionaccesslevelforalluser/databasepairstoRead/Write.
arangosh>users.grantCollection('JohnSmith','testdb','testcoll','rw');
Saveusers.save(user,passwd,active,extra)
ThiswillcreateanewArangoDBuser.Theusernamemustbespecifiedinuserandmustnotbeempty.
Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.IfyoupassthespecialvalueARANGODB_DEFAULT_ROOT_PASSWORD,thepasswordwillbesetthevaluestoredintheenvironmentvariableARANGODB_DEFAULT_ROOT_PASSWORD.ThiscanbeusedtopassaninstancevariableintoArangoDB.Forexample,theinstanceidentifierfromAmazon.
Iftheactiveattributeisnotspecified,itdefaultstotrue.Theextraattributecanbeusedtosavecustomdatawiththeuser.
Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,ortherealreadyexistsauserwiththespecifiedname.
Note:Theuserwillnothavepermissiontoaccessanydatabase.YouneedtogranttheaccessrightsforoneormoredatabasesusinggrantDatabase.
Examples
arangosh>require('@arangodb/users').save('my-user','my-secret-password');
showexecutionresults
InArangosh
416
GrantDatabaseusers.grantDatabase(user,database,type)
Thisgrantstype('rw','ro'or'none')accesstothedatabasefortheuser.Ifdatabaseis"*",thissetsthewildcarddatabaseaccesslevelfortheuseruser.
Theserveraccesslevelfollowsfromtheaccesslevelforthedatabase_system.
RevokeDatabaseusers.revokeDatabase(user,database)
Thisclearstheaccesslevelsettingtothedatabasefortheuserandthewildcarddatabaseaccesssettingforthisuserkicksin.IncasenowildcardaccesswasdefinedthedefaultisNoAccess.Thiswillalsocleartheaccesslevelsforallthecollectionsinthisdatabase.
GrantCollectionusers.grantCollection(user,database,collection,type)
Thisgrantstype('rw','ro'or'none')accessleveltothecollectionindatabasefortheuser.Ifcollectionis"*"thissetsthewildcardcollectionaccesslevelfortheuseruserindatabasedatabase.
RevokeCollectionusers.revokeCollection(user,database)
Thisclearstheaccesslevelsettingtothecollectioncollectionfortheuseruser.ThesystemwilleitherfallbacktothewildcardcollectionaccesslevelordefaulttoNoAccess
Replaceusers.replace(user,passwd,active,extra)
ThiswilllookupanexistingArangoDBuserandreplaceitsuserdata.
Theusernamemustbespecifiedinuser,andauserwiththespecifiednamemustalreadyexistinthedatabase.
Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.
Iftheactiveattributeisnotspecified,itdefaultstotrue.Theextraattributecanbeusedtosavecustomdatawiththeuser.
Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,orifthespecifiedusercannotbefoundinthedatabase.
Note:thisfunctionwillnotworkfromwithinthewebinterface
Examples
arangosh>require("@arangodb/users").replace("my-user","my-changed-password");
showexecutionresults
Updateusers.update(user,passwd,active,extra)
ThiswillupdateanexistingArangoDBuserwithanewpasswordandotherdata.
Theusernamemustbespecifiedinuserandtheusermustalreadyexistinthedatabase.
InArangosh
417
Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.
Iftheactiveattributeisnotspecified,thecurrentvaluesavedfortheuserwillnotbechanged.Thesameistruefortheextraattribute.
Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,orifthespecifiedusercannotbefoundinthedatabase.
Examples
arangosh>require("@arangodb/users").update("my-user","my-secret-password");
showexecutionresults
isValidusers.isValid(user,password)
Checkswhetherthegivencombinationofusernameandpasswordisvalid.Thefunctionwillreturnabooleanvalueifthecombinationofusernameandpasswordisvalid.
Eachcalltothisfunctionispenalizedbytheserversleepingarandomamountoftime.
Examples
arangosh>require("@arangodb/users").isValid("my-user","my-secret-password");
true
Removeusers.remove(user)
RemovesanexistingArangoDBuserfromthedatabase.
TheusernamemustbespecifiedinUserandthespecifiedusermustexistinthedatabase.
Thismethodwillfailiftheusercannotbefoundinthedatabase.
Examples
arangosh>require("@arangodb/users").remove("my-user");
Documentusers.document(user)
FetchesanexistingArangoDBuserfromthedatabase.
Theusernamemustbespecifiedinuser.
Thismethodwillfailiftheusercannotbefoundinthedatabase.
Examples
arangosh>require("@arangodb/users").document("my-user");
showexecutionresults
All
InArangosh
418
users.all()
FetchesallexistingArangoDBusersfromthedatabase.
Examples
arangosh>require("@arangodb/users").all();
showexecutionresults
Reloadusers.reload()
Reloadstheuserauthenticationdataontheserver
Alluserauthenticationdataisloadedbytheserveronceonstartuponlyandiscachedafterthat.Whenusersgetaddedordeleted,acacheflushisdoneautomatically,andthiscanbeperformedbyacalltothismethod.
Examples
arangosh>require("@arangodb/users").reload();
Permissionusers.permission(user,database[,collection])
Fetchestheaccessleveltothedatabaseoracollection.
Theuseranddatabasenamemustbespecified,optionallyyoucanspecifythecollectionname.
Thismethodwillfailiftheusercannotbefoundinthedatabase.
Examples
arangosh>require("@arangodb/users").permission("my-user","testdb");
rw
InArangosh
419
Command-lineoptions
GeneralOptions
Generalhelp--help
-h
Printsalistofthemostcommonoptionsavailableandthenexits.Inordertoseealloptionsuse--help-all.
Tomakeuseofthestartupoptionsfromaprogram,theoption--dump-optionswillprintoutalloptionsinJSONformatandthenexits.
Version
--version
-v
Printstheversionoftheserverandexits.
ConfigurationFiles
Optionscanbespecifiedonthecommandlineorinconfigurationfiles.IfastringVariableoccursinthevalue,itisreplacedbythecorrespondingenvironmentvariable.
--configurationfilename
-cfilename
Specifiesthenameoftheconfigurationfiletouse.
Ifthiscommandisnotpassedtotheserver,thenbydefault,theserverwillattempttofirstlocateafilenamed~/.arango/arangod.confintheuser'shomedirectory.
Ifnosuchfileisfound,theserverwillproceedtolookforafilearangod.confinthesystemconfigurationdirectory.Thesystemconfigurationdirectoryisplatform-specific,andmaybechangedwhencompilingArangoDByourself.Itmaydefaultto/etc/arangodbor/usr/local/etc/arangodb.Thisfileisinstalledwhenusingapackagemanagerlikerpmordpkg.IfyoumodifythisfileandlaterupgradetoanewversionofArangoDB,thenthepackagemanagernormallywarnsyouabouttheconflict.Inordertoavoidthesewarningforsmalladjustments,youcanputlocaloverridesintoafilearangod.conf.local.
Onlycommandlineoptionswithavalueshouldbesetwithintheconfigurationfile.Commandlineoptionswhichactasflagsshouldbeenteredonthecommandlinewhenstartingtheserver.
Eachoptionisspecifiedonaseparatelineintheform:
key=value
Alternatively,aheadersectioncanbespecifiedandoptionspertainingtothatsectioncanbespecifiedinashorterform
[log]
level=trace
ratherthanspecifying
log.level=trace
Soyouseeingeneral--section.paramvaluetranslatesto
[section]
param=value
ServerConfiguration
420
Whitespacearound=isignoredintheconfigurationfile.Donotputspacesaroundadditional=intheparametervaluehowever.Thefollowingexampleshowsthecorrectwaytospecifyalogleveloftraceforthetopicstartup:
log.level=startup=trace
Notethatthereisnowhitespacebetweenstartupand=,andalsonot=andtrace.
Whereonesectionmayoccurmultipletimes,andthelastoccuranceofparamwillbecomethefinalvalue.Incaseofparametersbeingvectors,multipleoccuranceaddsanotheritemtothevector.Vectorscanbeidentifiedbythe...inthe--helpoutputofthebinaries.
Commentscanbeplacedintheconfigurationfile,onlyifthelinebeginswithoneormorehashsymbols(#).
Theremaybeoccasionswhereaconfigurationfileexistsandtheuserwishestooverrideconfigurationsettingsstoredinaconfigurationfile.Anysettingsspecifiedonthecommandlinewilloverwritethesamesettingwhenitappearsinaconfigurationfile.Iftheuserwishestocompletelyignoreconfigurationfileswithoutnecessarilydeletingthefile(orfiles),thenaddthecommandlineoption
-cnone
or
--configurationnone
whenstartinguptheserver.Notethat,thewordnoneiscase-insensitive.
ServerConfiguration
421
OperatingSystemConfiguration
FileSystems
(LINUX)
WerecommendtonotuseBTRFSonlinux,it'sknowntonotworkwellinconjunctionwithArangoDB.WeexperiencedthatarangodbfacinglatencyissuesonaccessingitsdatabasefilesonBTRFSpartitions.InconjunctionwithBTRFSandAUFSwealsosawdatalossonrestart.
VirtualMemoryPageSizes(LINUX)
Bydefault,ArangoDBusesJemallocasthememoryallocator.Jemallocdoesagoodjobofreducingvirtualmemoryfragmentation,especiallyforlong-runningprocesses.Unfortunately,someOSconfigurationscaninterferewithJemalloc'sabilitytofunctionproperly.Specifically,Linux's"transparenthugepages",Windows'"largepages"andothersimilarfeaturessometimespreventJemallocfromreturningunusedmemorytotheoperatingsystemandresultinunnecessarilyhighmemoryuse.Therefore,werecommenddisablingthesefeatureswhenusingJemallocwithArangoDB.Pleaseconsultyouroperatingsystem'sdocumentationforhowtodothis.
Execute
sudobash-c"echomadvise>/sys/kernel/mm/transparent_hugepage/enabled"
sudobash-c"echomadvise>/sys/kernel/mm/transparent_hugepage/defrag"
beforeexecutingarangod.
SwapSpace
(LINUX)
Itisrecommendedtoassignswapspaceforaserverthatisrunningarangod.Configuringswapspacecanpreventtheoperatingsystem'sOOMkillerfromkillingArangoDBtooeagerlyonLinux.
Over-CommitMemory
FortheMMFilesstorageengine,execute
sudobash-c"echo0>/proc/sys/vm/overcommit_memory"
beforeexecutingarangod.
FortheRocksDBstorageengine,execute
sudobash-c"echo2>/proc/sys/vm/overcommit_memory"
beforestarting.
Fromwww.kernel.org:
Whenthisflagis0,thekernelattemptstoestimatetheamountoffreememoryleftwhenuserspacerequestsmorememory.
Whenthisflagis1,thekernelpretendsthereisalwaysenoughmemoryuntilitactuallyrunsout.
Whenthisflagis2,thekernelusesa"neverovercommit"policythatattemptstopreventanyovercommitofmemory.
OperatingSystemConfiguration
422
Notethatthenusinganovercommit_memorysettingof2,thiswillbydefaultallowprocessestouseallswapspacebutonlyhalfoftheavailableRAM.Thiscanbechangedbyadjustingthevalueofovercommit_ratioaswell.
Fromwww.kernel.org:
Whenovercommit_memoryissetto2,thecommittedaddressspaceisnotpermittedtoexceedswapplusthispercentageofphysicalRAM.
ZoneReclaim
Execute
sudobash-c"echo0>/proc/sys/vm/zone_reclaim_mode"
beforeexecutingarangod.
Fromwww.kernel.org:
ThisisvalueORedtogetherof
1=Zonereclaimon2=Zonereclaimwritesdirtypagesout4=Zonereclaimswapspages
NUMAMulti-processorsystemsoftenhavenon-uniformAccessMemory(NUMA).ArangoDBshouldbestartedwithinterleaveonsuchsystem.Thiscanbeachievedusing
numactl--interleave=allarangod...
MaxMemoryMappings(LINUX)
Linuxkernelsbydefaultrestrictthemaximumnumberofmemorymappingsofasingleprocesstoabout64Kmappings.Whilethisvalueissufficientformostworkloads,itmaybetoolowforaprocessthathaslotsofparallelthreadsthatallrequiretheirownmemorymappings.Inthiscaseallthethreads'memorymappingswillbeaccountedtothesinglearangodprocess,andthemaximumnumberof64Kmappingsmaybereached.Whenthemaximumnumberofmappingsisreached,callstommapwillfail,sotheprocesswillthinknomorememoryisavailablealthoughtheremaybeplentyofRAMleft.
Toavoidthisscenario,itisrecommendedtoraisethedefaultvalueforthemaximumnumberofmemorymappingstoasufficientlyhighvalue.Asaruleofthumb,onecoulduse8timesthenumberofavailablecorestimes8,000.
Fora32coreserver,agoodrule-of-thumbvaluethuswouldbe2,048,000(3288000).Forcertainworkloads,itmaybesensibletouseevenahighervalueforthenumberofmemorymappings.
Tosetthevalueonce,usethefollowingcommandbeforestartingarangod:
sudobash-c"sysctl-w'vm.max_map_count=2048000'"
Tomakethesettingsdurable,itwillbenecessarytostoretheadjustedsettingsin/etc/sysctl.conforotherplacesthattheoperatingsystemislookingat.
EnvironmentVariables
(LINUX)
OperatingSystemConfiguration
423
ItisrecommendedtosettheenvironmentvariableGLIBCXX_FORCE_NEWto1onsystemsthatuseglibc++inordertodisablethememorypoolingbuiltintoglibc++.ThatmemorypoolingisunnecessarybecauseJemallocwillalreadydomemorypooling.
Execute
exportGLIBCXX_FORCE_NEW=1
beforestartingarangod.
32bit
WhileitispossibletocompileArangoDBon32bitsystem,thisisnotarecommendedenvironment.64bitsystemscanaddressasignificantlybiggermemoryregion.
OperatingSystemConfiguration
424
ManagingEndpointsTheArangoDBservercanlistenforincomingrequestsonmultipleendpoints.
TheendpointsarenormallyspecifiedeitherinArangoDB'sconfigurationfileoronthecommand-line,usingthe--server.endpoint.ArangoDBsupportsdifferenttypesofendpoints:
tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryptionunix:///path/to/socket-Unixdomainsocketendpoint
IfaTCP/IPendpointisspecifiedwithoutaportnumber,thenthedefaultport(8529)willbeused.Ifmultipleendpointsneedtobeused,theoptioncanberepeatedmultipletimes.
ThedefaultendpointforArangoDBistcp://127.0.0.1:8529ortcp://localhost:8529.
EXAMPLES
unix>./arangod--server.endpointtcp://127.0.0.1:8529
--server.endpointssl://127.0.0.1:8530
--ssl.keyfileserver.pem/tmp/vocbase
2012-07-26T07:07:47Z[8161]INFOusingSSLprotocolversion'TLSv1'
2012-07-26T07:07:48Z[8161]INFOusingendpoint'ssl://127.0.0.1:8530'forhttpsslrequests
2012-07-26T07:07:48Z[8161]INFOusingendpoint'tcp://127.0.0.1:8529'forhttptcprequests
2012-07-26T07:07:49Z[8161]INFOArangoDB(version1.1.alpha)isreadyforbusiness
2012-07-26T07:07:49Z[8161]INFOHaveFun!
TCPEndpoints
Givenahostname:
--server.endpointtcp://hostname:port
GivenanIPv4address:
--server.endpointtcp://ipv4-address:port
GivenanIPv6address:
--server.endpointtcp://[ipv6-address]:port
Ononespecificethernetinterfaceeachportcanonlybeboundexactlyonce.YoucanlookupyouravailableinterfacesusingtheifconfigcommandonLinux/MacOSX-theWindowsequivalentisipconfig(SeeWikipediaformoredetails).ThegeneralnamesoftheinterfacesdifferonOS'sandhardwarestheyrunon.However,typicallyeveryhosthasasocalledloopbackinterface,whichisavirtualinterface.Byconventionitalwayshastheaddress127.0.0.1or::1(ipv6),andcanonlybereachedfromexactlytheverysamehost.Ethernetinterfacesusuallyhavenameslikeeth0,wlan0,eth1:17,le0oraplaintextnameinWindows.
Tofindoutwhichservicesalreadyuseports(soArangoDBcan'tbindthemanymore),youcanusethenetstatcommand(itbehavesalittledifferentoneachplatform,runitwith-lnptonLinux,-ptcponMacOSXorwith-anonwindowsforvaluableinformation).
ArangoDBcanalsodoasocalledbroadcastbindusingtcp://0.0.0.0:8529.Thiswayitwillbereachableonallinterfacesofthehost.Thismaybeusefulondevelopmentsystemsthatfrequentlychangetheirnetworksetuplikelaptops.
SpecialnoteonIPv6link-localaddresses
ArangoDBcanalsolistentoIPv6link-localaddressesviaaddingthezoneIDtotheIPv6addressintheform[ipv6-link-local-address%zone-id].However,whatyouprobablyinsteadwantistobindtoalocalIPv6address.LocalIPv6addressesstartwithfd.Ifyouonlyseeafe80:IPv6addressinyourinterfaceconfigurationbutnoIPv6addressstartingwithfdyourinterfacehasnolocalIPv6addressassigned.YoucanreadmoreaboutIPv6link-localaddresseshere.
Example
ManagingEndpoints
425
Bindtoalink-localandlocalIPv6address.
unix>ifconfig
Thiscommandlistsallinterfacesandassignedipaddresses.Thelink-localaddressmaybefe80::6257:18ff:fe82:3ec6%eth0(IPv6addressplusinterfacename).AlocalIPv6addressmaybefd12:3456::789a.TobindArangoDBtoitstartarangodwith--server.endpointtcp://[fe80::6257:18ff:fe82:3ec6%eth0]:8529.Usetelnettotesttheconnection.
unix>telnetfe80::6257:18ff:fe82:3ec6%eth08529
Tryingfe80::6257:18ff:fe82:3ec6...
Connectedtomy-machine.
Escapecharacteris'^]'.
GET/HTTP/1.1
HTTP/1.1301MovedPermanently
Location:/_db/_system/_admin/aardvark/index.html
Content-Type:text/html
Server:ArangoDB
Connection:Keep-Alive
Content-Length:197
<html><head><title>Moved</title></head><body><h1>Moved</h1><p>Thispagehasmovedto<ahref="/_db/_system/_admin/aardvark/inde
x.html">/_db/_system/_admin/aardvark/index.html</a>.</p></body></html>
Reuseaddress--tcp.reuse-address
IfthisbooleanoptionissettotruethenthesocketoptionSO_REUSEADDRissetonallserverendpoints,whichisthedefault.Ifthisoptionissettofalseitispossiblethatittakesuptoaminuteafteraserverhasterminateduntilitispossibleforanewservertousethesameendpointagain.Thisiswhythisisactivatedbydefault.
Pleasenotehoweverthatundersomeoperatingsystemsthiscanbeasecurityriskbecauseitmightbepossibleforanotherprocesstobindtothesameaddressandport,possiblyhijackingnetworktraffic.UnderWindows,ArangoDBadditionallysetstheflagSO_EXCLUSIVEADDRUSEasameasuretoalleviatethisproblem.
Backlogsize--tcp.backlog-size
AllowstospecifythesizeofthebacklogforthelistensystemcallThedefaultvalueis10.Themaximumvalueisplatform-dependent.Specifyingahighervaluethandefinedinthesystemheader'sSOMAXCONNmayresultinawarningonserverstart.Theactualvalueusedbylistenmayalsobesilentlytruncatedonsomeplatforms(thishappensinsidethelistensystemcall).
ManagingEndpoints
426
SSLConfiguration
SSLEndpoints
Givenahostname:
--server.endpointtcp://hostname:port
GivenanIPv4address:
--server.endpointtcp://ipv4-address:port
GivenanIPv6address:
--server.endpointtcp://[ipv6-address]:port
Note:IfyouareusingSSL-encryptedendpoints,youmustalsosupplythepathtoaservercertificateusingthe--ssl.keyfileoption.
Keyfile--ssl.keyfilefilename
IfSSLencryptionisused,thisoptionmustbeusedtospecifythefilenameoftheserverprivatekey.ThefilemustbePEMformattedandcontainboththecertificateandtheserver'sprivatekey.
Thefilespecifiedbyfilenamecanbegeneratedusingopenssl:
#createprivatekeyinfile"server.key"
opensslgenrsa-des3-outserver.key1024
#createcertificatesigningrequest(csr)infile"server.csr"
opensslreq-new-keyserver.key-outserver.csr
#copyawayoriginalprivatekeyto"server.key.org"
cpserver.keyserver.key.org
#removepassphrasefromtheprivatekey
opensslrsa-inserver.key.org-outserver.key
#signthecsrwiththekey,createscertificatePEMfile"server.crt"
opensslx509-req-days365-inserver.csr-signkeyserver.key-outserver.crt
#combinecertificateandkeyintosinglePEMfile"server.pem"
catserver.crtserver.key>server.pem
YoumayusecertificatesissuedbyaCertificateAuthorityorself-signedcertificates.Self-signedcertificatescanbecreatedbyatoolofyourchoice.WhenusingOpenSSLforcreatingtheself-signedcertificate,thefollowingcommandsshouldcreateavalidkeyfile:
-----BEGINCERTIFICATE-----
(base64encodedcertificate)
-----ENDCERTIFICATE-----
-----BEGINRSAPRIVATEKEY-----
(base64encodedprivatekey)
-----ENDRSAPRIVATEKEY-----
Forfurtherinformationpleasecheckthemanualsofthetoolsyouusetocreatethecertificate.
CAFile--ssl.cafilefilename
SSLConfiguration
427
ThisoptioncanbeusedtospecifyafilewithCAcertificatesthataresenttotheclientwhenevertheserverrequestsaclientcertificate.Ifthefileisspecified,TheserverwillonlyacceptclientrequestswithcertificatesissuedbytheseCAs.Donotspecifythisoptionifyouwantclientstobeabletoconnectwithoutspecificcertificates.
ThecertificatesinfilenamemustbePEMformatted.
SSLprotocol
--ssl.protocolvalue
Usethisoptiontospecifythedefaultencryptionprotocoltobeused.Thefollowingvariantsareavailable:
1:SSLv22:SSLv2orSSLv3(negotiated)3:SSLv34:TLSv15:TLSv1.2
Thedefaultvalueis5(TLSv1.2).
SSLcache--ssl.session-cachevalue
SettotrueifSSLsessioncachingshouldbeused.
valuehasadefaultvalueoffalse(i.e.nocaching).
SSLpeercertificate
ThisfeatureisavailableintheEnterpriseEdition.
--ssl.require-peer-certificate
Requireapeercertificatefromtheclientbeforeconnecting.
SSLoptions
--ssl.optionsvalue
ThisoptioncanbeusedtosetvariousSSL-relatedoptions.IndividualoptionvaluesmustbecombinedusingbitwiseOR.
WhichoptionsareavailableonyourplatformisdeterminedbytheOpenSSLversionyouuse.Thelistofoptionsavailableonyourplatformmightberetrievedbythefollowingshellcommand:
>grep"#defineSSL_OP_.*"/usr/include/openssl/ssl.h
#defineSSL_OP_MICROSOFT_SESS_ID_BUG0x00000001L
#defineSSL_OP_NETSCAPE_CHALLENGE_BUG0x00000002L
#defineSSL_OP_LEGACY_SERVER_CONNECT0x00000004L
#defineSSL_OP_NETSCAPE_REUSE_CIPHER_CHANGE_BUG0x00000008L
#defineSSL_OP_SSLREF2_REUSE_CERT_TYPE_BUG0x00000010L
#defineSSL_OP_MICROSOFT_BIG_SSLV3_BUFFER0x00000020L
...
AdescriptionoftheoptionscanbefoundonlineintheOpenSSLdocumentation
SSLcipher--ssl.cipher-listcipher-list
ThisoptioncanbeusedtorestricttheservertocertainSSLciphersonly,andtodefinetherelativeusagepreferenceofSSLciphers.
Theformatofcipher-listisdocumentedintheOpenSSLdocumentation.
Tocheckwhichciphersareavailableonyourplatform,youmayusethefollowingshellcommand:
SSLConfiguration
428
>opensslciphers-v
ECDHE-RSA-AES256-SHASSLv3Kx=ECDHAu=RSAEnc=AES(256)Mac=SHA1
ECDHE-ECDSA-AES256-SHASSLv3Kx=ECDHAu=ECDSAEnc=AES(256)Mac=SHA1
DHE-RSA-AES256-SHASSLv3Kx=DHAu=RSAEnc=AES(256)Mac=SHA1
DHE-DSS-AES256-SHASSLv3Kx=DHAu=DSSEnc=AES(256)Mac=SHA1
DHE-RSA-CAMELLIA256-SHASSLv3Kx=DHAu=RSAEnc=Camellia(256)
Mac=SHA1
...
Thedefaultvalueforcipher-listis"ALL".
SSLConfiguration
429
LDAPThisfeatureisonlyavailableintheEnterpriseEdition.
BasicsConcepts
ThebasicideaisthatonecankeeptheuserauthenticationsetupforanArangoDBinstance(singleorcluster)outsideofArangoDBinanLDAPserver.AcrucialfeatureofthisisthatonecanaddandwithdrawusersandpermissionsbyonlychangingtheLDAPserverandinparticularwithouttouchingtheArangoDBinstance.ChangeswillbeeffectiveinArangoDBwithinafewminutes.
SincetherearemanydifferentpossibleLDAPsetups,wemustsupportavarietyofpossibilitiesforauthenticationandauthorization.Hereisashortoverview:
TomapArangoDBusernamestoLDAPuserstherearetwoauthenticationmethodscalled"simple"and"search".Inthe"simple"methodtheLDAPbinduserisderivedfromtheArangoDBusernamebyprependingaprefixandappendingasuffix.Forexample,auser"alice"couldbemappedtothedistinguishednameuid=alice,dc=arangodb,dc=comtoperformtheLDAPbindandauthentication.SeeSimpleauthenticationmethodbelowfordetailsandconfigurationoptions.
Inthe"search"methodtherearetwophases.InPhase1agenericread-onlyadminLDAPuseraccountisusedtobindtotheLDAPserverfirstandsearchforanLDAPusermatchingtheArangoDBusername.InPhase2,theactualauthenticationisthenperformedagainsttheLDAPuserthatwasfoundinphase1.Bothmethodsaresensibleandarerecommendedtouseinproduction.SeeSearchauthenticationmethodbelowfordetailsandconfigurationoptions.
Oncetheuserisauthenticated,therearenowtwomethodsforauthorization:(a)"rolesattribute"and(b)"rolessearch".
Inmethod(a)ArangoDBacquiresalistofrolestheauthenticatedLDAPuserhasfromtheLDAPserver.TheactualaccessrightstodatabasesandcollectionsfortheserolesareconfiguredinArangoDBitself.Theusereffectivelyhastheunionofallaccessrightsofallroleshehas.Thismethodisprobablythemostcommononeforproductionusecases.ItcombinestheadvantagesofmanagingusersandrolesoutsideofArangoDBintheLDAPserverwiththefinegrainedaccesscontrolwithinArangoDBfortheindividualroles.SeeRolesattributebelowfordetailsaboutmethod(a)andfortheassociatedconfigurationoptions.
Method(b)isverysimilarandonlydiffersfrom(a)inthewaytheactuallistofrolesofauserisderivedfromtheLDAPserver.SeeRolessearchbelowfordetailsaboutmethod(b)andfortheassociatedconfigurationoptions.
Fundamentaloptions
ThefundamentaloptionsforspecifyinghowtoaccesstheLDAPserverarethefollowing:
--ldap.enabledthisisabooleanoptionwhichmustbesettotruetoactivatetheLDAPfeature--ldap.serverisastringspecifyingthehostnameorIPaddressoftheLDAPserver--ldap.portisanintegerspecifyingtheporttheLDAPserverisrunningon,thedefaultis389--ldap.basednspecifiesthebasedistinguishednameunderwhichthesearchtakesplace(canalternativelybesetvia--ldap.url)--ldap.binddnand--ldap.bindpasswdaredistinguishednameandpasswordforaread-onlyLDAPusertowhichArangoDBcanbindtosearchtheLDAPserver.Notethatitisnecessarytoconfiguretheseforboththe"simple"and"search"authenticationmethods,sinceeveninthe"simple"method,ArangoDBoccasionallyhastorefreshtheauthorizationinformationfromtheLDAPservereveniftheusersessionpersistsandnonewauthenticationisneeded!Itis,however,allowedtoleavebothempty,butthentheLDAPservermustbereadablewithanonymousaccess.--ldap.refresh-rateisafloatingpointvalueinseconds.Thedefaultis300,whichmeansthatArangoDBwillrefreshtheauthorizationinformationforauthenticatedusersafteratmost5minutes.ThismeansthatchangesintheLDAPserverlikeremovedusersoraddedorremovedrolesforauserwillbeeffectiveafteratmost5minutes.
Notethatthe--ldap.serverand--ldap.portoptionscanalternativelybespecifiedinthe--ldap.urlstringtogetherwithotherconfigurationoptions.FordetailsseeSection"LDAPURLs"below.
HereisanexampleonhowtoconfiguretheconnectiontotheLDAPserver,withanonymousbind:
--ldap.enabled=true\
--ldap.server=ldap.arangodb.com\
LDAPOptions
430
--ldap.basedn=dc=arangodb,dc=com
WiththisconfigurationArangoDBbindsanonymouslytotheLDAPserveronhostldap.arangodb.comonthedefaultport389andexecutesallsearchesunderthebasedistinguishednamedc=arangodb,dc=com.
IfweneedausertoreadinLDAPhereistheexampleforit:
--ldap.enabled=true\
--ldap.server=ldap.arangodb.com\
--ldap.basedn=dc=arangodb,dc=com\
--ldap.binddn=uid=arangoadmin,dc=arangodb,dc=com\
--ldap.bindpasswd=supersecretpassword
Theconnectionisidenticalbutthesearcheswillbeexecutedwiththegivendistinguishednameinbinddn.
Notehere:Thegivenuser(ortheanonymousone)needsatleastreadaccessonalluserobjectstofindthemandinthecaseofRolessearchalsoreadaccessontheobjectsstoringtheroles.
UptothispointArangoDBcannowconnecttoagivenLDAPserverbutitisnotyetabletoauthenticateusersproperlywithit.Forthispickoneofthefollowingtwoauthenticationmethods.
LDAPURLs
AsanalternativeonecanspecifythevaluesofmultipleLDAPrelatedconfigurationoptionsbyspecifyingasingleLDAPURL.Hereisanexample:
--ldap.urlldap://ldap.arangodb.com:1234/dc=arangodb,dc=com?uid?sub
Thisoneoptionhasthecombinedeffectofsettingthefollowing:
--ldap.server=ldap.arangodb.com\
--ldap.port=1234\
--ldap.basedn=dc=arangodb,dc=com\
--ldap.searchAttribute=uid\
--ldap.searchScope=sub
Thatis,theLDAPURLconsistsoftheLDAPserverandport,abasedn,asearchattributeandascopewhichcanbeoneofbase,oneorsub.Thereisalsothepossibilitytousetheldapsprotocolasin:
--ldap.urlldaps://ldap.arangodb.com:636/dc=arangodb,dc=com?uid?sub
Thisdoesexactlythesameastheoneabove,exceptthatitusestheLDAPoverTLSprotocol.Thisisanon-standardmethodwhichdoesnotinvolveusingtheSTARTTLSprotocol.NotethatthisdoesnotworkintheWindowsversion!WesuggesttousetheldapprotocolandSTARTTLSasdescribedinthenextsection.
TLSoptions
TLSisnotsupportedintheWindowsversionofArangoDB!
ToconfiguretheusageofencryptedTLStocommunicatewiththeLDAPserverthefollowingoptionsareavailable:
--ldap.tlsThemainswitchtoactiveTLS.caneitherbetrue(useTLS)orfalse(donotuseTLS).Itisswitchedoffbydefault.IfyouswitchthisonanddonotusetheldapsprotocolviatheLDAPURL,thenArangoDBwillusetheSTARTTLSprotocoltoinitiateTLS.Thisistherecommendedapproach.--ldap.tls-versiontheminimalTLSversionthatArangoDBshouldaccept.Availableversionsare1.0,1.1and1.2.Thedefaultis1.2.IfyourLDAPserverdoesnotsupportVersion1.2,youhavetochangethissetting.--ldap.tls-cert-check-strategystrategytovalidatetheLDAPservercertificate.Availablestrategiesarenever,hard,demand,allowandtry.Thedefaultishard.
LDAPOptions
431
--ldap.tls-cacert-fileAfilepathtooneormore(concatenated)certificateauthoritycertificatesinPEMformat.Asdefaultnofilepathisconfigured.Thiscertificateisusedtovalidatetheserverresponse.--ldap.tls-cacert-dirAdirectorypathtocertificateauthoritycertificatesinc_rehashformat.Asdefaultnodirectorypathisconfigured.
AssumingyouhavetheTLSCAcertfilethatisgiventotheserverat/path/to/certificate.pem,hereisanexampleonhowtoconfigureTLS:
--ldap.tlstrue\
--ldap.tls-cacert-file/path/to/certificate.pem
YoucanuseTLSwithanyofthefollowingauthenticationmechanisms.
Authenticationmethods
InordertoauthenticateusersinLDAPwehavetwooptionsavailable.Weneedtopickexactlyonethem.
Simpleauthenticationmethod
Thesimpleauthenticationmethodisusedifandonlyifboththe--ldap.prefixand--ldap.suffixconfigurationoptionsarespecifiedandarenon-empty.Inallothercasesthe"search"authenticationmethodisused.
Inthe"simple"methodtheLDAPbinduserisderivedfromtheArangoDBusernamebyprependingthevalueofthe--ldap.prefixconfigurationoptionandbyappendingthevalueofthe--ldap.suffixconfigurationoption.Forexample,anArangoDBuser"alice"wouldbemappedtothedistinguishednameuid=alice,dc=arangodb,dc=comtoperformtheLDAPbindandauthentication,if--ldap.prefixissettouid=and--ldap.suffixissetto,dc=arangodb,dc=com.
ArangoDBbindstotheLDAPserverandauthenticateswiththedistinguishednameandthepasswordprovidedbytheclient.IftheLDAPserversuccessfullyverifiesthepasswordthentheuserisauthenticated.
IfyouwanttousethismethodaddthefollowingexampletoyourArangoDBconfigurationtogetherwiththefundamentalconfiguration:
--ldap.prefixuid=\
--ldap.suffix,dc=arangodb,dc=com
ThismethodwillauthenticateanLDAPuserwiththedistinguishedname{PREFIX}{USERNAME}{SUFFIX},inthiscaseforthearangouseraliceitwillsearchfor:uid=alice,dc=arangodb,dc=com.Thisdistinguishednamewillbeusedas{{USER}}fortheroleslateron.
Searchauthenticationmethod
Thesearchauthenticationmethodisusedifatleastoneofthetwooptions--ldap.prefixand--ldap.suffixisemptyornotspecified.ArangoDBusestheLDAPusercredentialsgivenbythe--ldap.binddnand--ldap.bindpasswdtoperformasearchforLDAPusers.Inthiscase,thevaluesoftheoptions--ldap.basedn,--ldap.search-attribute,--ldap.search-filterand--ldap.search-scopeareusedinthefollowingway:
--ldap.search-scopeisanLDAPsearchscopewithpossiblevaluesbase(justsearchthebasedistinguishedname),sub(recursivesearchunderthebasedistinguishedname)orone(searchthebase'simmediatechildren)(default:sub)--ldap.search-filterisanLDAPfilterexpressionwhichlimitsthesetofLDAPusersbeingconsidered(default:objectClass=*whichmeansallobjects)--ldap.search-attributespecifiestheattributeintheuserobjectswhichisusedtomatchtheArangoDBusername(default:uid)
Hereisanexampleonhowtoconfigurethesearchmethod.AssumewehaveuserslikethefollowingstoredinLDAP:
dn:uid=alice,dc=arangodb,dc=com
uid:alice
objectClass:inetOrgPerson
objectClass:organizationalPerson
objectClass:top
objectClass:person
LDAPOptions
432
WhereuidistheusernameusedinArangoDB,andweonlysearchforobjectsoftypepersonthenwecanaddthefollowingtoourfundamentalLDAPconfiguration:
--ldap.search-attribute=uid\
--ldap.search-filter=objectClass=person
Thiswillusethesubsearchscopebydefaultandwillfindallpersonobjectswheretheuidisequaltothegivenusername.Fromthesethednwillbeextractedandusedas{{USER}}intheroleslateron.
FetchingrolesforauserAfterauthentication,thenextstepistoderiveauthorizationinformationfromtheauthenticatedLDAPuser.Inordertofetchtherolesandtherebytheaccessrightsforauserweagainhavetwopossibleoptionsandneedtopickoneofthem.Wecancombineeachauthenticationmethodwitheachrolemethod.Inanycaseausercanhavenoroleormorethanone.IfauserhasnoroletheuserwillnotgetanyaccesstoArangoDBatall.Ifauserhasmultipleroleswithdifferentrightsthentherightswillbecombinedandthestrongestrightwillwin.Example:
alicehastherolesproject-aandproject-b.project-ahasnoaccesstocollectionBData.project-bhasrwaccesstocollectionBData,hencealicewillhaverwonBData.
NotethattheactualdatabaseandcollectionaccessrightswillbeconfiguredinArangoDBitselfbyrolesintheusersmodule.Therolenameisalwaysprefixedwith:role:,e.g.::role:project-aand:role:project-brespectively.YoucanusethenormaluserpermissionstoolsintheWebinterfaceorarangoshtoconfigurethese.
Rolesattribute
ThemostimportantmethodforthisistoreadofftherolesanLDAPuserisassociatedwithfromanattributeintheLDAPuserobject.Iftheconfigurationoption
--ldap.roles-attribute-name
configurationoptionisset,thenthevalueofthatoptionisthenameoftheattributebeingused.
Hereistheexampletoaddtotheoverallconfiguration:
--ldap.roles-attribute-name=role
IfwehavetheuserstoredlikethefollowinginLDAP:
dn:uid=alice,dc=arangodb,dc=com
uid:alice
objectClass:inetOrgPerson
objectClass:organizationalPerson
objectClass:top
objectClass:person
role:project-a
role:project-b
Thentherequestwillgranttherolesproject-aandproject-bfortheuseraliceaftersuccessfulauthentication,astheyarestoredwithintheroleontheuserobject.
Rolessearch
AnalternativemethodforauthorizationistoconductasearchintheLDAPserverforLDAPobjectsrepresentingrolesauserhas.Iftheconfigurationoption
--ldap.roles-search=<search-expression>
LDAPOptions
433
isgiven,thenthestring{USER}in<search-expression>isreplacedwiththedistinguishednameoftheauthenticatedLDAPuserandtheresultingsearchexpressionisusedtomatchdistinguishednamesofLDAPobjectsrepresentingrolesofthatuser.
Example:
--ldap.roles-search'(&(objectClass=groupOfUniqueNames)(uniqueMember={USER}))'
AfteraLDAPuserwasfoundandauthenticatedasdescribedintheauthenticationsectionabovethe{USER}inthesearchexpressionwillbereplacedbyitsdistinguishedname,e.g.uid=alice,dc=arangodb,dc=com,andthuswiththeabovesearchexpressiontheactualsearchexpressionwouldendupbeing:
(&(objectClass=groupOfUniqueNames)(uniqueMember=uid=alice,dc=arangodb,dc=com}))
ThissearchwillfindallobjectsofgroupOfUniqueNameswhereatleastoneuniqueMemberhasthednofalice.Thelistofresultsofthatsearchwouldbethelistofrolesgivenbythevaluesofthednattributesofthefoundroleobjects.
Roletransformationsandfilters
Forbothoftheaboveauthorizationmethodstherearefurtherconfigurationoptionstotunetherolelookup.Inthissectionwedescribethesefurtheroptions:
--ldap.roles-includecanbeusedtospecifyaregularexpressionthatisusedtofilterroles.Onlyrolesthatmatchtheregularexpressionareused.
--ldap.roles-excludecanbeusedtospecifyaregularexpressionthatisusedtofilterroles.Onlyrolesthatdonotmatchtheregularexpressionareused.
--ldap.roles-transformationcanbeusedtospecifyaregularexpressionandreplacementtextas/re/text/.Thisregularexpressionisappliedtotherolenamefound.Thisisespeciallyusefulintheroles-searchvarianttoextracttherealrolenameoutofthednvalue.
--ldap.superuser-rolecanbeusedtospecifytheroleassociatedwiththesuperuser.Anyuserbelongingtothisrolegainssuperuserstatus.Thisroleischeckedafterapplyingtheroles-transformationexpression.
Example:
--ldap.roles-include"^arangodb"
willonlyconsiderrolesthatstartwitharangodb.
--ldap.roles-exclude=disabled
willonlyconsiderrolesthatdocontaintheworddisabled.
--ldap.superuser-role"arangodb-admin"
anyonebelongingtothegroup"arangodb-admin"willbecomeasuperuser.
Theroles-transformationdeservesalargerexample.Assumeweareusingrolessearchandhavestoredrolesinthefollowingway:
dn:cn=project-a,dc=arangodb,dc=com
objectClass:top
objectClass:groupOfUniqueNames
uniqueMember:uid=alice,dc=arangodb,dc=com
uniqueMember:uid=bob,dc=arangodb,dc=com
cn:project-a
description:InternalprojectA
dn:cn=project-b,dc=arangodb,dc=com
objectClass:top
objectClass:groupOfUniqueNames
uniqueMember:uid=alice,dc=arangodb,dc=com
uniqueMember:uid=charlie,dc=arangodb,dc=com
LDAPOptions
434
cn:project-b
description:ExternalprojectB
Inthiscasewewillfindcn=project-a,dc=arangodb,dc=comasoneroleofalice.Howeverweactuallywanttoconfigurearolename::role:project-awhichiseasiertoreadandmaintainforouradministrators.
Ifwenowapplythefollowingtransformation:
--ldap.roles-transformation=/^cn=([^,]*),.*$/$1/
Theregexwillextractoutproject-aresp.project-bofthednattribute.
Incombinationwiththesuperuser-rolewecouldmakeallproject-amembersarangodbadminsbyusing:
--ldap.roles-transformation=/^cn=([^,]*),.*$/$1/\
--ldap.superuser-role=project-a
Completeconfigurationexamples
InthissectionwewouldliketopresentcompleteexamplesforasuccessfulLDAPconfigurationofArangoDB.Allofthefollowingarejustcombinationsofthedetailsdescribedabove.
Simpleauthenticationwithrole-search,usinganonymousLDAPuser
ThisexampleconnectstotheLDAPserverwithananonymousread-onlyuser.Weusethesimpleauthenticationmode(prefix+suffix)toauthenticateusersandapplyarolesearchforgroupOfUniqueNamesobjectswheretheuserisauniqueMember.Furthermoreweextractonlythecnoutofthedistinguishedrolename.
--ldap.enabled=true\
--ldap.server=ldap.arangodb.com\
--ldap.basedn=dc=arangodb,dc=com\
--ldap.prefixuid=\
--ldap.suffix,dc=arangodb,dc=com\
--ldap.roles-search'(&(objectClass=groupOfUniqueNames)(uniqueMember={USER}))'\
--ldap.roles-transformation=/^cn=([^,]*),.*$/$1/\
--ldap.superuser-role=project-a
SearchauthenticationwithrolesattributeusingLDAPadminuserhavingTLSenabled
ThisexampleconnectstotheLDAPserverwithagivendistinguishednameofanadminuser+password.FurthermoreweactivateTLSandgivethecertificatefiletovalidateserverresponses.Weusethesearchauthenticationsearchingfortheuidattributeofpersonobjects.Thesepersonobjectshaveroleattribute(s)containingtherole(s)ofauser.
--ldap.enabled=true\
--ldap.server=ldap.arangodb.com\
--ldap.basedn=dc=arangodb,dc=com\
--ldap.binddn=uid=arangoadmin,dc=arangodb,dc=com\
--ldap.bindpasswd=supersecretpassword\
--ldap.tlstrue\
--ldap.tls-cacert-file/path/to/certificate.pem\
--ldap.search-attribute=uid\
--ldap.search-filter=objectClass=person\
--ldap.roles-attribute-name=role
LDAPOptions
435
Command-LineOptionsforLogging
Loglevelsandtopics
ArangoDB'slogoutputisgroupedintotopics.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.Thelogverbosityandoutputfilescanbeadjustedperlogtopic.Forexample
--log.levelstartup=trace--log.levelqueries=trace--log.levelinfo
willlogmessagesconcerningstartupattracelevel,AQLqueriesattracelevelandeverythingelseatinfolevel.
Inaconfigurationfile,itiswrittenlikethis:
[log]
level=startup=trace
level=queries=trace
level=info
Notethattheremustnotbeanywhitespacearoundthesecond=.
Theavailableloglevelsare:
fatal:onlylogsfatalerrorserror:onlylogserrorswarning:onlylogswarningsanderrorsinfo:logsinformationmessages,warningsanderrorsdebug:logsdebugandinformationmessages,warningsanderrorstrace:logstrace,debugandinformationmessages,warningsanderrors
Notethatlevelsdebugandtracewillbeveryverbose.
SomerelevantlogtopicsavailableinArangoDB3are:
agency:informationabouttheagencycollector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperations(includingmsync)performance:performance-releatedmessagesqueries:executedAQLqueries,slowqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads
Logoutputs
Thelogoption--log.output<definition>allowsdirectingtheglobalorper-topiclogoutputtodifferentoutputs.Theoutputdefinition<definition>canbeoneof
-forstdin+forstderrsyslog://<syslog-facility>
syslog://<syslog-facility>/<application-name>
file://<relative-path>
Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputfordifferentlogtopics.Tosetupaper-topicoutputconfiguration,use--log.output<topic>=<definition>,e.g.
LoggingOptions
436
queries=file://queries.txt
logsallqueriestothefile"queries.txt".
Theoldoption--log.fileisstillavailablein3.0forconveniencereasons.In3.0itisashortcutforthemoregeneraloption--log.outputfile://filename.
Theoldoption--log.requests-fileisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.outputrequests=file://....
Using--log.outputalsoallowsdirectinglogoutputtodifferentfilesbasedontopics.Forexample,tologallAQLqueriestoafile"queries.log"onecanusetheoptions:
--log.levelqueries=trace--log.outputqueries=file:///path/to/queries.log
ToadditionallylogHTTPrequesttoafilenamed"requests.log"addtheoptions:
--log.levelrequests=info--log.outputrequests=file:///path/to/requests.log
Forcingdirectoutput
Theoption--log.force-directcanbeusedtodisablelogginginanextraloggingthread.Ifsettotrue,anylogmessagesareimmediatelyprintedinthethreadthattriggeredthelogmessage.Thisisnon-optimalforperformancebutcanaiddebugging.Ifsettofalse,logmessagesarehandedofftoanextraloggingthread,whichasynchronouslywritesthelogmessages.
Localtime
Logdatesandtimesinlocaltimezone:--log.use-local-time
Ifspecified,alldatesandtimesinlogmessageswillusetheserver'slocaltime-zone.Ifnotspecified,alldatesandtimesinlogmessageswillbeprintedinUTC/Zulutime.ThedateandtimeformatusedinlogsisalwaysYYYY-MM-DDHH:MM:SS,regardlessofthissetting.IfUTCtimeisused,aZwillbeappendedtoindicateZulutime.
Colorlogging--log.colorvalue
Loggingtoterminaloutputisbydefaultcolored.Colorfulloggingcanbeturnedoffbysettingthevaluetofalse.
SourcefileandLinenumber
Loglinenumber:--log.line-number
Normally,ifanhumanreadablefatal,error,warningorinfomessageislogged,noinformationaboutthefileandlinenumberisprovided.Thefileandlinenumberisonlyloggedfordebugandtracemessage.Thisoptioncanbeusetoalwayslogthesepiecesofinformation.
Prefix
Logprefix:--log.prefixprefix
Thisoptionisusedspecifyanprefixtologgedtext.
Threads
Logthreadidentifier:--log.threadtrue
Wheneverlogoutputisgenerated,theprocessIDiswrittenaspartoftheloginformation.Settingthisoptionappendsthethreadidofthecallingthreadtotheprocessid.Forexample,
2010-09-20T13:04:01Z[19355]INFOreadyforbusiness
LoggingOptions
437
whennothreadisloggedand
2010-09-20T13:04:17Z[19371-18446744072487317056]readyforbusiness
whenthiscommandlineoptionisset.
Toalsologthreadnames,itispossibletosetthe--log.thread-nameoption.Bydefault--log.thread-nameissettofalse.
Role
Logrole:--log.roletrue
Whensettotrue,thisoptionwillmaketheArangoDBloggerprintasinglecharacterwiththeserver'sroleintoeachloggedmessage.Therolesare:
U:undefined/unclear(usedatstartup)S:singleserverC:coordinatorP:primaryA:agent
Thedefaultvalueforthisoptionisfalse,sonoroleswillbelogged.
LoggingOptions
438
GeneralOptions
DatabaseUpgrade
--database.auto-upgrade
Specifyingthisoptionwillmaketheserverperformadatabaseupgradeinsteadofstartingtheservernormally.AdatabaseupgradewillfirstcomparetheversionnumberstoredinthefileVERSIONinthedatabasedirectorywiththecurrentserverversion.
Iftheversionnumberfoundinthedatabasedirectoryishigherthantheversionnumbertheserverisrunning,theserverexpectsthisisanunintentionaldowngradeandwillwarnaboutthis.Usingtheserverintheseconditionsisneitherrecommendednorsupported.
Iftheversionnumberfoundinthedatabasedirectoryislowerthantheversionnumbertheserverisrunning,theserverwillcheckwhetherthereareanyupgradetaskstoperform.Itwillthenexecuteallrequiredupgradetasksandprinttheirstatuses.Ifoneoftheupgradetasksfails,theserverwillexitwithanerror.Re-startingtheserverwiththeupgradeoptionwillthenagaintriggertheupgradecheckandexecutionuntiltheproblemisfixed.
Whetherornotthisoptionisspecified,theserverwillalwaysperformaversioncheckonstartup.Runningtheserverwithanon-matchingversionnumberintheVERSIONfilewillmaketheserverrefusetostart.
StorageEngine
AsofArangoDB3.2twostorageenginesaresupported.The"traditional"engineiscalledMMFiles,whichisalsothedefaultstorageengine.
AnalternativeenginebasedonRocksDBisalsoprovidedandcanbeturnedonmanually.
Onestorageenginetypeissupportedperserverperinstallation.Liveswitchingofstorageenginesonalreadyinstalledsystemsisn'tsupported.Configuringthewrongengine(notmatchingthepreviouslyusedone)willresultintheserverrefusingtostart.YoumayhoweveruseautotoletArangoDBchoosethepreviouslyusedone.
--server.storage-engine[auto|mmfiles|rocksdb]
Daemon
--daemon
Runstheserverasadaemon(asabackgroundprocess).Thisparametercanonlybesetifthepid(processid)fileisspecified.Thatis,unlessavaluetotheparameterpid-fileisgiven,thentheserverwillreportanerrorandexit.
DefaultLanguage--default-languagedefault-language
Thedefaultlanguageistusedforsortingandcomparingstrings.Thelanguagevalueisatwo-letterlanguagecode(ISO-639)oritiscomposedbyatwo-letterlanguagecodewithandatwolettercountrycode(ISO-3166).Validlanguagesare"de","en","en_US"or"en_UK".
Thedefaultdefault-languageissettobethesystemlocaleonthatplatform.
Supervisor
--supervisor
Executestheserverinsupervisormode.Intheeventthattheserverunexpectedlyterminatesduetoaninternalerror,thesupervisorwillautomaticallyrestarttheserver.Settingthisflagautomaticallyimpliesthattheserverwillrunasadaemon.Notethat,aswiththedaemonflag,thisflagrequiresthatthepid-fileparameterwillset.
unix>./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/
2012-06-27T15:58:28Z[10133]INFOstartingupinsupervisormode
Ascanbeseen(e.g.byexecutingthepscommand),thiswillstartasupervisorprocessandtheactualdatabaseprocess:
GeneralOptions
439
unix>psfax|greparangod
10137?Ssl0:00./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/
10142?Sl0:00\_./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/
Whenthedatabaseprocessterminatesunexpectedly,thesupervisorprocesswillstartupanewdatabaseprocess:
>kill-SIGSEGV10142
>psfax|greparangod
10137?Ssl0:00./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/
10168?Sl0:00\_./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/
Useridentity
--uiduid
Thename(identity)oftheusertheserverwillrunas.Ifthisparameterisnotspecified,theserverwillnotattempttochangeitsUID,sothattheUIDusedbytheserverwillbethesameastheUIDoftheuserwhostartedtheserver.Ifthisparameterisspecified,thentheserverwillchangeitsUIDafteropeningportsandreadingconfigurationfiles,butbeforeacceptingconnectionsoropeningotherfiles(suchasrecoveryfiles).Thisisusefulwhentheservermustbestartedwithraisedprivileges(incertainenvironments)butsecurityconsiderationsrequirethattheseprivilegesbedroppedoncetheserverhasstartedwork.
Observethatthisparametercannotbeusedtobypassoperatingsystemsecurity.Ingeneral,thisparameter(anditscorrespondingrelativegid)canlowerprivilegesbutnotraisethem.
Groupidentity
--gidgid
Thename(identity)ofthegrouptheserverwillrunas.Ifthisparameterisnotspecified,thentheserverwillnotattempttochangeitsGID,sothattheGIDtheserverrunsaswillbetheprimarygroupoftheuserwhostartedtheserver.Ifthisparameterisspecified,thentheserverwillchangeitsGIDafteropeningportsandreadingconfigurationfiles,butbeforeacceptingconnectionsoropeningotherfiles(suchasrecoveryfiles).
Thisparameterisrelatedtotheparameteruid.
Processidentity
--pid-filefilename
ThenameoftheprocessIDfiletousewhenrunningtheserverasadaemon.Thisparametermustbespecifiedifeithertheflagdaemonorsupervisorisset.
Checkmaxmemorymappings
--server.check-max-memory-mappingscanbeusedonLinuxtomakearangodcheckthenumberofmemorymappingscurrentlyusedbytheprocess(asreportedin/proc//maps)andcompareitwiththemaximumnumberofallowedmappingsasdeterminedby/proc/sys/vm/max_map_count.Ifthecurrentnumberofmemorymappingsgetsnearthemaximumallowedvalue,arangodwilllogawarninganddisallowthecreationoffurtherV8contextstemporarilyuntilthecurrentnumberofmappingsgoesdownagain.
Iftheoptionissettofalse,nosuchcheckswillbeperformed.Allnon-Linuxoperatingsystemsdonotprovidethisoptionandwillignoreit.
Console--console
Runstheserverinanexclusiveemergencyconsolemode.Whenstartingtheserverwiththisoption,theserverisstartedwithaninteractiveJavaScriptemergencyconsole,withallnetworkingandHTTPinterfacesoftheserverdisabled.
Norequestscanbemadetotheserverinthismode,andtheonlywaytoworkwiththeserverinthismodeisbyusingtheemergencyconsole.Notethattheservercannotbestartedinthismodeifitisalreadyrunninginthisoranothermode.
GeneralOptions
440
RandomGenerator
--random.generatorarg
Theargumentisaninteger(1,2,3or4)whichsetsthemannerinwhichrandomnumbersaregenerated.Thedefaultmethod(3)istousetheanon-blockingrandom(orpseudorandom)numbergeneratorsuppliedbytheoperatingsystem.
Specifyinganargumentof2,usesablockingrandom(orpseudorandom)numbergenerator.Specifyinganargument1setsapseudorandomnumbergeneratorusinganimplicationoftheMersenneTwisterMT19937algorithm.Algorithm4isacombinationoftheblockingrandomnumbergeneratorandtheMersenneTwister.
Enable/disableauthentication
--server.authentication
Settingthisoptiontofalsewillturnoffauthenticationontheserversidesoallclientscanexecuteanyactionwithoutauthorizationandprivilegechecks.
Thedefaultvalueistrue.
JWTSecret
--server.jwt-secretsecret
ArangoDBwilluseJWTstoauthenticaterequests.UsingthisoptionletsyouspecifyaJWT.
InsingleserversetupsandwhennotspecifyingthissecretArangoDBwillgenerateasecret.
Inclusterdeploymentswhichhaveauthenticationenabledasecretmustbesetconsistentlyacrossallclustertaskssotheycantalktoeachother.
Enable/disableauthenticationforUNIXdomainsockets
--server.authentication-unix-socketsvalue
SettingvaluetotruewillturnoffauthenticationontheserversideforrequestscominginviaUNIXdomainsockets.Withthisflagenabled,clientslocatedonthesamehostastheArangoDBservercanuseUNIXdomainsocketstoconnecttotheserverwithoutauthentication.Requestscominginbyothermeans(e.g.TCP/IP)arenotaffectedbythisoption.
Thedefaultvalueisfalse.
Note:thisoptionisonlyavailableonplatformsthatsupportUNIXdomainsockets.
Enable/disableauthenticationforsystemAPIrequestsonly--server.authentication-system-onlyboolean
ControlswhetherincomingrequestsneedauthenticationonlyiftheyaredirectedtotheArangoDB'sinternalAPIsandfeatures,locatedat/_api/,/_admin/etc.
Iftheflagissettotrue,thenHTTPauthenticationisonlyrequiredforrequestsgoingtoURLsstartingwith/_,butnotforotherURLs.Theflagcanthusbeusedtoexposeauser-madeAPIwithoutHTTPauthenticationtotheoutsideworld,buttopreventtheoutsideworldfromusingtheArangoDBAPIandtheadmininterfacewithoutauthentication.NotethatcheckingtheURLisperformedafteranydatabasenameprefixhasbeenremoved.ThatmeanswhentheactualURLcalledis/_db/_system/myapp/myaction,theURL/myapp/myactionwillbeusedforauthentication-system-onlycheck.
Thedefaultistrue.
NotethatauthenticationstillneedstobeenabledfortheserverregularlyinorderforHTTPauthenticationtobeforcedfortheArangoDBAPIandthewebinterface.Settingonlythisflagisnotenough.
YoucancontrolArangoDB'sgeneralauthenticationfeaturewiththe--server.authenticationflag.
Enableauthenticationcachetimeout--server.authentication-timeoutvalue
GeneralOptions
441
Setsthecachetimeouttovalue(inseconds).ThisisonlynecessaryifyouuseanexternalauthenticationsystemlikeLDAP.
Enablelocalauthentication
--server.local-authenticationvalue
Ifsettofalseonlyusetheexternalauthenticationsystem.Iftruealsousethelocal_userscollections.
Thedefaultvalueistrue.
Enable/disablereplicationapplier--database.replication-applierflag
Iffalsetheserverwillstartwithreplicationappliersturnedoff,evenifthereplicationappliersareconfiguredwiththeautoStartoption.Usingthecommand-lineoptionwillnotchangethevalueoftheautoStartoptionintheapplierconfiguration,butwillsuppressauto-startingthereplicationapplierjustonce.
Iftheoptionisnotused,ArangoDBwillreadtheapplierconfigurationfromthefileREPLICATION-APPLIER-CONFIGonstartup,andusethevalueoftheautoStartattributefromthisfile.
Thedefaultistrue.
Keep-alivetimeout
--http.keep-alive-timeout
AllowstospecifythetimeoutforHTTPkeep-aliveconnections.Thetimeoutvaluemustbespecifiedinseconds.Idlekeep-aliveconnectionswillbeclosedbytheserverautomaticallywhenthetimeoutisreached.Akeep-alive-timeoutvalue0willdisablethekeepalivefeatureentirely.
HideProductheader
--http.hide-product-header
Iftrue,theserverwillexcludetheHTTPheader"Server:ArangoDB"inHTTPresponses.Ifsettofalse,theserverwillsendtheheaderinresponses.
Thedefaultisfalse.
Allowmethodoverride
--http.allow-method-override
Whenthisoptionissettotrue,theHTTPrequestmethodwilloptionallybefetchedfromoneofthefollowingHTTPrequestheadersifpresentintherequest:
x-http-methodx-http-method-overridex-method-override
Iftheoptionissettotrueandanyoftheseheadersisset,therequestmethodwillbeoverriddenbythevalueoftheheader.Forexample,thisallowsissuinganHTTPDELETErequestwhichtotheoutsideworldwilllooklikeanHTTPGETrequest.Thisallowsbypassingproxiesandtoolsthatwillonlyletcertainrequesttypespass.
Settingthisoptiontotruemayimposeasecurityrisksoitshouldonlybeusedincontrolledenvironments.
Thedefaultvalueforthisoptionisfalse.
Serverthreads
--server.threadsnumber
Specifiesthenumberofthreadsthatarespawnedtohandlerequests.
GeneralOptions
442
Togglingserverstatistics
--server.statisticsvalue
Ifthisoptionisvalueisfalse,thenArangoDB'sstatisticsgatheringisturnedoff.StatisticsgatheringcausesregularCPUactivitysousingthisoptiontoturnitoffmightrelieveheavy-loadedinstancesabit.
Sessiontimeout
timetoliveforserversessions--server.session-timeoutvalue
Thetimeoutforwebinterfacesessions,usingforauthenticatingrequeststothewebinterface(/_admin/aardvark)andrelatedareas.
Sessionsareonlyusedwhenauthenticationisturnedon.
Foxxqueues
enableordisabletheFoxxqueuesfeature--foxx.queuesflag
Iftrue,theFoxxqueueswillbeavailableandjobsinthequeueswillbeexecutedasynchronously.
Thedefaultistrue.Whensettofalsethequeuemanagerwillbedisabledandanyjobsarepreventedfrombeingprocessed,whichmayreduceCPUloadabit.
Foxxqueuespollinterval
pollintervalforFoxxqueues--foxx.queues-poll-intervalvalue
ThepollintervalfortheFoxxqueuesmanager.Thevalueisspecifiedinseconds.LowervalueswillmeanmoreimmediateandmorefrequentFoxxqueuejobexecution,butwillmakethequeuethreadwakeupandquerythequeuesmoreoften.Whensettoalowvalue,thequeuethreadmightcauseCPUload.
Thedefaultis1second.IfFoxxqueuesarenotusedmuch,thenthisvaluemaybeincreasedtomakethequeuesthreadwakeupless.
Directory
--database.directorydirectory
Thedirectorycontainingthecollectionsanddatafiles.Defaultsto/var/lib/arango.Whenspecifyingthedatabasedirectory,pleasemakesurethedirectoryisactuallywritablebythearangodprocess.
YoushouldfurthernotuseadatabasedirectorywhichisprovidedbyanetworkfilesystemsuchasNFS.Thereasonisthatnetworkedfilesystemsmightcauseinconsistencieswhentherearemultipleparallelreadersorwritersortheylackfeaturesrequiredbyarangod(e.g.flock()).
directory
Whenusingthecommandlineversion,youcansimplysupplythedatabasedirectoryasargument.
Examples
>./arangod--server.endpointtcp://127.0.0.1:8529--database.directory
/tmp/vocbase
Databasedirectorystateprecondition
--database.require-directory-statestate
Usingthisoptionitispossibletorequirethedatabasedirectorytobeinaspecificstateonstartup.theoptionsforthisvalueare:
non-existing:databasedirectorymustnotexistexisting:databasedirectorymustexistempty:databasedirectorymustexistbutbeemptypopulated:databasedirectorymustexistandcontainspecificfilesalreadyany:anydirectorystateallowed
GeneralOptions
443
Journalsize
--database.maximal-journal-sizesize
Maximalsizeofjournalinbytes.Canbeoverwrittenwhencreatinganewcollection.Notethatthisalsolimitsthemaximalsizeofasingledocument.
Thedefaultis32MB.
Waitforsync
defaultwaitforsyncbehavior--database.wait-for-syncboolean
Defaultwait-for-syncvalue.Canbeoverwrittenwhencreatinganewcollection.
Thedefaultisfalse.
Forcesyncingofproperties
forcesyncingofcollectionpropertiestodisk--database.force-sync-propertiesboolean
Forcesyncingofcollectionpropertiestodiskaftercreatingacollectionorupdatingitsproperties.
Ifturnedoff,nofsyncwillhappenforthecollectionanddatabasepropertiesstoredinparameter.jsonfilesinthefilesystem.Turningoffthisoptionwillspeedupworkloadsthatcreateanddropalotofcollections(e.g.testsuites).
Thedefaultistrue.
LimitingmemoryforAQLqueries
--query.memory-limitvalue
Thedefaultmaximumamountofmemory(inbytes)thatasingleAQLquerycanuse.WhenasingleAQLqueryreachesthespecifiedlimitvalue,thequerywillbeabortedwitharesourcelimitexceededexception.Inacluster,thememoryaccountingisdonepershard,sothelimitvalueiseffectivelyamemorylimitperquerypershard.
ThegloballimitvaluecanbeoverridenperquerybysettingthememoryLimitoptionvalueforindividualquerieswhenrunninganAQLquery.
Thedefaultvalueis0,meaningthatthereisnomemorylimit.
TurningAQLwarningsintoerrors--query.fail-on-warningvalue
Whensettotrue,AQLqueriesthatproducewarningswillinstantlyabortandthrowanexception.ThisoptioncanbesettocatchobviousissueswithAQLqueriesearly.Whensettofalse,AQLqueriesthatproducewarningswillnotabortandreturnthewarningsalongwiththequeryresults.TheoptioncanalsobeoverriddenforeachindividualAQLquery.
Enable/disableAQLquerytracking
--query.trackingflag
Iftrue,theserver'sAQLslowquerytrackingfeaturewillbeenabledbydefault.Trackingofqueriescanbedisabledbysettingtheoptiontofalse.
Thedefaultistrue.
Enable/disabletrackingofbindvariablesinAQLqueries--query.tracking-with-bindvarsflag
Iftrue,thenthebindvariableswillbetrackedforallrunningandslowAQLqueries.Thisoptiononlyhasaneffectif--query.trackingwassettotrue.Trackingofbindvariablescanbedisabledbysettingtheoptiontofalse.
Thedefaultistrue.
GeneralOptions
444
ThresholdforslowAQLqueries
--query.slow-thresholdvalue
BysettingvalueitcanbecontrolledafterwhatexecutiontimeanAQLqueryisconsidered"slow".Anyslowqueriesthatexceedtheexecutiontimespecifiedinvaluewillbeloggedwhentheyarefinished.Thethresholdvalueisspecifiedinseconds.Trackingofslowqueriescanbeturnedoffentirelybysettingtheoption--query.trackingtofalse.
Thedefaultvalueis10.0.
Queryregistrytimeout
--query.registry-ttlvalue
ThedefaulttimeoutforAQLquerypartstostayaliveinthecluster.Thedefaultvalueis600seconds.Querypartsthatarenotusedfortheconfiguredamountoftimewillexpireautomaticallyandwillbeaborted.Thevalueofthisoptionnormallyonlyneedstobeincreasedforqueriesthatarerunninglongerthanthedefaulttimeoutvalue(600seconds)andthattimeout.Theoptionhasnoeffectinsingle-servermode.
Throwcollectionnotloadederror--database.throw-collection-not-loaded-errorflag
Accessinganot-yetloadedcollectionwillautomaticallyloadacollectiononfirstaccess.Thisflagcontrolswhathappensincaseanoperationwouldneedtowaitforanotherthreadtofinalizeloadingacollection.Ifsettotrue,thenthefirstoperationthataccessesanunloadedcollectionwillloadit.Furtherthreadsthattrytoaccessthesamecollectionwhileitisstillloadingwillgetanerror(1238,collectionnotloaded).Whentheinitialoperationhascompletedloadingthecollection,alloperationsonthecollectioncanbecarriedoutnormally,anderror1238willnotbethrown.
Ifsettofalse,thefirstthreadthataccessesanot-yetloadedcollectionwillstillloadit.Otherthreadsthattrytoaccessthecollectionwhileloadingwillnotfailwitherror1238butinsteadblockuntilthecollectionisfullyloaded.Thisconfigurationmightleadtoallserverthreadsbeingblockedbecausetheyareallwaitingforthesamecollectiontocompleteloading.Settingtheoptiontotruewillpreventthisfromhappening,butrequiresclientstocatcherror1238andreactonit(maybebyschedulingaretryforlater).
Thedefaultvalueisfalse.
AQLQuerycachingmode--query.cache-mode
TogglestheAQLquerycachebehavior.Possiblevaluesare:
off:donotusequerycacheon:alwaysusequerycache,exceptforqueriesthathavetheircacheattributesettofalsedemand:usequerycacheonlyforqueriesthathavetheircacheattributesettotrue
AQLQuerycachesize--query.cache-entries
Maximumnumberofqueryresultsthatcanbestoredperdatabase-specificquerycache.Ifaqueryiseligibleforcachingandthenumberofitemsinthedatabase'squerycacheisequaltothisthresholdvalue,anothercachedqueryresultwillberemovedfromthecache.
Thisoptiononlyhasaneffectifthequerycachemodeissettoeitheronordemand.
JavaScriptcodeexecution--javascript.allow-admin-execute
Thisoptioncanbeusedtocontrolwhetheruser-definedJavaScriptcodeisallowedtobeexecutedonserverbysendingviaHTTPtotheAPIendpoint/_admin/executewithanauthenticateduseraccount.Thedefaultvalueisfalse,whichdisablestheexecutionofuser-definedcode.Thisisalsotherecommendedsettingforproduction.Intestenvironments,itmaybeconvenienttoturntheoptiononinordertosendarbitrarysetuporteardowncommandsforexecutionontheserver.
GeneralOptions
445
V8contexts
--javascript.v8-contextsnumber
SpecifiesthemaximumnumberofV8contextsthatarecreatedforexecutingJavaScriptcode.MorecontextsallowexecutingmoreJavaScriptactionsinparallel,providedthattherearealsoenoughthreadsavailable.PleasenotethateachV8contextwilluseasubstantialamountofmemoryandrequiresperiodicCPUprocessingtimeforgarbagecollection.
NotethatthisvalueconfiguresthemaximumnumberofV8contextsthatcanbeusedinparallel.UponserverstartonlyasmanyV8contextswillbecreatedasareconfiguredinoption--javascript.v8-contexts-minimum.TheactualnumberofavailableV8contextsmayfloatatruntimebetween--javascript.v8-contexts-minimumand--javascript.v8-contexts.WhenthereareunusedV8contextsthatlingeraround,theserver'sgarbagecollectorthreadwillautomaticallydeletethem.
--javascript.v8-contexts-minimumnumber
SpecifiestheminimumnumberofV8contextsthatwillbepresentatanytimetheserverisrunning.TheactualnumberofV8contextswillneverdropbelowthisvalue,butitmaygoupashighasspecifiedviatheoption--javascript.v8-contexts.
WhenthereareunusedV8contextsthatlingeraroundandthenumberofV8contextsisgreaterthan--javascript.v8-contexts-minimumtheserver'sgarbagecollectorthreadwillautomaticallydeletethem.
--javascript.v8-contexts-max-invocations
SpecifiesthemaximumnumberofinvocationsafterwhichausedV8contextisdisposed.Thedefaultvalueof--javascript.v8-contexts-max-invocationsis0,meaningthatthemaximumnumberofinvocationspercontextisunlimited.
--javascript.v8-contexts-max-age
Specifiesthetimeduration(inseconds)afterwhichtimeaV8contextisdisposedautomaticallyafteritscreation.Ifthetimeiselapsed,thecontextwillbedisposed.Thedefaultvaluefor--javascript.v8-contexts-max-ageis60seconds.
Ifboth--javascript.v8-contexts-max-invocationsand--javascript.v8-contexts-max-ageareset,thenthecontextwillbedestroyedwheneitherofthespecifiedthresholdvaluesisreached.
Garbagecollectionfrequency(time-based)
--javascript.gc-frequencyfrequency
Specifiesthefrequency(inseconds)fortheautomaticgarbagecollectionofJavaScriptobjects.Thissettingisusefultohavethegarbagecollectionstillworkinperiodswithnoorlittlenumbersofrequests.
Garbagecollectioninterval(request-based)--javascript.gc-intervalinterval
Specifiestheinterval(approximatelyinnumberofrequests)thatthegarbagecollectionforJavaScriptobjectswillberunineachthread.
V8options
--javascript.v8-optionsoptions
OptionalargumentstopasstotheV8Javascriptengine.TheV8enginewillrunwithdefaultsettingsunlessexplicitoptionsarespecifiedusingthisoption.TheoptionspassedwillbeforwardedtotheV8enginewhichwillparsethemonitsown.Passinginvalidoptionsmayresultinanerrorbeingprintedonstderrandtheoptionbeingignored.
Optionsneedtobepassedinonestring,withV8optionnamesbeingprefixedwithdoubledashes.Multipleoptionsneedtobeseparatedbywhitespace.TogetalistofallavailableV8options,youcanusethevalue"--help" asfollows:
--javascript.v8-options="--help"
AnotherexampleofspecificV8optionsbeingsetatstartup:
--javascript.v8-options="--log"
GeneralOptions
446
NamesandfeaturesorusableoptionsdependontheversionofV8beingused,andmightchangeinthefutureifadifferentversionofV8isbeingusedinArangoDB.NotalloptionsofferedbyV8mightbesensibletouseinthecontextofArangoDB.Usethespecificoptionsonlyifyouaresurethattheyarenotharmfulfortheregulardatabaseoperation.
GeneralOptions
447
MMFilesWrite-aheadlogoptionsSinceArangoDB2.2,theMMFilesstorageenginewillwritealldata-modificationoperationsintoitswrite-aheadlog.
WithArangoDB3.2anotherStorageengineoptionbecomesavailable-RocksDB.IncaseofusingRocksDBmostofthesubsequentoptionsdon'thaveausefulmeaning.
Thewrite-aheadlogisasequenceoflogfilesthatarewritteninanappend-onlyfashion.Fulllogfileswilleventuallybegarbage-collected,andtherelevantdatamightbetransferredintocollectionjournalsanddatafiles.Unneededandalreadygarbage-collectedlogfileswilleitherbedeletedorkeptforthepurposeofkeepingareplicationbacklog.
Directory
TheWALlogfilesdirectory:--wal.directory
Specifiesthedirectoryinwhichthewrite-aheadlogfilesshouldbestored.Ifthisoptionisnotspecified,itdefaultstothesubdirectoryjournalsintheserver'sglobaldatabasedirectory.Ifthedirectoryisnotpresent,itwillbecreated.
Logfilesize
thesizeofeachWALlogfile--wal.logfile-size
Specifiesthefilesize(inbytes)foreachwrite-aheadlogfile.Thelogfilesizeshouldbechosensothateachlogfilecanstoreaconsiderableamountofdocuments.Thebiggerthelogfilesizeischosen,thelongeritwilltaketofillupasinglelogfile,whichalsoinfluencesthedelayuntilthedatainalogfilewillbegarbage-collectedandwrittentocollectionjournalsanddatafiles.Italsoaffectshowlonglogfilerecoverywilltakeatserverstart.
Allowoversizeentries
whetherornotoversizeentriesareallowed--wal.allow-oversize-entries
Whetherornotitisallowedtostoreindividualdocumentsthatarebiggerthanwouldfitintoasinglelogfile.Settingtheoptiontofalsewillmakesuchoperationsfailwithanerror.Settingtheoptiontotruewillmakesuchoperationssucceed,butwithahighpotentialperformanceimpact.Thereasonisthatforeachoversizeoperation,anindividualoversizelogfileneedstobecreatedwhichmayalsoblockotheroperations.Theoptionshouldbesettofalseifitiscertainthatdocumentswillalwayshaveasizesmallerthanasinglelogfile.
Numberofreservelogfiles
maximumnumberofreservelogfiles--wal.reserve-logfiles
ThemaximumnumberofreservelogfilesthatArangoDBwillcreateinabackgroundprocess.Reservelogfilesareusefulinthesituationwhenanoperationneedstobewrittentoalogfilebutthereservespaceinthelogfileistoolowforstoringtheoperation.Inthiscase,anewlogfileneedstobecreatedtostoretheoperation.Creatingnewlogfilesisnormallyslow,soArangoDBwilltrytopre-createlogfilesinabackgroundprocesssotherearealwaysreservelogfileswhentheactivelogfilegetsfull.ThenumberofreservelogfilesthatArangoDBkeepsinthebackgroundisconfigurablewiththisoption.
Numberofhistoriclogfiles
maximumnumberofhistoriclogfiles--wal.historic-logfiles
ThemaximumnumberofhistoriclogfilesthatArangoDBwillkeepaftertheyhavebeengarbage-collected.Ifnoreplicationisused,thereisnoneedtokeephistoriclogfilesexceptforhavingalocalchangelog.
Inareplicationsetup,thenumberofhistoriclogfilesaffectstheamountofdataaslavecanfetchfromthemaster'slogs.Themorehistoriclogfiles,themorehistoricdataisavailableforaslave,whichisusefuliftheconnectionbetweenmasterandslaveisunstableorslow.Nothavingenoughhistoriclogfilesavailablemightleadtologfiledatabeingdeletedonthemasteralreadybeforeaslavehasfetchedit.
Syncinterval
Write-AheadLogOptions
448
intervalforautomatic,non-requesteddisksyncs--wal.sync-interval
Theinterval(inmilliseconds)thatArangoDBwillusetoautomaticallysynchronizedatainitswrite-aheadlogstodisk.Automaticsyncswillonlybeperformedfornot-yetsynchronizeddata,andonlyforoperationsthathavebeenexecutedwithoutthewaitForSyncattribute.
Flushtimeout
WALflushtimeout`--wal.flush-timeout
Thetimeout(inmilliseconds)thatArangoDBwillatmostwaitwhenflushingafullWALlogfiletodisk.Whenthetimeoutisreachedandtheflushisnotcompleted,theoperationthatrequestedtheflushwillfailwithalocktimeouterror.
Throttling
ThrottlewritestoWALwhenatleastsuchmanyoperationsarewaitingforgarbagecollection:--wal.throttle-when-pending
Themaximumvalueforthenumberofwrite-aheadloggarbage-collectionqueueelements.Ifsetto0,thequeuesizeisunbounded,andnowrite-throttlingwilloccur.Ifsettoanon-zerovalue,write-throttlingwillautomaticallykickinwhenthegarbage-collectionqueuecontainsatleastasmanyelementsasspecifiedbythisoption.Whilewrite-throttlingisactive,data-modificationoperationswillintentionallybedelayedbyaconfigurableamountoftime.Thisistoensurethewrite-aheadloggarbagecollectorcancatchupwiththeoperationsexecuted.Write-throttlingwillstayactiveuntilthegarbage-collectionqueuesizegoesdownbelowthespecifiedvalue.Write-throttlingisturnedoffbydefault.
--wal.throttle-wait
Thisoptiondeterminesthemaximumwaittime(inmilliseconds)foroperationsthatarewrite-throttled.Ifwrite-throttlingisactiveandanewwriteoperationistobeexecuted,itwillwaitforatmostthespecifiedamountoftimeforthewrite-aheadloggarbage-collectionqueuesizetofallbelowthethrottlingthreshold.Ifthequeuesizedecreasesbeforethemaximumwaittimeisover,theoperationwillbeexecutednormally.Ifthequeuesizedoesnotdecreasebeforethewaittimeisover,theoperationwillbeabortedwithanerror.Thisoptiononlyhasaneffectif--wal.throttle-when-pendinghasanon-zerovalue,whichisnotthedefault.
Numberofslots
Maximumnumberofslotstobeusedinparallel:--wal.slots
Configurestheamountofwriteslotsthewrite-aheadlogcangivetowriteoperationsinparallel.Anywriteoperationwillleaseaslotandreturnittothewrite-aheadlogwhenitisfinishedwritingthedata.Aslotwillremainblockeduntilthedatainitwassynchronizedtodisk.Afterthat,aslotbecomesreusablebyfollowingoperations.Therequirednumberofslotsisthusdeterminedbytheparallelityofwriteoperationsandthedisksynchronizationspeed.Slowdisksprobablyneedhighervalues,andfastdisksmayonlyrequireavaluelowerthanthedefault.
Ignorelogfileerrors
Ignorelogfileerrorswhenopeninglogfiles:--wal.ignore-logfile-errors
Ignoresanyrecoveryerrorscausedbycorruptedlogfilesonstartup.Whensettofalse,therecoveryprocedureonstartupwillfailwithanerrorwheneveritencountersacorrupted(thatincludesonlyhalf-written)logfile.Thisisasecurityprecautiontopreventdatalossincaseofdiskerrorsetc.Whentherecoveryprocedureabortsbecauseofcorruption,anycorruptedfilescanbeinspectedandfixed(orremoved)manuallyandtheservercanberestartedafterwards.
Settingtheoptiontotruewillmaketheservercontinuewiththerecoveryprocedureevenincaseitdetectscorruptlogfileentries.Inthiscaseitwillstopatthefirstcorruptedlogfileentryandignoreallothers,whichmightcausedataloss.
Ignorerecoveryerrors
Ignorerecoveryerrors:--wal.ignore-recovery-errors
Ignoresanyrecoveryerrorsnotcausedbycorruptedlogfilesbutbylogicalerrors.Logicalerrorscanoccuriflogfilesoranyotherserverdatafileshavebeenmanuallyeditedortheserverissomehowmisconfigured.
Write-AheadLogOptions
449
Ignore(non-WAL)datafileerrors
Ignoredatafileerrorswhenloadingcollections:--database.ignore-datafile-errorsboolean
Ifsettofalse,CRCmismatchandothererrorsincollectiondatafileswillleadtoacollectionnotbeingloadedatall.Thecollectioninthiscasebecomesunavailable.IfsuchcollectionneedstobeloadedduringWALrecovery,theWALrecoverywillalsoabort(ifnotforcedwithoption--wal.ignore-recovery-errorstrue).
Settingthisflagtofalseprotectsusersfromunintentionallyusingacollectionwithcorrupteddatafiles,fromwhichonlyasubsetoftheoriginaldatacanberecovered.Workingwithsuchcollectioncouldleadtodatalossandfollowuperrors.Inordertoaccesssuchcollection,itisrequiredtoinspectandrepairthecollectiondatafilewiththedatafiledebugger(arango-dfdb).
Ifsettotrue,CRCmismatchandothererrorsduringtheloadingofacollectionwillleadtothedatafilebeingpartiallyloaded,uptothepositionofthefirsterror.Alldatauptountiltheinvalidpositionwillbeloaded.Thiswillenableuserstocontinuewithcollectiondatafileseveniftheyarecorrupted,butthiswillresultinonlyapartialloadoftheoriginaldataandpotentialfollowuperrors.TheWALrecoverywillstillabortwhenencounteringacollectionwithacorrupteddatafile,atleastif--wal.ignore-recovery-errorsisnotsettotrue.
SettingtheoptiontotruewillalsoautomaticallrepairpotentiallycorruptedVERSIONfilesofdatabasesonstartup,sothatthestartupcanproceed.
Thedefaultvalueisfalse,socollectionswithcorrupteddatafileswillnotbeloadedatall,preventingpartialloadsandfollowuperrors.However,ifsuchcollectionisrequiredatserverstartup,duringWALrecovery,theserverwillaborttherecoveryandrefusetostart.
Write-AheadLogOptions
450
MMFilesCompactionoptionsTheArangoDBMMFilesstorageenginewillrunacompactionoverdatafiles.
ArangoDBwritesDocumentsintheWALfile.Oncetheyhavebeensealedinthewalfile,thecollectormaycopythemintoapercollectionjournalfile.
Oncejournalfilesfillup,they'resealedtobecomedatafiles.
=>onecollectionmayhavedocumentsintheWALlogs,itsjournalfile,andanarbitrarynumberofdatafiles.
Ifacollectionisloaded,eachofthesefilesareopened(thususeafilehandle)andaremmap'ed.Sincefilehandlesandmemorymappedfilesarealsoasparseresource,thatnumbershouldbekeptlow.
Onceyouupdateorremovedocumentsfromdatafiles(oralreadydidwhileitwasthejournalfile)thesedocumentsaremarkedas'dead'withadeletionmarker.
Overtimethenumberofdeaddocumentsmayrise,andwedon'twanttousethepreviouslymentionedresources,plusthediskspaceshouldbegivenbacktothesystem.Thusseveraljournalfilescanbecombinedtoone,ommittingthedeaddocuments.
Combiningseveralofthesedatafilesintooneiscalledcompaction.Thecompactionprocessreadsthealivedocumentsfromtheoriginaldatafiles,andwritesthemintonewdatafile.
Oncethatisdone,thememorymappingstotheolddatafilesisreleased,andthefilesareerased.
Sincethecompactionlocksthecollection,andalsousesI/Oresources,itscarefullyconfigurableunderwhichconditionsthesystemshouldperformwhichamountofthesecompactionjobs:
ArangoDBspawnsonecompactorthreadperdatabase.Thesettingsbelowvaryinscope.
ActivitycontrolTheactivitycontrolparametersalterthebehaviourintermsofscan/executionfrequencyofthecompaction.
Sleepintervalbetweentwocompactionruns(inseconds):--compaction.db-sleep-timeThenumberofsecondsthecollectorthreadwillwaitbetweentwoattemptstosearchforcompactabledatafilesofcollectionsinoneDatabase.Ifthecompactorhasactuallyexecutedwork,asubsequentlookupisdone.Scope:Database.
Minimumsleeptimebetweentwocompactionruns(inseconds):--compaction.min-intervalWhenanactualcompactionwasexecutedforonecollection,wewaitforthistimebeforeweexecutethecompactiononthiscollectionagain.Thisisheretoleteventuallypiledupuserloadbeworkedout.Scope:collection.
SourcedatafilesTheseparameterscontrolwhichdatafilesaretakenintoaccountforacompactionrun.Youcanspecifyseveralcriteriawhicheachoffmaybesufficcientalone.
Thescanoverthedatafilesbelongingtoonecollectionisexecutedfromoldestdatafiletonewest;iffilesqualifyforacompactiontheymaybemergedwithnewerfiles(containingyoungerdocuments)
Scope:Collectionlevel,someareinfluencedbycollectionsettings.
minimalfilesizethresholdoriginaldatafileshavetobebelowforacompaction:--compaction.min-small-data-file-sizeThisisthethresholdwhichcontrolsbelowwhichminimumtotalsizeadatafilewillalwaysbetakenintoaccountforthecompaction.
Minimumunusedcountofdocumentsinadatafile:--compaction.dead-documents-thresholdDatafileswilloftencontaindeaddocuments.Thisparameterspecifiestheirtopmostaccetpeablecountuntilthedatafilequalifiesforcompaction.
Howmanybytesofthesourcedatafileareallowedtobeunusedatmost:--compaction.dead-size-thresholdThedeaddatasizevariesalongwiththesizeofyourdocuments.Ifyouhavemanybigdocuments,thisthresholdmayhitbeforethedocumentcountthreshold.
CompactionOptions
451
Howmanypercentofthesourcedatafileshouldbeunusedatleast:--compaction.dead-size-percent-thresholdsincethesizeofthedocumentsmayvarythisthresholdworksonthepercentageofthedeaddocumentssize.Thus,ifyouhavemanyhugedeaddocuments,thisthresholdkicksinearlier.
Tonameanexamplewithnumbers,ifthedatafilecontains800kbytesofaliveand400kbytesofdeaddocuments,theshareofthedeaddocumentsis:
400/(400+800)=33%.
Ifthisvalueifhigherthanthespecifiedthreshold,thedatafilewillbecompacted.
CompactedtargetfilesOncedatafilesofacollectionarequalifiedforacompactionrun,theseparameterscontrolhowmanydatafilesaremergedintoone,(orevenonesourcedatafilemaybecompactedintoonesmallertargetdatafile)
Scope:Collectionlevel,someareinfluencedbycollectionsettings.
Maximumnumberoffilestomergetoonefile:--compaction.dest-max-filesHowmanydatafiles(atmost)wemaymergeintooneresultingdatafileduringonecompactionrun.
Howlargetheresultingfilemaybeincomparisontothecollectionsdatabase.maximal-journal-sizesetting:--compaction.dest-max-file-size-factorInArangoDByoucanconfigureadefaultjournalfilesizegloballyandoverrideitonapercollectionlevel.Thisvaluecontrolsthesizeofcollecteddatafilesrelativetotheconfiguredjournalfilesizeofthecollectioninquestion.
Afactorof3meansthatthemaximumfilesizeofthecompactedfileis3timesthesizeofthemaximumcollectionjournalfilesize.
howlargemaythecompactionresultfilebecome:--compaction.dest-max-result-file-sizenexttothefactorabove,atotallymaximumallowedfilesizeinbytesmaybespecified.Thiswilloverruleallpreviousparameters.
CompactionOptions
452
ClustersOptions
Agencyendpoint
Listofagencyendpoints:--cluster.agency-endpointendpoint
Anagencyendpointtheservercanconnectto.Theoptioncanbespecifiedmultipletimes,sotheservercanuseaclusterofagencyservers.Endpointshavethefollowingpattern:
tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryption
AtleastoneendpointmustbespecifiedorArangoDBwillrefusetostart.ItisrecommendedtospecifyatleasttwoendpointssoArangoDBhasanalternativeendpointifoneofthembecomesunavailable.
Examples
--cluster.agency-endpointtcp://192.168.1.1:4001--cluster.agency-endpointtcp://192.168.1.2:4002...
Myaddress
Thisserver'saddress/endpoint:--cluster.my-addressendpoint
Theserver'sendpointforcluster-internalcommunication.Ifspecified,itmusthavethefollowingpattern:
tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryption
Ifnoendpointisspecified,theserverwilllookupitsinternalendpointaddressintheagency.Ifnoendpointcanbefoundintheagencyfortheserver'sid,ArangoDBwillrefusetostart.
Examples
Listenonlyoninterfacewithaddress192.168.1.1
--cluster.my-addresstcp://192.168.1.1:8530
Listenonallipv4andipv6addresses,whichareconfiguredonport8530
--cluster.my-addressssl://[::]:8530
Myrole
Thisserver'srole:--cluster.my-role[dbserver|coordinator]
Theserver'srole.Isthisinstanceadbserver(backenddataserver)oracoordinator(frontendserverforexternalandapplicationaccess)
NodeID(deprecated)
Thisserver'sid:--cluster.my-local-infoinfo
Somelocalinformationabouttheserverinthecluster,thiscanforexamplebeanIPaddresswithaprocessIDoranystringuniquetotheserver.Specifyinginfoismandatoryonstartupiftheserverid(seebelow)isnotspecified.Eachserveroftheclustermusthaveauniquelocalinfo.Thisisignoredifmy-idbelowisspecified.
ClusterOptions
453
Thisoptionisdeprecatedandwillberemovedinafuturerelease.TheclusternodeidshavebeendroppedinfavourofoncegeneratedUUIDs.
Moreadvancedoptions(shouldgenerallyremainuntouched)
Synchroneousreplicationtiming:--cluster.synchronous-replication-timeout-factordouble
Strechorclinchtimeoutsforinternalsynchroneousreplicationmechanismbetweendbservers.Allsuchtimeoutsareaffectedbythischange.Pleasechangeonlywithintentandgreatcare.Defaultat1.0.
Systemreplicationfactor:--cluster.system-replication-factorinteger
Changedefaultreplicationfactorforsystemcollections.Defaultat2.
ClusterOptions
454
RocksDBengineoptionsRocksDBisahighlyconfigurablekey-valuestoreusedtopowerourRocksDBstorageengine.Mostoftheoptionsonthispagearepass-throughoptionstotheunderlyingRocksDBinstance,andwechangeveryfewoftheirdefaultsettings.
Dependingonthestorageengineyouhavechosentheavailabilityandthescopeoftheseoptionschanges.
Incaseyouhavechosenmmfilessomeofthefollowingoptionsapplytopersistentindexes.Incaseofrocksdbitwillapplytoalldatastoredaswellasindexes.
Pass-throughoptions--rocksdb.wal-directory
AbsolutepathfortheRocksDBWALfiles.Ifleftempty,thiswilluseasubdirectoryjournalsinsidethedatadirectory.
Writebuffers
--rocksdb.write-buffer-size
Theamountofdatatobuildupineachin-memorybuffer(backedbyalogfile)beforeclosingthebufferandqueuingittobeflushedintostandardstorage.Default:64MiB.Largervaluesmayimproveperformance,especiallyforbulkloads.
--rocksdb.max-write-buffer-number
Themaximumnumberofwritebuffersthatbuiltupinmemory.Ifthisnumberisreachedbeforethebufferscanbeflushed,writeswillbeslowedorstalled.Default:2.
--rocksdb.min-write-buffer-number-to-merge
Minimumnumberofwritebuffersthatwillbemergedtogetherwhenflushingtonormalstorage.Default:1.
--rocksdb.max-total-wal-size
MaximumtotalsizeofWALfilesthat,whenreached,willforceaflushofallcolumnfamilieswhosedataisbackedbytheoldestWALfiles.Settingthistoalowvaluewilltriggerregularflushingofcolumnfamilydatafrommemtables,sothatWALfilescanbemovedtothearchive.SettingthistoahighvaluewillavoidregularflushingbutmaypreventWALfilesfrombeingmovedtothearchiveandbeingremoved.
--rocksdb.delayed-write-rate(Hidden)
LimitedwriteratetoDB(inbytespersecond)ifwearewritingtothelastin-memorybufferallowedandweallowmorethan3buffers.Default:16MiB/s.
LSMtreestructure
--rocksdb.num-levels
ThenumberoflevelsforthedatabaseintheLSMtree.Default:7.
--rocksdb.num-uncompressed-levels
Thenumberoflevelsthatdonotusecompression.Thedefaultvalueis2.LevelsabovethisnumberwilluseSnappycompressiontoreducethediskspacerequirementsforstoringdataintheselevels.
--rocksdb.dynamic-level-bytes
Iftrue,theamountofdataineachleveloftheLSMtreeisdetermineddynamicallysoastominimizethespaceamplification;otherwise,thelevelsizesarefixed.ThedynamicsizingallowsRocksDBtomaintainawell-structuredLSMtreeregardlessoftotaldatasize.Default:true.
--rocksdb.max-bytes-for-level-base
Themaximumtotaldatasizeinbytesinlevel-1oftheLSMtree.Onlyeffectiveif--rocksdb.dynamic-level-bytesisfalse.Default:256MiB.
RocksDBEngineOptions
455
--rocksdb.max-bytes-for-level-multiplier
ThemaximumtotaldatasizeinbytesforlevelLoftheLSMtreecanbecalculatedasmax-bytes-for-level-base*(max-bytes-for-level-multiplier^(L-1)).Onlyeffectiveif--rocksdb.dynamic-level-bytesisfalse.Default:10.
--rocksdb.level0-compaction-trigger
Compactionoflevel-0tolevel-1istriggeredwhenthismanyfilesexistinlevel-0.Settingthistoahighernumbermayhelpbulkwritesattheexpenseofslowingdownreads.Default:2.
--rocksdb.level0-slowdown-trigger
Whenthismanyfilesaccumulateinlevel-0,writeswillbesloweddownto--rocksdb.delayed-write-ratetoallowcompactiontocatchup.Default:20.
--rocksdb.level0-stop-trigger
Whenthismanyfilesaccumulateinlevel-0,writeswillbestoppedtoallowcompactiontocatchup.Default:36.
FileI/O
--rocksdb.compaction-read-ahead-size
Ifnon-zero,weperformbiggerreadswhendoingcompaction.Ifyou'rerunningRocksDBonspinningdisks,youshouldsetthistoatleast2MiB.ThatwayRocksDB'scompactionisdoingsequentialinsteadofrandomreads.Default:0.
--rocksdb.use-direct-reads(Hidden)
OnlymeaningfulonLinux.Ifset,useO_DIRECTforreadingfiles.Default:false.
--rocksdb.use-direct-io-for-flush-and-compaction(Hidden)
OnlymeaningfulonLinux.Ifset,useO_DIRECTforwritingfiles.Default:false.
--rocksdb.use-fsync(Hidden)
Ifset,issueanfsynccallwhenwritingtodisk(settofalsetoissuefdatasynconly.Default:false.
Backgroundtasks
--rocksdb.max-background-jobs
Maximumnumberofconcurrentbackgroundcompactionjobs,submittedtothelowprioritythreadpool.Default:numberofprocessors.
--rocksdb.num-threads-priority-high
Numberofthreadsforhighpriorityoperations(e.g.flush).Werecommendsettingthisequaltomax-background-flushes.Default:numberofprocessors/2.
--rocksdb.num-threads-priority-low
Numberofthreadsforlowpriorityoperations(e.g.compaction).Default:numberofprocessors/2.
Caching
--rocksdb.block-cache-size
Thisisthesizeoftheblockcacheinbytes.Increasingthismayimproveperformance.Ifthereislessthan4GiBofRAMonthesystem,thedefaultvalueis256MiB.Ifthereismore,thedefaultis(systemRAMsize-2GiB)*0.3.
--rocksdb.block-cache-shard-bits
Thenumberofbitsusedtoshardtheblockcachetoallowconcurrentoperations.Tokeepindividualshardsatareasonablesize(i.e.atleast512KB),keepthisvaluetoatmostblock-cache-shard-bits/512KB.Default:block-cache-size/2^19.
--rocksdb.table-block-size
Approximatesizeofuserdata(inbytes)packedperblockforuncompresseddata.
--rocksdb.recycle-log-file-num(Hidden)
Numberoflogfilestokeeparoundforrecycling.Default:0.
RocksDBEngineOptions
456
Miscellaneous
--rocksdb.optimize-filters-for-hits(Hidden)
Thisflagspecifiesthattheimplementationshouldoptimizethefiltersmainlyforcaseswherekeysarefoundratherthanalsooptimizeforthecasewherekeysarenot.Thiswouldbeusedincaseswheretheapplicationknowsthatthereareveryfewmissesortheperformanceinthecaseofmissesisnotasimportant.Default:false.
--rocksdb.wal-recovery-skip-corrupted(Hidden)
Iftrue,skipcorruptedrecordsinWALrecovery.Default:false.
Non-Pass-ThroughOptions--rocksdb.wal-file-timeout(Hidden)
TimeoutafterwhichunusedWALfilesaredeleted(inseconds).Default:10.0s.
DataofongoingtransactionsisstoredinRAM.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatacreatedormodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.
ThefollowingoptionscanbeusedtocontroltheRAMusageandautomaticintermediatecommitsfortheRocksDBengine:
--rocksdb.max-transaction-size
Transactionsizelimit(inbytes).TransactionsstoreallkeysandvaluesinRAM,solargetransactionsruntheriskofcausingout-of-memorysitations.Thissettingallowsyoutoensurethatdoesnothappenbylimitingthesizeofanyindividualtransaction.TransactionswhoseoperationswouldconsumemoreRAMthanthisthresholdvaluewillabortautomaticallywitherror32("resourcelimitexceeded").
--rocksdb.intermediate-commit-size
Ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.
--rocksdb.intermediate-commit-count
Ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.
--rocksdb.throttle
Ifenabled,throttlestheingestrateofwritesifnecessarytoreducechancesofcompactionsgettingtoofarbehindandblockingincomingwrites.Thisoptionistruebydefault.
RocksDBEngineOptions
457
HashcacheoptionsSinceArangoDB3.2,theseveralcorecomponentsoftheserveruseacachesystemwhichpoolsmemoryacrossmanydifferentcachetables.Inordertoprovideintelligentinternalmemorymanagement,thesystemperiodicallyreclaimsmemoryfromcacheswhichareusedlessoftenandreallocatesittocacheswhichgetmoreactivity.
Cachesize
Globalsizelimitforallhashcaches:--cache.size
Theglobalcachingsystem,allcaches,andallthedatacontainedthereinwillfitinsidethislimit.Thesizeisspecifiedinbytes.Ifthereislessthan4GiBofRAMonthesystem,thedefaultvalueis256MiB.Ifthereismore,thedefaultis(systemRAMsize-2GiB)*0.3.
Rebalancinginterval
Timebetweencacherebalancingattempts:--cache.rebalancing-interval
Thevalueisspecifiedinmicrosecondswithadefaultof2secondsandaminimumof500milliseconds.
HashCacheOptions
458
AsynchronousTasks
maximalqueuesize
Maximumsizeofthequeueforrequests:--server.maximal-queue-sizesize
Specifiesthemaximumsizeofthequeueforasynchronoustaskexecution.Ifthequeuealreadycontainssizetasks,newtaskswillberejecteduntilothertasksarepoppedfromthequeue.Settingthisvaluemayhelppreventingfromrunningoutofmemoryifthequeueisfilledupfasterthantheservercanprocessrequests.
AsynchronousTasks
459
DurabilityConfiguration
GlobalConfiguration
Thereareglobalconfigurationvaluesfordurability,whichcanbeadjustedbyspecifyingthefollowingconfigurationoptions:
defaultwaitforsyncbehavior--database.wait-for-syncboolean
Defaultwait-for-syncvalue.Canbeoverwrittenwhencreatinganewcollection.
Thedefaultisfalse.
forcesyncingofcollectionpropertiestodisk--database.force-sync-propertiesboolean
Forcesyncingofcollectionpropertiestodiskaftercreatingacollectionorupdatingitsproperties.
Ifturnedoff,nofsyncwillhappenforthecollectionanddatabasepropertiesstoredinparameter.jsonfilesinthefilesystem.Turningoffthisoptionwillspeedupworkloadsthatcreateanddropalotofcollections(e.g.testsuites).
Thedefaultistrue.
intervalforautomatic,non-requesteddisksyncs--wal.sync-interval
Theinterval(inmilliseconds)thatArangoDBwillusetoautomaticallysynchronizedatainitswrite-aheadlogstodisk.Automaticsyncswillonlybeperformedfornot-yetsynchronizeddata,andonlyforoperationsthathavebeenexecutedwithoutthewaitForSyncattribute.
Per-collectionconfigurationYoucanalsoconfigurethedurabilitybehavioronaper-collectionbasis.UsetheArangoDBshelltochangetheseproperties.
getsorsetsthepropertiesofacollectioncollection.properties()
Returnsanobjectcontainingallcollectionproperties.
waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.
journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
isVolatile:IftruethenthecollectiondatawillbekeptinmemoryonlyandArangoDBwillnotwriteorsyncthedatatodisk.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
keyOptions(optional)additionaloptionsforkeygeneration.ThisisaJSONarraycontainingthefollowingattributes(note:someoftheattributesareoptional):
type:thetypeofthekeygeneratorusedforthecollection.allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.
indexBuckets:numberofbucketsintowhichindexesusingahashtablearesplit.Thedefaultis16andthisnumberhastobeapowerof2andlessthanorequalto1024.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
Forverylargecollectionsoneshouldincreasethistoavoidlongpauseswhenthehashtablehastobeinitiallybuiltorresized,sincebucketsareresizedindividuallyandcanbeinitiallybuiltinparallel.Forexample,64mightbeasensiblevalueforacollectionwith100000000documents.Currently,onlytheedgeindexrespectsthisvalue,butotherindextypesmightfollowinfutureArangoDBversions.Changes(seebelow)areappliedwhenthecollectionisloadedthenexttime.
Inaclustersetup,theresultwillalsocontainthefollowingattributes:
numberOfShards:thenumberofshardsofthecollection.
Durability
460
shardKeys:containsthenamesofdocumentattributesthatareusedtodeterminethetargetshardfordocuments.
replicationFactor:determineshowmanycopiesofeachshardarekeptondifferentDBServers.
collection.properties(properties)
Changesthecollectionproperties.propertiesmustbeanobjectwithoneormoreofthefollowingattribute(s):
waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.
journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
indexBuckets:Seeabove,changesareonlyappliedwhenthecollectionisloadedthenexttime.ThisoptionismeaningfulfortheMMFilesstorageengineonly.
replicationFactor:ChangethenumberofshardcopieskeptondifferentDBServers,validvaluesareintegernumbersintherangeof1-10(Clusteronly)
Note:itisnotpossibletochangethejournalsizeafterthejournalordatafilehasbeencreated.Changingthisparameterwillonlyeffectnewlycreatedjournals.Alsonotethatyoucannotlowerthejournalsizetolessthensizeofthelargestdocumentalreadystoredinthecollection.
Note:someothercollectionproperties,suchastype,isVolatile,orkeyOptionscannotbechangedoncethecollectioniscreated.
Examples
Readallproperties
arangosh>db.example.properties();
showexecutionresultsChangeaproperty
arangosh>db.example.properties({waitForSync:true});
showexecutionresults
Per-operationconfigurationManydata-modificationoperationsandalsoArangoDB'stransactionsallowtospecifyawaitForSyncattribute,whichwhensetensurestheoperationdatahasbeensynchronizedtodiskwhentheoperationreturns.
Disk-UsageConfigurationTheamountofdiskspaceusedbyArangoDBisdeterminedbyafewconfigurationoptions.
GlobalConfigurationThetotalamountofdiskstoragerequiredbyArangoDBisdeterminedbythesizeofthewrite-aheadlogfilesplusthesizesofthecollectionjournalsanddatafiles.
Therearethefollowingoptionsforconfiguringthenumberandsizesofthewrite-aheadlogfiles:
maximumnumberofreservelogfiles--wal.reserve-logfiles
ThemaximumnumberofreservelogfilesthatArangoDBwillcreateinabackgroundprocess.Reservelogfilesareusefulinthesituationwhenanoperationneedstobewrittentoalogfilebutthereservespaceinthelogfileistoolowforstoringtheoperation.Inthiscase,anewlogfileneedstobecreatedtostoretheoperation.Creatingnewlogfilesisnormallyslow,soArangoDBwilltrytopre-createlogfilesinabackgroundprocesssotherearealwaysreservelogfileswhentheactivelogfilegetsfull.ThenumberofreservelogfilesthatArangoDBkeepsinthebackgroundisconfigurablewiththisoption.
Durability
461
maximumnumberofhistoriclogfiles--wal.historic-logfiles
ThemaximumnumberofhistoriclogfilesthatArangoDBwillkeepaftertheyhavebeengarbage-collected.Ifnoreplicationisused,thereisnoneedtokeephistoriclogfilesexceptforhavingalocalchangelog.
Inareplicationsetup,thenumberofhistoriclogfilesaffectstheamountofdataaslavecanfetchfromthemaster'slogs.Themorehistoriclogfiles,themorehistoricdataisavailableforaslave,whichisusefuliftheconnectionbetweenmasterandslaveisunstableorslow.Nothavingenoughhistoriclogfilesavailablemightleadtologfiledatabeingdeletedonthemasteralreadybeforeaslavehasfetchedit.
thesizeofeachWALlogfile--wal.logfile-size
Specifiesthefilesize(inbytes)foreachwrite-aheadlogfile.Thelogfilesizeshouldbechosensothateachlogfilecanstoreaconsiderableamountofdocuments.Thebiggerthelogfilesizeischosen,thelongeritwilltaketofillupasinglelogfile,whichalsoinfluencesthedelayuntilthedatainalogfilewillbegarbage-collectedandwrittentocollectionjournalsanddatafiles.Italsoaffectshowlonglogfilerecoverywilltakeatserverstart.
whetherornotoversizeentriesareallowed--wal.allow-oversize-entries
Whetherornotitisallowedtostoreindividualdocumentsthatarebiggerthanwouldfitintoasinglelogfile.Settingtheoptiontofalsewillmakesuchoperationsfailwithanerror.Settingtheoptiontotruewillmakesuchoperationssucceed,butwithahighpotentialperformanceimpact.Thereasonisthatforeachoversizeoperation,anindividualoversizelogfileneedstobecreatedwhichmayalsoblockotheroperations.Theoptionshouldbesettofalseifitiscertainthatdocumentswillalwayshaveasizesmallerthanasinglelogfile.Whendatagetscopiedfromthewrite-aheadlogfilesintothejournalsordatafilesofcollections,fileswillbecreatedonthecollectionlevel.Howbigthesefilesareisdeterminedbythefollowingglobalconfigurationvalue:
--database.maximal-journal-sizesize
Maximalsizeofjournalinbytes.Canbeoverwrittenwhencreatinganewcollection.Notethatthisalsolimitsthemaximalsizeofasingledocument.
Thedefaultis32MB.
Per-collectionconfigurationThejournalsizecanalsobeadjustedonaper-collectionlevelusingthecollection'spropertiesmethod.
Durability
462
EncryptionThisfeatureisonlyavailableintheEnterpriseEdition.
WhenyoustoresensitivedatainyourArangoDBdatabase,youwanttoprotectthatdataunderallcircumstances.AtruntimeyouwillprotectitwithSSLtransportencryptionandstrongauthentication,butwhenthedataisalreadyondisk,youalsoneedprotection.ThatiswheretheEncryptionfeaturecomesin.
TheEncryptionfeatureofArangoDBwillencryptalldatathatArangoDBisstoringinyourdatabasebeforeitiswrittentodisk.
ThedataisencryptedwithAES-256-CTR,whichisastrongencryptionalgorithm,thatisverysuitableformulti-processorenvironments.Thismeansthatyourdataissafe,butyourdatabaseisstillfast,evenunderload.
MostmodernCPU'shavebuiltinsupportforhardwareAESencryption,whichmakesitevenfaster.
Note:TheEncryptionfeaturerequirestheRocksDBstorageengine.
Encryptionkeys
TheEncryptionfeatureofArangoDBrequiresasingle32-bytekeyperserver.Itisrecommendedtouseadifferentkeyforeachserver(whenoperatinginaclusterconfiguration).Makesuretoprotectthesekeys!
Thatmeans:
Donotwritethemtopersistentdisksoryourserver(s),alwaysstorethemonanin-memory(tmpfs)filesystem.Transportyourkeyssafelytoyourserver(s).Therearevarioustoolsformanagingsecretslikethis(e.g.vaultproject.io).Storeacopyofyourkeyofflineinasafeplace.Ifyouloseyourkey,thereisNOwaytogetyourdataback.
ConfigurationToactivateencryptionofyourdatabase,youneedtosupplyanencryptionkeytotheserver.
Makesuretopassthisoptiontheveryfirsttimeyoustartyourdatabase.Youcannotencryptadatabasethatalreadyexists.
Note:Youalsohavetoactivatetherocksdbstorageengine.
Encryptionkeystoredinfile
Passthefollowingoptiontoarangod:
$arangod\
--rocksdb.encryption-keyfile=/mytmpfs/mySecretKey\
--server.storage-engine=rocksdb
Thefile/mytmpfs/mySecretKeymustcontaintheencryptionkey.Thisfilemustbesecured,sothatonlyarangodcanaccessit.Youshouldalsoensurethatincasesome-onestealsthehardware,hewillnotbeabletoreadthefile.Forexample,byencryption/mytmpfsorcreatingain-memoryfile-systemunder/mytmpfs.
Encryptionkeygeneratedbyaprogram
Passthefollowingoptiontoarangod:
$arangod\
--rocksdb.encryption-key-generator=path-to-my-generator\
--server.storage-engine=rocksdb
Theprogrampath-to-my-generatoroutputtheencryptiononstandardoutputandexit.
Encryption
463
Creatingkeys
Theencryptionkeyfilemustcontain32bytesofrandomdata.
Youcancreateitwithacommandlinethis.
ddif=/dev/randombs=1count=32of=yourSecretKeyFile
Forsecurity,itisbesttocreatethesekeysoffline(awayfromyourdatabaseservers)anddirectlystoretheminyousecretmanagementtool.
Encryption
464
AuditingThisfeatureisavailableintheEnterpriseEdition.
Auditingallowsyoutomonitoraccesstothedatabaseindetail.Ingeneralauditlogsareoftheform
2016-01-0112:00:00|server|username|database|client-ip|authentication|text1|text2|...
Thetime-stampisinGMT.Thisallowstoeasilymatchlogentriesfromserversindifferenttimezones.
Thenameoftheserver.Youcanspecifyacustomnameonstartup.Otherwisethedefaulthostnameisused.
Theusernameisthe(authenticatedorunauthenticated)namesuppliedbytheclient.Adash-isprintedifnonamewasgivenbytheclient.
Thedatabasedescribesthedatabasethatwasaccessed.Pleasenotethattherearenodatabasecrossingqueries.Eachaccessisrestrictedtoonedatabase.
Theclient-ipdescribesthesourceoftherequest.
Theauthenticationdetailsthemethodsusedtoauthenticatetheuser.
Detailsabouttherequestsfollowintheadditionalfields.
Auditing
465
AuditConfigurationThisfeatureisavailableintheEnterpriseEdition.
Output--audit.outputoutput
Specifiesthetargetoftheauditlog.Possiblevaluesare
file://filenamewherefilenamecanberelativeorabsolute.
syslog://facilityorsyslog://facility/application-nametologintoasyslogserver.
Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputformultipletargets.
Hostname--audit.hostnamename
Thenameoftheserverusedinauditlogmessages.Bydefaultthesystemhostnameisused.
Configuration
466
AuditEventsThisfeatureisavailableintheEnterpriseEdition.
Authentication
Unknownauthenticationmethods
2016-10-0315:44:23|server1|-|database1|127.0.0.1:61525|-|unknownauthenticationmethod|/_api/version
Missingcredentials
2016-10-0315:39:49|server1|-|database1|127.0.0.1:61498|-|credentialsmissing|/_api/version
Wrongcredentials
2016-10-0315:47:26|server1|user1|database1|127.0.0.1:61528|httpbasic|credentialswrong|/_api/version
Passwordchangerequired
2016-10-0316:18:53|server1|user1|database1|127.0.0.1:62257|-|passwordchangerequired|/_api/version
JWTloginsucceeded
2016-10-0317:21:22|server1|-|database1|127.0.0.1:64214|httpjwt|user'root'authenticated|/_open/auth
Pleasenote,thattheusergivenasthirdpartistheuserthatrequestedthelogin.Ingeneral,itwillbeempty.
JWTloginfailed
2016-10-0317:21:22|server1|-|database1|127.0.0.1:64214|httpjwt|user'root'wrongcredentials|/_open/auth
Pleasenote,thattheusergivenasthirdpartistheuserthatrequestedthelogin.Ingeneral,itwillbeempty.
Authorization
Usernotauthorizedtoaccessdatabase
2016-10-0316:20:52|server1|user1|database2|127.0.0.1:62262|httpbasic|notauthorized|/_api/version
Databases
Createadatabase
2016-10-0415:33:25|server1|user1|database1|127.0.0.1:56920|httpbasic|createdatabase'database1'|ok|/_api/dat
abase
Events
467
Dropadatabase
2016-10-0415:33:25|server1|user1|database1|127.0.0.1:56920|httpbasic|deletedatabase'database1'|ok|/_api/dat
abase
Collections
Createacollection
2016-10-0517:35:57|server1|user1|database1|127.0.0.1:51294|httpbasic|createcollection'collection1'|ok|/_api
/collection
Truncateacollection
2016-10-0517:36:08|server1|user1|database1|127.0.0.1:51294|httpbasic|truncatecollection'collection1'|ok|/_a
pi/collection/collection1/truncate
Dropacollection
2016-10-0517:36:30|server1|user1|database1|127.0.0.1:51294|httpbasic|deletecollection'collection1'|ok|/_api
/collection/collection1
Indexes
Createaindex
2016-10-0518:19:40|server1|user1|database1|127.0.0.1:52467|httpbasic|createindexin'collection1'|ok|{"field
s":["a"],"sparse":false,"type":"skiplist","unique":false}|/_api/index?collection=collection1
Dropaindex
2016-10-0518:18:28|server1|user1|database1|127.0.0.1:52464|httpbasic|dropindex':44051'|ok|/_api/index/colle
ction1/44051
Documents
Readingasingledocument
2016-10-0412:27:55|server1|user1|database1|127.0.0.1:53699|httpbasic|createdocumentok|/_api/document/collecti
on1
Replacingasingledocument
2016-10-0412:28:08|server1|user1|database1|127.0.0.1:53699|httpbasic|replacedocumentok|/_api/document/collect
ion1/21456?ignoreRevs=false
Modifyingasingledocument
2016-10-0412:28:15|server1|user1|database1|127.0.0.1:53699|httpbasic|modifydocumentok|/_api/document/collecti
on1/21456?keepNull=true&ignoreRevs=false
Events
468
Deletingasingledocument
2016-10-0412:28:23|server1|user1|database1|127.0.0.1:53699|httpbasic|deletedocumentok|/_api/document/collecti
on1/21456?ignoreRevs=false
Forexample,ifsomeonestriestodeleteanon-existingdocument,itwillbeloggedas
2016-10-0412:28:26|server1|user1|database1|127.0.0.1:53699|httpbasic|deletedocumentfailed|/_api/document/coll
ection1/21456?ignoreRevs=false
Queries
2016-10-0612:12:10|server1|user1|database1|127.0.0.1:54232|httpbasic|querydocument|ok|foriincollection1r
eturni|/_api/cursor
Events
469
IntroductiontoReplicationReplicationallowsyoutoreplicatedataontoanothermachine.ItformsthebaseofalldisasterrecoveryandfailoverfeaturesArangoDBoffers.
ArangoDBoffersasynchronousandsynchronousreplication,dependingonwhichtypeofarangodbdeploymentyouareusing.SinceArangoDB3.2thesynchronousreplicationreplicationistheonlyreplicationtypeusedinaclusterwhereastheasynchronousreplicationisonlyavailablebetweensingle-servernodes.FutureversionsofArangoDBmayreintroduceasynchronousreplicationforthecluster.
Wewilldescribeprosandconsofeachoftheminthefollowingsections.
Asynchronousreplication
InArangoDBanywriteoperationwillbeloggedtothewrite-aheadlog.WhenusingAsynchronousreplicationslaveswillconnecttoamasterandapplyalltheeventsfromtheloginthesameorderlocally.Afterthat,theywillhavethesamestateofdataasthemasterdatabase.
Synchronousreplication
Synchronousreplicationonlyworkswithinaclusterandistypicallyusedformissioncriticaldatawhichmustbeaccessibleatalltimes.Synchronousreplicationgenerallystoresacopyofashard'sdataonanotherdbserverandkeepsitinsync.Essentially,whenstoringdataafterenablingsynchronousreplicationtheclusterwillwaitforallreplicastowriteallthedatabeforegreenlightingthewriteoperationtotheclient.Thiswillnaturallyincreasethelatencyabit,sinceonemorenetworkhopisneededforeachwrite.However,itwillenabletheclustertoimmediatelyfailovertoareplicawheneveranoutagehasbeendetected,withoutlosinganycommitteddata,andmostlywithoutevensignalinganerrorconditiontotheclient.
Synchronousreplicationisorganizedsuchthateveryshardhasaleaderandr-1followers,whererdenotedthereplicationfactor.ThenumberoffollowerscanbecontrolledusingthereplicationFactorparameterwheneveryoucreateacollection,thereplicationFactorparameteristhetotalnumberofcopiesbeingkept,thatis,itisoneplusthenumberoffollowers.
Satellitecollections
SatellitecollectionsaresynchronouslyreplicatedcollectionshavingadynamicreplicationFactor.Theywillreplicatealldatatoalldatabaseserversallowingthedatabaseserverstojoindatalocallyinsteadofdoingheavynetworkoperations.
Satellitecollectionsareanenterpriseonlyfeature.
Replication
470
AsynchronousreplicationAsynchronousreplicationworksbyloggingeverydatamodificationonamasterandreplayingtheseeventsonanumberofslaves.
Transactionsarehonoredinreplication,i.e.transactionalwriteoperationswillbecomevisibleonslavesatomically.
Asallwriteoperationswillbeloggedtoamasterdatabase'swrite-aheadlog,thereplicationinArangoDBcurrentlycannotbeusedforwrite-scaling.ThemainpurposesofthereplicationincurrentArangoDBaretoprovideread-scalabilityand"hotbackups"ofspecificdatabases.
Itispossibletoconnectmultipleslavedatabasestothesamemasterdatabase.Slavedatabasesshouldbeusedasread-onlyinstances,andnouser-initiatedwriteoperationsshouldbecarriedoutonthem.Otherwisedataconflictsmayoccurthatcannotbesolvedautomatically,andthatwillmakethereplicationstop.
Inanasynchronousreplicationscenarioslaveswillpullchangesfromthemasterdatabase.Slavesneedtoknowtowhichmasterdatabasetheyshouldconnectto,butamasterdatabaseisnotawareoftheslavesthatreplicatefromit.Whenthenetworkconnectionbetweenthemasterdatabaseandaslavegoesdown,writeoperationsonthemastercancontinuenormally.Whenthenetworkisupagain,slavescanreconnecttothemasterdatabaseandtransfertheremainingchanges.Thiswillhappenautomaticallyprovidedslavesareconfiguredappropriately.
Replicationlag
Inthissetup,writeoperationsareappliedfirstinthemasterdatabase,andappliedintheslavedatabase(s)afterwards.
Forexample,let'sassumeawriteoperationisexecutedinthemasterdatabaseatpointintimet0.Tomakeaslavedatabaseapplythesameoperation,itmustfirstfetchthewriteoperation'sdatafrommasterdatabase'swrite-aheadlog,thenparseitandapplyitlocally.Thiswillhappenatsomepointintimeaftert0,let'ssayt1.
Thedifferencebetweent1andt0iscalledthereplicationlag,anditisunavoidableinasynchronousreplication.Theamountofreplicationlagdependsonmanyfactors,afewofwhichare:
thenetworkcapacitybetweentheslavesandthemastertheloadofthemasterandtheslavesthefrequencyinwhichslavespollthemasterforupdates
Betweent0andt1,thestateofdataonthemasterisnewerthanthestateofdataontheslave(s).Atpointintimet1,thestateofdataonthemasterandslave(s)isconsistentagain(providednonewdatamodificationshappenedonthemasterinbetween).Thus,thereplicationwillleadtoaneventuallyconsistentstateofdata.
Replicationconfiguration
Thereplicationisturnedoffbydefault.Inordertocreateamaster-slavesetup,theso-calledreplicationapplierneedstobeenabledontheslavedatabases.
Replicationisconfiguredonaper-databaselevel.Ifmultipledatabasearetobereplicated,thereplicationmustbesetupindividuallyperdatabase.
Thereplicationapplierontheslavecanbeusedtoperformaone-timesynchronizationwiththemaster(andthenstop),ortoperformanongoingreplicationofchanges.Toresumereplicationonslaverestart,theautoStartattributeofthereplicationappliermustbesettotrue.
Replicationoverhead
Asthemasterserversarelogginganywriteoperationinthewrite-ahead-loganywayreplicationdoesn'tcauseanyextraoverheadonthemaster.Howeveritwillofcoursecausesomeoverheadforthemastertoserveincomingreadrequestsoftheslaves.Returningtherequesteddataishoweveratrivialtaskforthemasterandshouldnotresultinanotableperformancedegrationinproduction.
AsynchronousReplication
471
Components
ReplicationLogger
Purpose
Thereplicationloggerwillwritealldata-modificationoperationsintothewrite-aheadlog.Thislogmaythenbereadbyclientstoreplayanydatamodificationonadifferentserver.
Checkingthestate
Toquerythecurrentstateofthelogger,usethestatecommand:
require("@arangodb/replication").logger.state();
Theresultmightlooklikethis:
{
"state":{
"running":true,
"lastLogTick":"133322013",
"totalEvents":16,
"time":"2014-07-06T12:58:11Z"
},
"server":{
"version":"2.2.0-devel",
"serverId":"40897075811372"
},
"clients":{
}
}
Therunningattributewillalwaysbetrue.InearlierversionsofArangoDBthereplicationwasoptionalandthiscouldhavebeenfalse.
ThetotalEventsattributeindicateshowmanylogeventshavebeenloggedsincethestartoftheArangoDBserver.Finally,thelastLogTickvalueindicatestheidofthelastoperationthatwaswrittentotheserver'swrite-aheadlog.Itcanbeusedtodeterminewhethernewoperationswerelogged,andisalsousedbythereplicationapplierforincrementalfetchingofdata.
Note:ThereplicationloggerstatecanalsobequeriedviatheHTTPAPI.
Toquerywhichdatarangesarestillavailableforreplicationclientstofetch,theloggerprovidesthefirstTickandtickRangesfunctions:
require("@arangodb/replication").logger.firstTick();
ThiswillreturntheminimumtickvaluethattheservercanprovidetoreplicationclientsviaitsreplicationAPIs.ThetickRangesfunctionreturnstheminimumandmaximumtickvaluesperlogfile:
require("@arangodb/replication").logger.tickRanges();
ReplicationApplier
Purpose
Thepurposeofthereplicationapplieristoreaddatafromamasterdatabase'seventlog,andapplythemlocally.Theapplierwillcheckthemasterdatabasefornewoperationsperiodically.Itwillperformanincrementalsynchronization,i.e.onlyaskingthemasterforoperationsthatoccurredafterthelastsynchronization.
AsynchronousReplication
472
Thereplicationapplierdoesnotgetnotifiedbythemasterdatabasewhenthereare"new"operationsavailable,butinsteadusesthepullprinciple.Itmightthustakesometime(theso-calledreplicationlag)beforeanoperationfromthemasterdatabasegetsshippedtoandappliedinaslavedatabase.
Thereplicationapplierofadatabaseisruninaseparatethread.Itmayencounterproblemswhenanoperationfromthemastercannotbeappliedsafely,orwhentheconnectiontothemasterdatabasegoesdown(networkoutage,masterdatabaseisdownorunavailableetc.).Inthiscase,thedatabase'sreplicationapplierthreadmightterminateitself.Itisthenuptotheadministratortofixtheproblemandrestartthedatabase'sreplicationapplier.
Ifthereplicationappliercannotconnecttothemasterdatabase,orthecommunicationfailsatsomepointduringthesynchronization,thereplicationapplierwilltrytoreconnecttothemasterdatabase.Itwillgiveupreconnectingonlyafteraconfigurableamountofconnectionattempts.
Thereplicationapplierstateisqueryableatanytimebyusingthestatecommandoftheapplier.Thiswillreturnthestateoftheapplierofthecurrentdatabase:
require("@arangodb/replication").applier.state();
Theresultmightlooklikethis:
{
"state":{
"running":true,
"lastAppliedContinuousTick":"152786205",
"lastProcessedContinuousTick":"152786205",
"lastAvailableContinuousTick":"152786205",
"progress":{
"time":"2014-07-06T13:04:57Z",
"message":"fetchingmasterlogfromoffset152786205",
"failedConnects":0
},
"totalRequests":38,
"totalFailedConnects":0,
"totalEvents":1,
"lastError":{
"errorNum":0
},
"time":"2014-07-06T13:04:57Z"
},
"server":{
"version":"2.2.0-devel",
"serverId":"210189384542896"
},
"endpoint":"tcp://master.example.org:8529",
"database":"_system"
}
Therunningattributeindicateswhetherthereplicationapplierofthecurrentdatabaseiscurrentlyrunningandpollingtheserveratendpointfornewevents.
Theprogress.failedConnectsattributeshowshowmanyfailedconnectionattemptsthereplicationappliercurrentlyhasencounteredinarow.Incontrast,thetotalFailedConnectsattributeindicateshowmanyfailedconnectionattemptstheapplierhasmadeintotal.ThetotalRequestsattributeshowshowmanyrequeststheapplierhassenttothemasterdatabaseintotal.ThetotalEventsattributeshowshowmanylogeventstheapplierhasreadfromthemaster.
Theprogress.messagesub-attributeprovidesabriefhintofwhattheappliercurrentlydoes(ifitisrunning).ThelastErrorattributealsohasanoptionalerrorMessagesub-attribute,showingthelatesterrormessage.TheerrorNumsub-attributeofthelastErrorattributecanbeusedbyclientstoprogrammaticallycheckforerrors.Itshouldbe0ifthereisnoerror,anditshouldbenon-zeroiftheapplierterminateditselfduetoaproblem.
Hereisanexampleofthestateafterthereplicationapplierterminateditselfdueto(repeated)connectionproblems:
{
"state":{
"running":false,
"progress":{
"time":"2014-07-06T13:14:37Z",
AsynchronousReplication
473
"message":"applierstopped",
"failedConnects":6
},
"totalRequests":79,
"totalFailedConnects":11,
"totalEvents":0,
"lastError":{
"time":"2014-07-06T13:09:41Z",
"errorMessage":"couldnotconnecttomasterattcp://master.example.org:8529:Couldnotconnectto'tcp:/...",
"errorNum":1400
},
...
}
}
Note:thestateofadatabase'sreplicationapplierisqueryableviatheHTTPAPI,too.PleaserefertoHTTPInterfaceforReplicationformoredetails.
All-in-onesetup
Tocopytheinitialdatafromtheslavetothemasterandstartthecontinuousreplication,thereisanall-in-onecommandsetupReplication:
require("@arangodb/replication").setupReplication(configuration);
Thefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthe_systemdatabase.Notethatitshouldberunontheslaveandnotthemaster:
db._useDatabase("_system");
require("@arangodb/replication").setupReplication({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
verbose:false,
includeSystem:false,
incremental:true,
autoResync:true
});
Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationisstarted,orincasetheinitialsynchronizationhasfailed.
Iftheinitialsynchronizationissuccessful,thecommandwillstorethegivenconfigurationontheslave.Italsoconfiguresthecontinuousreplicationtostartautomaticallyiftheslaveisrestarted,i.e.autoStartissettotrue.
Ifthecommandisrunwhiletheslave'sreplicationapplierisalreadyrunning,itwillfirststoptherunningapplier,dropitsconfigurationanddoaresynchronizationofdatawiththemaster.Itwillthenusetheprovidedconfigration,overwritinganypreviouslyexistingreplicationconfigurationontheslave.
StartingandStopping
Tomanuallystartandstoptheapplierinthecurrentdatabase,thestartandstopcommandscanbeusedlikethis:
require("@arangodb/replication").applier.start(<tick>);
require("@arangodb/replication").applier.stop();
Note:Startingareplicationapplierwithoutsettingupaninitialconfigurationwillfail.ThereplicationapplierwilllookforitsconfigurationinafilenamedREPLICATION-APPLIER-CONFIGinthecurrentdatabase'sdirectory.Ifthefileisnotpresent,ArangoDBwillusesomedefaultconfiguration,butitcannotguesstheendpoint(theaddressofthemasterdatabase)theappliershouldconnectto.Thusstartingtheapplierwithoutconfigurationwillfail.
Notethatatthefirsttimeyoustarttheapplier,youshouldpassthevaluereturnedinthelastLogTickattributeoftheinitialsyncoperation.
AsynchronousReplication
474
Note:Startingadatabase'sreplicationapplierviathestartcommandwillnotnecessarilystarttheapplieronthenextandfollowingArangoDBserverrestarts.Additionally,stoppingadatabase'sreplicationappliermanuallywillnotnecessarilypreventtheapplierfrombeingstartedagainonthenextserverstart.Allofthisisconfigurableseparately(hangonreading).
Note:whenstoppingandrestartingthereplicationapplierofdatabase,itwillresumewhereitlaststopped.Thisissensiblebecausereplicationlogeventsshouldbeappliedincrementally.Ifthereplicationapplierofadatabasehasneverbeenstartedbefore,itneedssometickvaluefromthemaster'slogfromwhichtostartfetchingevents.
Thereisonecaveattoconsiderwhenstoppingareplicationontheslave:iftherearestillongoingreplicatedtransactionsthatareneithercommittedoraborted,stoppingthereplicationapplierwillcausetheseoperationstobelostfortheslave.Ifthesetransactionscommitonthemasterlaterandthereplicationisresumed,theslavewillnotbeabletocommitthesetransactions,too.Thusstoppingthereplicationapplierontheslavemanuallyshouldonlybedoneifthereiscertaintythattherearenoongoingtransactionsonthemaster.
Configuration
Toconfigurethereplicationapplierofaspecificdatabase,usethepropertiescommand.Usingitwithoutanyargumentswillreturntheapplier'scurrentconfiguration:
require("@arangodb/replication").applier.properties();
Theresultmightlooklikethis:
{
"requestTimeout":600,
"connectTimeout":10,
"ignoreErrors":0,
"maxConnectRetries":10,
"chunkSize":0,
"autoStart":false,
"adaptivePolling":true,
"includeSystem":true,
"requireFromPresent":false,
"autoResync":false,
"autoResyncRetries":2,
"verbose":false
}
Note:Thereisnoendpointattributeconfiguredyet.Theendpointattributeisrequiredforthereplicationappliertobestartable.Youmayalsowanttoconfigureausernameandpasswordfortheconnectionviatheusernameandpasswordattributes.
require("@arangodb/replication").applier.properties({
endpoint:"tcp://master.domain.org:8529",
username:"root",
password:"secret",
verbose:false
});
Thiswillre-configurethereplicationapplierforthecurrentdatabase.Theconfigurationwillbeusedfromthenextstartofthereplicationapplier.Thereplicationappliercannotbere-configuredwhileitisrunning.Itmustbestoppedfirsttobere-configured.
TomakethereplicationapplierofthecurrentdatabasestartautomaticallywhentheArangoDBserverstarts,usetheautoStartattribute.
SettingtheadaptivePollingattributetotruewillmakethereplicationapplierpollthemasterdatabaseforchangeswithavariablefrequency.Thereplicationapplierwillthenlowerthefrequencywhenthemasterisidle,andincreaseitwhenthemastercanprovidenewevents).Otherwisethereplicationapplierwillpollthemasterdatabaseforchangeswithaconstantfrequency.
TheidleMinWaitTimeattributecontrolstheminimumwaittime(inseconds)thatthereplicationapplierwillintentionallyidlebeforefetchingmorelogdatafromthemasterincasethemasterhasalreadysentallitslogdata.ThiswaittimecanbeusedtocontrolthefrequencywithwhichthereplicationappliersendsHTTPlogfetchrequeststothemasterincasethereisnowriteactivityonthemaster.
TheidleMaxWaitTimeattributecontrolsthemaximumwaittime(inseconds)thatthereplicationapplierwillintentionallyidlebeforefetchingmorelogdatafromthemasterincasethemasterhasalreadysentallitslogdataandtherehavebeenpreviouslogfetchattemptsthatresultedinnomorelogdata.Thiswaittimecanbeusedtocontrolthemaximumfrequencywithwhichthereplicationappliersends
AsynchronousReplication
475
HTTPlogfetchrequeststothemasterincasethereisnowriteactivityonthemasterforlongerperiods.NotethatthisconfigurationvaluewillonlybeusediftheoptionadaptivePollingissettotrue.
Tosetatimeoutforconnectionandfollowingrequestattempts,usetheconnectTimeoutandrequestTimeoutvalues.ThemaxConnectRetriesattributeconfiguresafterhowmanyfailedconnectionattemptsinarowthereplicationapplierwillgiveupandturnitselfoff.Youmaywanttosetthistoahighvaluesothattemporarynetworkoutagesdonotleadtothereplicationapplierstoppingitself.TheconnectRetryWaitTimeattributeconfigureshowlongthereplicationapplierwillwaitbeforeretryingtheconnectiontothemasterincaseofconnectionproblems.
ThechunkSizeattributecanbeusedtocontroltheapproximatemaximumsizeofamaster'sresponse(inbytes).Settingittoalowvaluemaymakethemasterrespondfaster(lessdataisassembledbeforethemastersendstheresponse),butmayrequiremorerequest-responseroundtrips.Setitto0touseArangoDB'sbuilt-indefaultvalue.
TheincludeSystemattributecontrolswhetherchangestosystemcollections(suchas_graphsor_users)shouldbeapplied.Ifsettotrue,changesinthesecollectionswillbereplicated,otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.
TherequireFromPresentattributecontrolswhethertheapplierwillstartsynchronizingincaseitdetectsthatthemastercannotprovidedatafortheinitialtickvalueprovidedbytheslave.ThismaybethecaseifthemasterdoesnothaveabigenoughbacklogofhistoricWALlogfiles,andwhenthereplicationisre-startedafteralongerpause.WhenrequireFromPresentissettotrue,thenthereplicationapplierwillcheckatstartwhetherthestarttickfromwhichitstartsorresumesreplicationisstillpresentonthemaster.Ifnot,thentherewouldbedataloss.IfrequireFromPresentistrue,thereplicationapplierwillabortwithanappropriateerrormessage.Ifsettofalse,thenthereplicationapplierwillstillstart,andignorethedataloss.
TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslaverequests,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.
Automaticre-synchronizationmaytransferalotofdatafromthemastertotheslaveandcanbeexpensive.Itisthereforeturnedoffbydefault.Whenturnedoff,theslavewillneverperformanautomaticre-synchronizationwiththemaster.
TheautoResyncRetriesoptioncanbeusedtocontrolthenumberofresynchronizationretriesthatwillbeperformedinarowwhenautomaticresynchronizationisenabledandkicksin.Settingthisto0willeffectivelydisableautoResync.Settingittosomeothervaluewilllimitthenumberofretriesthatareperformed.Thishelpspreventingendlessretriesincaseresynchronizationsalwaysfail.
Theverboseattributecontrolstheverbosityofthereplicationlogger.Settingittotruewillmakethereplicationapplierwritealinetothelogforeveryoperationitperforms.Thisshouldonlybeusedfordiagnosingreplicationproblems.
Thefollowingexamplewillsetmostofthediscussedpropertiesforthecurrentdatabase'sapplier:
require("@arangodb/replication").applier.properties({
endpoint:"tcp://master.domain.org:8529",
username:"root",
password:"secret",
adaptivePolling:true,
connectTimeout:15,
maxConnectRetries:100,
chunkSize:262144,
autoStart:true,
includeSystem:true,
autoResync:true,
autoResyncRetries:2,
});
Aftertheapplierisnowfullyconfigured,itcouldtheoreticallybestarted.However,wemayfirstneedaninitialsynchronizationofallcollectionsandtheirdatafromthemasterbeforewestartthereplicationapplier.
Theonlysafemethodfordoingafullsynchronization(orre-synchronization)isthusto
stopthereplicationapplierontheslave(ifcurrentlyrunning)performaninitialfullsyncwiththemasterdatabasenotethemasterdatabase'slastLogTickvalueand
AsynchronousReplication
476
startthecontinuousreplicationapplierontheslaveusingthistickvalue.
Theinitialsynchronizationforthecurrentdatabaseisexecutedwiththesynccommand:
require("@arangodb/replication").sync({
endpoint:"tcp://master.domain.org:8529",
username:"root",
password:"secret,
includeSystem:true
});
TheincludeSystemoptioncontrolswhetherdatafromsystemcollections(suchas_graphsand_users)shallbesynchronized.
TheinitialsynchronizationcanoptionallybeconfiguredtoincludeorexcludespecificcollectionsusingtherestrictTypeandrestrictCollectionparameters.
Thefollowingcommandonlysynchronizescollectionfooandbar:
require("@arangodb/replication").sync({
endpoint:"tcp://master.domain.org:8529",
username:"root",
password:"secret,
restrictType:"include",
restrictCollections:["foo","bar"]
});
UsingarestrictTypeofexclude,allcollectionsbutthespecifiedwillbesynchronized.
Warning:syncwilldoafullsynchronizationofthecollectionsinthecurrentdatabasewithcollectionspresentinthemasterdatabase.Anylocalinstancesofthecollectionsandalltheirdataareremoved!Onlyexecutethiscommandifyouaresureyouwanttoremovethelocaldata!
Assyncdoesafullsynchronization,itmighttakeawhiletoexecute.Whensynccompletessuccessfully,itreturnsanarrayofcollectionsithassynchronizedinitscollectionsattribute.Itwillalsoreturnthemasterdatabase'slastlogtickvalueatthetimethesyncwasstartedonthemaster.ThetickvalueiscontainedinthelastLogTickattributeofthesynccommand:
{
"lastLogTick":"231848833079705",
"collections":[...]
}
Nowyoucanstartthecontinuoussynchronizationforthecurrentdatabaseontheslavewiththecommand
require("@arangodb/replication").applier.start("231848833079705");
Note:Thetickvaluesshouldbetreatedasstrings.Usingnumericdatatypesfortickvaluesisunsafebecausetheymightexceedthe32bitvalueandtheIEEE754doubleaccuracyranges.
AsynchronousReplication
477
Per-DatabaseSetupThispagedescribesthereplicationprocessbasedonaspecificdatabasewithinanArangoDBinstance.Thatmeansthatonlythespecifieddatabasewillbereplicated.
Settingupaworkingmaster-slavereplicationrequirestwoArangoDBinstances:
master:thisistheinstancethatalldata-modificationoperationsshouldbedirectedtoslave:onthisinstance,we'llstartareplicationapplier,andthiswillfetchdatafromthemasterdatabase'swrite-aheadlogandapplyitsoperationslocally
Forthefollowingexamplesetup,we'llusetheinstancetcp://master.domain.org:8529asthemaster,andtheinstancetcp://slave.domain.org:8530asaslave.
Thegoalistohavealldatafromthedatabase_systemonmastertcp://master.domain.org:8529bereplicatedtothedatabase_systemontheslavetcp://slave.domain.org:8530.
Onthemaster,nothingspecialneedstobedone,asallwriteoperationswillautomaticallybeloggedinthemaster'swrite-aheadlog(WAL).
All-in-onesetup
Tomakethereplicationcopytheinitialdatafromthemastertotheslaveandstartthecontinuousreplicationontheslave,thereisanall-in-onecommand:
require("@arangodb/replication").setupReplication(configuration);
Thefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthe_systemdatabase.Notethatitshouldberunontheslaveandnotthemaster:
db._useDatabase("_system");
require("@arangodb/replication").setupReplication({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
verbose:false,
includeSystem:false,
incremental:true,
autoResync:true
});
Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationhasbeenstarted,orincasetheinitialsynchronizationhasfailed.
Iftheinitialsynchronizationissuccessful,thecommandwillstorethegivenconfigurationontheslave.Italsoconfiguresthecontinuousreplicationtostartautomaticallyiftheslaveisrestarted,i.e.autoStartissettotrue.
Ifthecommandisrunwhiletheslave'sreplicationapplierisalreadyrunning,itwillfirststoptherunningapplier,dropitsconfigurationanddoaresynchronizationofdatawiththemaster.Itwillthenusetheprovidedconfigration,overwritinganypreviouslyexistingreplicationconfigurationontheslave.
Initialsynchronization
Theinitialsynchronizationandcontinuousreplicationappliercanalsobestartedseparately.Tostartreplicationontheslave,makesuretherecurrentlyisnoreplicationapplierrunning.
Thefollowingcommandsstoparunningapplierintheslave's_systemdatabase:
db._useDatabase("_system");
AsynchronousReplication
478
require("@arangodb/replication").applier.stop();
Thestopoperationwillterminateanyreplicationactivityinthe_systemdatabaseontheslave.
Afterthat,theinitialsynchronizationcanberun.Itwillcopythecollectionsfromthemastertotheslave,overwritingexistingdata.Toruntheinitialsynchronization,executethefollowingcommandsontheslave:
db._useDatabase("_system");
require("@arangodb/replication").sync({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
verbose:false
});
Usernameandpasswordonlyneedtobespecifiedwhenthemasterrequiresauthentication.Tocheckwhatthesynchronizationiscurrentlydoing,supplysettheverboseoptiontotrue.Ifset,thesynchronizationwillcreatelogmessageswiththecurrentsynchronizationstatus.
Warning:Thesynccommandwillreplacedataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!
ThesyncoperationwillreturnanattributenamedlastLogTickwhichwe'llneedtonote.Thelastlogtickwillbeusedasthestartingpointforsubsequentreplicationactivity.Let'sassumewegotthefollowinglastlogtick:
{
"lastLogTick":"40694126",
...
}
InitialsynchronizationfromtheArangoShell
Theinitialsynchronizationviathesynccommandmaytakealongtimetocomplete.Theshellwillblockuntiltheslavehascompletedtheinitialsynchronizationoruntilanerroroccurs.Bydefault,thesynccommandintheArangoShellwillpolltheslaveforastatusupdateevery10seconds.
Optionallythesynccommandcanbemadenon-blockingbysettingitsasyncoptiontotrue.Inthiscase,thesynccommandwillreturninstantlywithanidstring,andtheinitialsynchronizationwillrundetachedonthemaster.TofetchthecurrentstatusofthesyncprogressfromtheArangoShell,thegetSyncResultfunctioncanbeusedasfollows:
db._useDatabase("_system");
varreplication=require("@arangodb/replication");
/*runcommandinasyncmode*/
varid=replication.sync({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
async:true
});
/*nowquerythestatusofouroperation*/
print(replication.getSyncResult(id));
getSyncResultwillreturnfalseaslongasthesynchronizationisnotcomplete,andreturnthesynchronizationresultotherwise.
Continuoussynchronization
Whentheinitialsynchronizationisfinished,thecontinuousreplicationappliercanbestartedusingthelastlogtickprovidedbythesynccommand.Beforestartingit,thereisatleastoneconfigurationoptiontoconsider:replicationontheslavewillberunninguntiltheslavegetsshutdown.Whentheslaveservergetsrestarted,replicationwillbeturnedoffagain.Tochangethis,wefirstneedtoconfiguretheslave'sreplicationapplierandsetitsautoStartattribute.
AsynchronousReplication
479
Here'sthecommandtoconfigurethereplicationapplierwithseveraloptions,includingtheautoStartattribute:
db._useDatabase("_system");
require("@arangodb/replication").applier.properties({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
autoStart:true,
autoResync:true,
autoResyncRetries:2,
adaptivePolling:true,
includeSystem:false,
requireFromPresent:false,
idleMinWaitTime:0.5,
idleMaxWaitTime:1.5,
verbose:false
});
Animportantconsiderationforreplicationiswhetherdatafromsystemcollections(suchas_graphsor_users)shouldbeapplied.TheincludeSystemoptioncontrolsthat.Ifsettotrue,changesinsystemcollectionswillbereplicated.Otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.
TherequireFromPresentattributecontrolswhethertheapplierwillstartsynchronizingincaseitdetectsthatthemastercannotprovidedatafortheinitialtickvalueprovidedbytheslave.ThismaybethecaseifthemasterdoesnothaveabigenoughbacklogofhistoricWALlogfiles,andwhenthereplicationisre-startedafteralongerpause.WhenrequireFromPresentissettotrue,thenthereplicationapplierwillcheckatstartwhetherthestarttickfromwhichitstartsorresumesreplicationisstillpresentonthemaster.Ifnot,thentherewouldbedataloss.IfrequireFromPresentistrue,thereplicationapplierwillabortwithanappropriateerrormessage.Ifsettofalse,thenthereplicationapplierwillstillstart,andignorethedataloss.
TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslavehadrequested,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.
Notethatautomaticre-synchronization(autoResyncoptionsettotrue)maytransferalotofdatafromthemastertotheslaveandcanthereforebeexpensive.Stillit'sturnedonheresothere'slessneedformanualintervention.
TheautoResyncRetriesoptioncanbeusedtocontrolthenumberofresynchronizationretriesthatwillbeperformedinarowwhenautomaticresynchronizationisenabledandkicksin.Settingthisto0willeffectivelydisableautoResync.Settingittosomeothervaluewilllimitthenumberofretriesthatareperformed.Thishelpspreventingendlessretriesincaseresynchronizationsalwaysfail.
Nowit'stimetostartthereplicationapplierontheslaveusingthelastlogtickwegotbefore:
db._useDatabase("_system");
require("@arangodb/replication").applier.start("40694126");
Thiswillreplicatealloperationshappeninginthemaster'ssystemdatabaseandapplythemontheslave,too.
Afterthat,youshouldbeabletomonitorthestateandprogressofthereplicationapplierbyexecutingthestatecommandontheslaveserver:
db._useDatabase("_system");
require("@arangodb/replication").applier.state();
Pleasenotethatstoppingthereplicationapplierontheslaveusingthestopcommandshouldbeavoided.Thereasonisthatcurrentlyongoingtransactions(thathavepartlybeenreplicatedtotheslave)willbeneedtoberestartedafterarestartofthereplicationapplier.Stoppingandrestartingthereplicationapplierontheslaveshouldthusonlybeperformedifthereiscertaintythatthemasteriscurrentlyfullyidleandalltransactionshavebeenreplicatedfully.
Notethatwhileaslavehasonlypartlyexecutedatransactionfromthemaster,itmightkeepawritelockonthecollectionsinvolvedinthetransaction.
AsynchronousReplication
480
YoumayalsowanttocheckthemasterandslavestatesviatheHTTPAPIs(seeHTTPInterfaceforReplication).
AsynchronousReplication
481
Server-levelSetupThispagedescribesthereplicationprocessbasedonacompleteArangoDBinstance.Thatmeansthatallincludeddatabaseswillbereplicated.
Settingupaworkingmaster-slavereplicationrequirestwoArangoDBinstances:
master:thisistheinstancethatalldata-modificationoperationsshouldbedirectedtoslave:onthisinstance,we'llstartareplicationapplier,andthiswillfetchdatafromthemasterdatabase'swrite-aheadlogandapplyitsoperationslocally
Forthefollowingexamplesetup,we'llusetheinstancetcp://master.domain.org:8529asthemaster,andtheinstancetcp://slave.domain.org:8530asaslave.
Thegoalistohavealldataofalldatabasesonmastertcp://master.domain.org:8529bereplicatedtotheslaveinstancetcp://slave.domain.org:8530.
Onthemaster,nothingspecialneedstobedone,asallwriteoperationswillautomaticallybeloggedinthemaster'swrite-aheadlog(WAL).
All-in-onesetup
Tomakethereplicationcopytheinitialdatafromthemastertotheslaveandstartthecontinuousreplicationontheslave,thereisanall-in-onecommand:
require("@arangodb/replication").setupReplicationGlobal(configuration);
ThefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthecompleteArangoDBinstance.Notethatitshouldberunontheslaveandnotthemaster:
db._useDatabase("_system");
require("@arangodb/replication").setupReplicationGlobal({
endpoint:"tcp://127.0.0.1:8529",
username:"root",
password:"",
autoStart:true
});
Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationhasbeenstarted,orincasetheinitialsynchronizationhasfailed.
Iftheinitialsynchronizationissuccessful,thecommandwillstorethegivenconfigurationontheslave.Italsoconfiguresthecontinuousreplicationtostartautomaticallyiftheslaveisrestarted,i.e.autoStartissettotrue.
Ifthecommandisrunwhiletheslave'sreplicationapplierisalreadyrunning,itwillfirststoptherunningapplier,dropitsconfigurationanddoaresynchronizationofdatawiththemaster.Itwillthenusetheprovidedconfigration,overwritinganypreviouslyexistingreplicationconfigurationontheslave.
Stoppingsynchronization
Theinitialsynchronizationandcontinuousreplicationappliercanalsobestartedseparately.Tostartreplicationontheslave,makesuretherecurrentlyisnoreplicationapplierrunning.
Thefollowingcommandsstoparunningapplierintheslave'sinstance:
db._useDatabase("_system");
require("@arangodb/replication").globalApplier.stop();
AsynchronousReplication
482
ThestopoperationwillterminateanyreplicationactivityintheArangoDBinstanceontheslave.
Afterthat,theinitialsynchronizationcanberun.Itwillcopythecollectionsfromthemastertotheslave,overwritingexistingdata.Toruntheinitialsynchronization,executethefollowingcommandsontheslave:
db._useDatabase("_system");
require("@arangodb/replication").syncGlobal({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
verbose:false
});
Usernameandpasswordonlyneedtobespecifiedwhenthemasterrequiresauthentication.Tocheckwhatthesynchronizationiscurrentlydoing,supplysettheverboseoptiontotrue.Ifset,thesynchronizationwillcreatelogmessageswiththecurrentsynchronizationstatus.
Warning:ThesyncGlobalcommandwillreplacedataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!
ThesyncoperationwillreturnanattributenamedlastLogTickwhichwe'llneedtonote.Thelastlogtickwillbeusedasthestartingpointforsubsequentreplicationactivity.Let'sassumewegotthefollowinglastlogtick:
{
"lastLogTick":"40694126",
...
}
InitialsynchronizationfromtheArangoShellTheinitialsynchronizationviathesyncGlobalcommandmaytakealongtimetocomplete.Theshellwillblockuntiltheslavehascompletedtheinitialsynchronizationoruntilanerroroccurs.Bydefault,thesyncGlobalcommandintheArangoShellwillpolltheslaveforastatusupdateevery10seconds.
OptionallythesyncGlobalcommandcanbemadenon-blockingbysettingitsasyncoptiontotrue.Inthiscase,thesyncGlobalcommandwillreturninstantlywithanidstring,andtheinitialsynchronizationwillrundetachedonthemaster.TofetchthecurrentstatusofthesyncGlobalprogressfromtheArangoShell,thegetSyncResultfunctioncanbeusedasfollows:
db._useDatabase("_system");
varreplication=require("@arangodb/replication");
/*runcommandinasyncmode*/
varid=replication.syncGlobal({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
async:true
});
/*nowquerythestatusofouroperation*/
print(replication.getSyncResult(id));
getSyncResultwillreturnfalseaslongasthesynchronizationisnotcomplete,andreturnthesynchronizationresultotherwise.
Continuoussynchronization
Whentheinitialsynchronizationisfinished,thecontinuousreplicationappliercanbestartedusingthelastlogtickprovidedbythesyncGlobalcommand.Beforestartingit,thereisatleastoneconfigurationoptiontoconsider:replicationontheslavewillberunninguntiltheslavegetsshutdown.Whentheslaveservergetsrestarted,replicationwillbeturnedoffagain.Tochangethis,wefirstneedtoconfiguretheslave'sreplicationapplierandsetitsautoStartattribute.
Here'sthecommandtoconfigurethereplicationapplierwithseveraloptions,includingtheautoStartattribute:
AsynchronousReplication
483
db._useDatabase("_system");
require("@arangodb/replication").globalApplier.properties({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
autoStart:true,
autoResync:true,
autoResyncRetries:2,
adaptivePolling:true,
includeSystem:false,
requireFromPresent:false,
idleMinWaitTime:0.5,
idleMaxWaitTime:1.5,
verbose:false
});
Animportantconsiderationforreplicationiswhetherdatafromsystemcollections(suchas_graphsor_users)shouldbeapplied.TheincludeSystemoptioncontrolsthat.Ifsettotrue,changesinsystemcollectionswillbereplicated.Otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.
TherequireFromPresentattributecontrolswhethertheapplierwillstartsynchronizingincaseitdetectsthatthemastercannotprovidedatafortheinitialtickvalueprovidedbytheslave.ThismaybethecaseifthemasterdoesnothaveabigenoughbacklogofhistoricWALlogfiles,andwhenthereplicationisre-startedafteralongerpause.WhenrequireFromPresentissettotrue,thenthereplicationapplierwillcheckatstartwhetherthestarttickfromwhichitstartsorresumesreplicationisstillpresentonthemaster.Ifnot,thentherewouldbedataloss.IfrequireFromPresentistrue,thereplicationapplierwillabortwithanappropriateerrormessage.Ifsettofalse,thenthereplicationapplierwillstillstart,andignorethedataloss.
TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslavehadrequested,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.
Notethatautomaticre-synchronization(autoResyncoptionsettotrue)maytransferalotofdatafromthemastertotheslaveandcanthereforebeexpensive.Stillit'sturnedonheresothere'slessneedformanualintervention.
TheautoResyncRetriesoptioncanbeusedtocontrolthenumberofresynchronizationretriesthatwillbeperformedinarowwhenautomaticresynchronizationisenabledandkicksin.Settingthisto0willeffectivelydisableautoResync.Settingittosomeothervaluewilllimitthenumberofretriesthatareperformed.Thishelpspreventingendlessretriesincaseresynchronizationsalwaysfail.
Nowit'stimetostartthereplicationapplierontheslaveusingthelastlogtickwegotbefore:
db._useDatabase("_system");
require("@arangodb/replication").globalApplier.start("40694126");
Thiswillreplicatealloperationshappeninginthemaster'ssystemdatabaseandapplythemontheslave,too.
Afterthat,youshouldbeabletomonitorthestateandprogressofthereplicationapplierbyexecutingthestatecommandontheslaveserver:
db._useDatabase("_system");
require("@arangodb/replication").globalApplier.state();
Pleasenotethatstoppingthereplicationapplierontheslaveusingthestopcommandshouldbeavoided.Thereasonisthatcurrentlyongoingtransactions(thathavepartlybeenreplicatedtotheslave)willbeneedtoberestartedafterarestartofthereplicationapplier.Stoppingandrestartingthereplicationapplierontheslaveshouldthusonlybeperformedifthereiscertaintythatthemasteriscurrentlyfullyidleandalltransactionshavebeenreplicatedfully.
Notethatwhileaslavehasonlypartlyexecutedatransactionfromthemaster,itmightkeepawritelockonthecollectionsinvolvedinthetransaction.
YoumayalsowanttocheckthemasterandslavestatesviatheHTTPAPIs(seeHTTPInterfaceforReplication).
AsynchronousReplication
484
SyncingCollectionsInordertosynchronizedataforasinglecollectionfromamastertoaslaveinstance,thereisthesyncCollectionfunction:
Itwillfetchalldocumentsofthespecifiedcollectionfromthemasterdatabaseandstoretheminthelocalinstance.Afterthesynchronization,thecollectiondataontheslavewillbeidenticaltothedataonthemaster,providednofurtherdatachangeshappenonthemaster.AnydatachangesthatareperformedonthemasterafterthesynchronizationwasstartedwillnotbecapturedbysyncCollection,butneedtobereplicatedusingtheregularreplicationappliermechanism.
Forthefollowingexamplesetup,we'llusetheinstancetcp://master.domain.org:8529asthemaster,andtheinstancetcp://slave.domain.org:8530asaslave.
Thegoalistohavealldatafromthecollectiontestindatabase_systemonmastertcp://master.domain.org:8529bereplicatedtothecollectiontestindatabase_systemontheslavetcp://slave.domain.org:8530.
Onthemaster,thecollectiontestneedstobepresentinthe_systemdatabase,withanydatainit.
Totransferthiscollectiontotheslave,issuethefollowingcommandsthere:
db._useDatabase("_system");
require("@arangodb/replication").syncCollection("test",{
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd"
});
Warning:ThesyncCollectioncommandwillreplacethecollection'sdataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!
SettingtheoptionalincrementalattributeinthecalltosyncCollectionwillstartanincrementaltransferofdata.Thismaybeusefulincasewhentheslavealreadyhaspartsoralmostallofthedatainthecollectionandonlythedifferencesneedtobesynchronized.Notethattocomputethedifferencestheincrementaltransferwillbuildasortedlistofalldocumentkeysinthecollectiononboththeslaveandthemaster,whichmaystillbeexpensiveforhugecollectionsintermsofmemoryusageandruntime.Duringbuildingthelistofkeysthecollectionwillberead-lockedonthemaster.
TheinitialSyncMaxWaitTimeattributeinthecalltosyncCollectioncontrolshowlongtheslavewillwaitforamaster'sresponse.Thiswaittimecanbeusedtocontrolafterwhattimethesynchronizationwillgiveupandfail.
ThesyncCollectioncommandmaytakealongtimetocompleteifthecollectionisbig.Theshellwillblockuntiltheslavehassynchronizedtheentirecollectionfromthemasteroruntilanerroroccurs.Bydefault,thesyncCollectioncommandintheArangoShellwillpollforastatusupdateevery10seconds.
WhensyncCollectioniscalledfromtheArangoShell,theoptionalasyncattributecanbeusedtostartthesynchronizationasabackgroundprocessontheslave.Ifasyncissettotrue,thecalltosyncCollectionwillreturnalmostinstantlywithanidstring.Usingthisidstring,thestatusofthesyncjobontheslavecanbequeriedusingthegetSyncResultfunctionasfollows:
db._useDatabase("_system");
varreplication=require("@arangodb/replication");
/*runcommandinasyncmode*/
varid=replication.syncCollection("test",{
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
async:true
});
/*nowquerythestatusofouroperation*/
print(replication.getSyncResult(id));
getSyncResultwillreturnfalseaslongasthesynchronizationisnotcomplete,andreturnthesynchronizationresultotherwise.
AsynchronousReplication
486
ReplicationLimitationsThereplicationinArangoDBhasafewlimitations.SomeoftheselimitationsmayberemovedinlaterversionsofArangoDB:
thereisnofeedbackfromtheslavestothemaster.Ifaslavecannotapplyaneventitgotfromthemaster,themasterwillhaveadifferentstateofdata.Inthiscase,thereplicationapplierontheslavewillstopandreportanerror.Administratorscantheneither"fix"theproblemorre-syncthedatafromthemastertotheslaveandstarttheapplieragain.atthemomentitisassumedthatonlythereplicationapplierexecuteswriteoperationsonaslave.ArangoDBcurrentlydoesnotpreventusersfromcarryingouttheirownwriteoperationsonslaves,thoughthismightleadtoundefinedbehaviorandthereplicationapplierstopping.whenareplicationslaveasksamasterforlogevents,thereplicationmasterwillreturnallwriteoperationsforuser-definedcollections,butitwillexcludewriteoperationsforcertainsystemcollections.Thefollowingcollectionsareexcludedintentionallyfromreplication:_apps,_trx,_replication,_configuration,_jobs,_queues,_sessions,_foxxlogandallstatisticscollections.Writeoperationsforthefollowingsystemcollectionscanbequeriedfromamaster:_aqlfunctions,_graphs,_users.Foxxapplicationsconsistofdatabaseentriesandapplicationscriptsinthefilesystem.ThefilesystempartsofFoxxapplicationsarenottrackedanywhereandthusnotreplicatedincurrentversionsofArangoDB.ToreplicateaFoxxapplication,itisrequiredtocopytheapplicationtotheremoteserverandinstallitthereusingthefoxx-managerutility.masterserversdonotknowwhichslavesareorwillbeconnectedtothem.Allserversinareplicationsetuparecurrentlyonlylooselycoupled.Therecurrentlyisnowayforaclienttoquerywhichserversarepresentinareplication.whennotusingourmesosintegrationfailovermustbehandledbyclientsorclientAPIs.therecurrentlyisonereplicationapplierperArangoDBdatabase.Itisthusnotpossibletohaveaslaveapplyoperationsfrommultiplemastersintothesametargetdatabase.replicationissetuponaper-databaselevel.WhenusingArangoDBwithmultipledatabases,replicationmustbeconfiguredindividuallyforeachdatabase.thereplicationapplierissingle-threaded,butwriteoperationsonthemastermaybeexecutedinparalleliftheyaffectdifferentcollections.Thusthereplicationappliermightnotbeabletocatchupwithaverypowerfulandloadedmaster.replicationisonlysupportedbetweenthetwoArangoDBserversrunningthesameArangoDBversion.ItiscurrentlynotpossibletoreplicatebetweendifferentArangoDBversions.areplicationappliercannotapplydatafromitself.
AsynchronousReplication
488
SynchronousReplicationAtitscoresynchronousreplicationwillreplicatewriteoperationstomultiplehosts.ThisfeatureisonlyavailablewhenoperatingArangoDBinacluster.Wheneveracoordinatorexecutesasychronouslyreplicatedwriteoperationitwillonlybereportedtobesuccessfulifitwascarriedoutonallreplicas.IncontrasttomultimasterreplicationsetupsknownfromothersystemsArangoDB'ssynchronousoperationguaranteesaconsistentstateacrossthecluster.
SynchronousReplication
489
Implementation
Architectureinsidethecluster
SynchronousreplicationcanbeconfiguredpercollectionviathepropertyreplicationFactor.Synchronousreplicationrequiresaclustertooperate.
WheneveryouspecifyareplicationFactorgreaterthan1whencreatingacollection,synchronousreplicationwillbeactivatedforthiscollection.Theclusterwilldeterminesuitableleadersandfollowersforeveryrequestedshard(numberOfShards)withinthecluster.Whenrequestingdataofashardonlythecurrentleaderwillbeaskedwhereasfollowerswillonlykeeptheircopyinsync.Thisisduetothecurrentimplementationoftransactions.
Usingsynchronousreplicationalonewillguaranteeconsistencyandhighavailabiltyatthecostofreducedperformance:Writerequestswillhaveahigherlatency(duetoeverywrite-requesthavingtobeexecutedonthefollowers)andreadrequestswon'tscaleoutasonlytheleaderisbeingasked.
Inaclustersynchronousreplicationwillbemanagedbythecoordinatorsfortheclient.Thedatawillalwaysbestoredonprimaries.
ThefollowingexamplewillgiveyouanideaofhowsynchronousoperationhasbeenimplementedinArangoDB.
1. Connecttoacoordinatorviaarangosh2. Createacollection
127.0.0.1:8530@_system>db._create("test",{"replicationFactor":2})
3. thecoordinatorwillfigureoutaleaderand1followerandcreate1shard(asthisisthedefault)
4. Insertdata
127.0.0.1:8530@_system>db.test.insert({"replication":"ᚇ"})
5. Thecoordinatorwillwritethedatatotheleader,whichinturnwillreplicateittothefollower.
6. Onlywhenbothweresuccessfultheresultisreportedtobesuccessful
{
"_id":"test/7987",
"_key":"7987",
"_rev":"7987"
}
Whenafollowerfails,theleaderwillgiveuponitafter3secondsandproceedwiththeoperation.Assoonasthefollower(orthenetworkconnectiontotheleader)isbackup,thetwowillresynchronizeandsynchronousreplicationisresumed.Thishappensalltransparentlytotheclient.
ThecurrentimplementationofArangoDBdoesnotallowchangingthereplicationFactorlater.Thisissubjecttochange.Inthemeantimetheonlywayistodumpandrestorethecollection.Seethecookbookrecipeaboutmigrating.
Automaticfailover
Whenevertheleaderofashardisfailingandthereisaquerytryingtoaccessdataofthatshardthecoordinatorwillcontinuetryingtocontacttheleaderuntilittimeouts.Theinternalclustersupervisionrunningontheagencywillcheckclusterhealtheveryfewsecondsandwilltakeactionifthereisnoheartbeatfromaserverfor15seconds.Iftheleaderdoesn'tcomebackintimethesupervisionwillreorganizetheclusterbypromotingforeachshardafollowerthatisinsyncwithitsleadertobethenewleader.Fromthenonthecoordinatorswillcontactthenewleader.
Theprocessisbestoutlinedusinganexample:
1. Theleaderofashard(letsnameitDBServer001)isgoingdown.2. Acoordinatorisaskedtoreturnadocument:
SynchronousReplication
490
127.0.0.1:8530@_system>db.test.document("100069")
3. ThecoordinatordetermineswhichserverisresponsibleforthisdocumentandfindsDBServer001
4. ThecoordinatortriestocontactDBServer001andtimeoutsbecauseitisnotreachable.5. Afterashortwhilethesupervision(runninginparallelontheagency)willseethatheartbeatsfromDBServer001arenotcomingin6. Thesupervisionpromotesoneofthefollowers(sayDBServer002)thatisinsynctobeleaderandmakesDBServer001afollower.7. AsthecoordinatorcontinuestryingtofetchthedocumentitwillseethattheleaderchangedtoDBServer0028. Thecoordinatortriestocontactthenewleader(DBServer002)andreturnstheresult:
{
"_key":"100069",
"_id":"test/100069",
"_rev":"513",
"replication":"ᚇ"
}
9. AfterawhilethesupervisiondeclaresDBServer001tobecompletelydead.10. AnewfollowerisdeterminedfromthepoolofDBservers.11. Thenewfollowersyncsitsdatafromtheleaderandorderisrestored.
Pleasenotethattheremaystillbetimeouts.Dependingonwhenexactlytherequesthasbeendone(inregardtothesupervision)anddependingonthetimeneededtoreconfiguretheclusterthecoordinatormightfailwithatimeouterror!
SynchronousReplication
491
Configuration
Requirements
SynchronousreplicationrequiresanoperationalArangoDBcluster.
Enablingsynchronousreplication
Synchronousreplicationcanbeenabledpercollection.WhencreatingacollectionyoumayspecifythenumberofreplicasusingthereplicationFactorparameter.Thedefaultvalueissetto1whicheffectivelydisablessynchronousreplication.
Example:
127.0.0.1:8530@_system>db._create("test",{"replicationFactor":3})
Intheabovecase,anywriteoperationwillrequire2replicastoreportsuccessfromnowon.
Preparinggrowth
Youmaycreateacollectionwithhigherreplicationfactorthanavailable.Whenadditionaldbserversbecomeavailabletheshardsareautomaticallyreplicatedtothenewlyavailablemachines.
Multiplereplicasofthesameshardcannevercoexistonthesamedbserverinstance.
SynchronousReplication
492
SatelliteCollectionsSatelliteCollectionsareanEnterpriseonlyfeature.WhendoingJoinsinanArangoDBclusterdatahastoexchangedbetweendifferentservers.
Joinswillbeexecutedonacoordinator.Itwillprepareanexecutionplanandexecuteit.Whenexecutingthecoordinatorwillcontactallshardsofthestartingpointofthejoinandaskfortheirdata.Thedatabaseserverscarryingoutthisoperationwillloadalltheirlocaldataandthenasktheclusterfortheotherpartofthejoin.Thisagainwillbedistributedtoallinvolvedshardsofthisjoinpart.
Insumthisresultsinmuchnetworktrafficandslowresultsdependingoftheamountofdatathathastobesentthroughoutthecluster.
Satellitecollectionsarecollectionsthatareintendedtoaddressthisissue.
Theywillfacilitatethesynchronousreplicationandreplicateallitsdatatoalldatabaseserversthatarepartofthecluster.
ThisenablesthedatabaseserverstoexecutethatpartofanyJoinlocally.
Thisgreatlyimprovesperformanceforsuchjoinsatthecostsofincreasedstoragerequirementsandpoorerwriteperformanceonthisdata.
TocreateasatellitecollectionsetthereplicationFactorofthiscollectionto"satellite".
Usingarangosh:
arangosh>db._create("satellite",{"replicationFactor":"satellite"});
Afullexample
arangosh>varexplain=require("@arangodb/aql/explainer").explain
arangosh>db._create("satellite",{"replicationFactor":"satellite"})
arangosh>db._create("nonsatellite",{numberOfShards:8})
arangosh>db._create("nonsatellite2",{numberOfShards:8})
Let'sanalyseanormaljoinnotinvolvingsatellitecollections:
arangosh>explain("FORdocinnonsatelliteFORdoc2innonsatellite2RETURN1")
Querystring:
FORdocinnonsatelliteFORdoc2innonsatellite2RETURN1
Executionplan:
IdNodeTypeSiteEst.Comment
1SingletonNodeDBS1*ROOT
4CalculationNodeDBS1-LET#2=1/*jsonexpression*//*constassignment*/
2EnumerateCollectionNodeDBS0-FORdocINnonsatellite/*fullcollectionscan*/
12RemoteNodeCOOR0-REMOTE
13GatherNodeCOOR0-GATHER
6ScatterNodeCOOR0-SCATTER
7RemoteNodeDBS0-REMOTE
3EnumerateCollectionNodeDBS0-FORdoc2INnonsatellite2/*fullcollectionscan*/
8RemoteNodeCOOR0-REMOTE
9GatherNodeCOOR0-GATHER
5ReturnNodeCOOR0-RETURN#2
Indexesused:
none
Optimizationrulesapplied:
IdRuleName
1move-calculations-up
2scatter-in-cluster
3remove-unnecessary-remote-scatter
SatelliteCollections
493
Allshardsinvolvedqueryingthenonsatellitecollectionwillfanoutviathecoordinatortotheshardsofnonsatellite.Insum8shardswillopen8connectionstothecoordinatoraskingfortheresultsofthenonsatellite2join.Thecoordinatorwillfanouttothe8shardsofnonsatellite2.Sotherewillbequitesomenetworktraffic.
Let'snowhavealookatthesameusingsatellitecollections:
arangosh>db._query("FORdocinnonsatelliteFORdoc2insatelliteRETURN1")
Querystring:
FORdocinnonsatelliteFORdoc2insatelliteRETURN1
Executionplan:
IdNodeTypeSiteEst.Comment
1SingletonNodeDBS1*ROOT
4CalculationNodeDBS1-LET#2=1/*jsonexpression*//*constassignment*/
2EnumerateCollectionNodeDBS0-FORdocINnonsatellite/*fullcollectionscan*/
3EnumerateCollectionNodeDBS0-FORdoc2INsatellite/*fullcollectionscan,satellite*/
8RemoteNodeCOOR0-REMOTE
9GatherNodeCOOR0-GATHER
5ReturnNodeCOOR0-RETURN#2
Indexesused:
none
Optimizationrulesapplied:
IdRuleName
1move-calculations-up
2scatter-in-cluster
3remove-unnecessary-remote-scatter
4remove-satellite-joins
Inthisscenarioallshardsofnonsatellitewillbecontacted.Howeverasthejoinisasatellitejoinallshardscandothejoinlocallyasthedataisreplicatedtoallserversreducingthenetworkoverheaddramatically.
Caveats
Theclusterwillautomaticallykeepallsatellitecollectionsonallserversinsyncbyfacilitatingthesynchronousreplication.Thismeansthatwritewillbeexecutedontheleaderonlyandthisserverwillcoordinatereplicationtothefollowers.Ifafollowerdoesn'tanswerintime(duetonetworkproblems,temporaryshutdownetc.)itmayberemovedasafollower.ThisisbeingreportedtotheAgency.
Thefollower(oncebackinbusiness)willthenperiodicallychecktheAgencyandknowthatitisoutofsync.Itwillthenautomaticallycatchup.Thismaytakeawhiledependingonhowmuchdatahastobesynced.WhendoingajoininvolvingthesatelliteyoucanspecifyhowlongtheDBServerisallowedtowaitforsyncuntilthequeryisbeingaborted.
CheckAccessingCursorsfordetails.
DuringnetworkfailurethereisalsoaminimalchancethataquerywasproperlydistributedtotheDBServersbutthataprevioussatellitewritecouldnotbereplicatedtoafollowerandtheleaderdroppedthefollower.Thefollowerhoweveronlycheckseveryfewsecondsifitisreallyinsyncsoitmightindeeddeliverstaleresults.
SatelliteCollections
494
ShardingArangoDBisorganizingitscollectiondatainshards.ShardingallowstousemultiplemachinestorunaclusterofArangoDBinstancesthattogetherconstituteasingledatabase.Thisenablesyoutostoremuchmoredata,sinceArangoDBdistributesthedataautomaticallytothedifferentservers.Inmanysituationsonecanalsoreapabenefitindatathroughput,againbecausetheloadcanbedistributedtomultiplemachines.
Shardsareconfiguredpercollectionsomultipleshardsofdataformthecollectionasawhole.TodetermineinwhichshardthedataistobestoredArangoDBperformsahashacrossthevalues.Bydefaultthishashisbeingcreatedfrom_key.
Toconfigurethenumberofshards:
127.0.0.1:8529@_system>db._create("sharded_collection",{"numberOfShards":4});
Toconfigurethehashingforanotherattribute:
127.0.0.1:8529@_system>db._create("sharded_collection",{"numberOfShards":4,"shardKeys":["country"]});
Thiswouldbeusefultokeepdataofeverycountryinoneshardwhichwouldresultinbetterperformanceforqueriesworkingonapercountrybase.YoucanalsospecifymultipleshardKeys.Notehoweverthatifyouchangetheshardkeysfromtheirdefault["_key"],thenfindingadocumentinthecollectionbyitsprimarykeyinvolvesarequesttoeverysingleshard.Furthermore,inthiscaseonecannolongerprescribetheprimarykeyvalueofanewdocumentbutmustusetheautomaticallygeneratedone.Thislatterrestrictioncomesfromthefactthatensuringuniquenessoftheprimarykeywouldbeveryinefficientiftheusercouldspecifytheprimarykey.
Onwhichnodeinaclusteraparticularshardiskeptisundefined.Thereisnooptiontoconfigureanaffinitybasedoncertainshardkeys.
Uniqueindexes(hash,skiplist,persistent)onshardedcollectionsareonlyallowedifthefieldsusedtodeterminetheshardkeyarealsoincludedinthelistofattributepathsfortheindex:
shardKeys indexKeys
a a ok
a b notok
a a,b ok
a,b a notok
a,b b notok
a,b a,b ok
a,b a,b,c ok
a,b,c a,b notok
a,b,c a,b,c ok
Sharding
495
GeneralUpgradeInformation
Recommendedmajorupgradeprocedure
ToupgradeanexistingArangoDB2.xto3.0pleaseusetheproceduredescribedhere.
Recommendedminorupgradeprocedure
ToupgradeanexistingArangoDBdatabasetoanewerversionofArangoDB(e.g.3.0to3.1,or3.1to3.2),thefollowingmethodisrecommended:
ChecktheCHANGELOGandthelistofincompatiblechangesforAPIorotherchangesinthenewversionofArangoDBandmakesureyourapplicationscandealwiththemStopthe"old"arangodserviceorbinaryCopytheentire"old"datadirectorytoasafeplace(thatis,abackup)InstallthenewversionofArangoDBandstarttheserverwiththe--database.auto-upgradeoptiononce.ThismightwritetothelogfileofArangoDB,soyoumaywanttocheckthelogsforanyissuesbeforegoingon.Startthe"new"arangodserviceorbinaryregularlyandcheckthelogsforanyissues.Whenyou'reconfidenteverythingwentwell,youmaywanttocheckthedatabasedirectoryforanyfileswiththeending.old.ThesefilesarecreatedbyArangoDBduringupgradesandcanbesafelyremovedmanuallylater.
Ifanythinggoeswrongduringorshortlyaftertheupgrade:
Stopthe"new"arangodserviceorbinaryReverttothe"old"arangodbinaryandrestorethe"old"datadirectoryStartthe"old"versionagain
ItisnotsupportedtousedatafilescreatedormodifiedbyanewerversionofArangoDBwithanolderArangoDBversion.Forexample,itisunsupportedandislikelytocauseproblemswhenusing3.2datafileswithanArangoDB3.0instance.
Switchingthestorageengine
Inordertouseadifferentstorageenginewithanexistingdatadirectory,itisrequiredtofirstcreatealogicalbackupofthedatausingarangodump.
Afterthat,thearangodservershouldberestartedwiththedesiredstorageengineselected(thiscanbedonebysettingtheoption--server.storage-engine)andusinganon-existingdatadirectory.
Whentheserverisupandrunningwiththedesiredstorageengine,thedatacanbere-importedusingarangorestore.
Upgrading
496
UpgradingtoArangoDB3.3PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.3.Pleasebesurethatyouhavecheckedthelistofchangesin3.3beforeupgrading.
Upgradingto3.3
497
UpgradingtoArangoDB3.2PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.2.Pleasebesurethatyouhavecheckedthelistofchangesin3.2beforeupgrading.
Switchingthestorageengine
Inordertouseadifferentstorageenginewithanexistingdatadirectory,itisrequiredtofirstcreatealogicalbackupofthedatausingarangodump.Thatbackupshouldbecreatedbeforetheupgradeto3.2.
Afterthat,theArangoDBinstallationcanbeupgradedandstopped.Theservershouldthenberestartedwiththedesiredstorageengineselected(thiscanbedonebysettingtheoption--server.storage-engine)andusinganon-existingdatadirectory.Thiswillstarttheserverwiththeselectedstorageenginebutwithnodata.
Whentheserverisupandrunning,thedatafromthelogicalbackupcanbere-importedusingarangorestore.
Upgradingto3.2
498
UpgradingtoArangoDB3.1PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.1.Pleasebesurethatyouhavecheckedthelistofchangesin3.1beforeupgrading.
Upgradingto3.1
499
UpgradingtoArangoDB3.0PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.0.Pleasebesurethatyouhavecheckedthelistofchangesin3.0beforeupgrading.
MigratingdatabasesandcollectionsfromArangoDB2.8to3.0
ArangoDB3.0doesnotprovideanautomaticupdatemechanismfordatabasedirectoriescreatedwiththe2.xbranchesofArangoDB.
InordertomigratedatafromArangoDB2.8(oranolder2.xversion)intoArangoDB3.0,itisnecessarytoexportthedatafrom2.8usingarangodump,andthenimportthedumpintoafreshArangoDB3.0witharangorestore.
Todothis,firstrunthe2.8versionofarangodumptoexportthedatabasedataintoadirectory.arangodumpwilldumpthe_systemdatabasebydefault.Inordertomakeitdumpmultipledatabases,itneedstobeinvokedoncepersourcedatabase,e.g.
#in2.8
arangodump--server.database_system--output-directorydump-system
arangodump--server.databasemydb--output-directorydump-mydb
...
Thatwillproduceadumpdirectoryforeachdatabasethatarangodumpiscalledfor.Iftheserverhasauthenticationturnedon,itmaybenecessarytoprovidetherequiredcredentialswheninvokingarangodump,e.g.
arangodump--server.database_system--server.usernamemyuser--server.passwordmypasswd--output-directorydump-system
ThedumpsproducedbyarangodumpcannowbeimportedintoArangoDB3.0usingthe3.0versionofarangodump:
#in3.0
arangorestore--server.database_system--input-directorydump-system
arangorestore--server.databasemydb--input-directorydump-mydb
...
arangorestorewillbydefaultfailifthetargetdatabasedoesnotexist.Itcanbetoldtocreateitautomaticallyusingtheoption--create-databasetrue:
arangorestore--server.databasemydb--create-databasetrue--input-directorydump-mydb
Andagainitmayberequiredtoprovideaccesscredentialswheninvokingarangorestore:
arangorestore--server.databasemydb--create-databasetrue--server.usernamemyuser--server.passwordmypasswd--input-directo
rydump-system
Pleasenotethattheversionofdump/restoreshouldmatchtheserverversion,i.e.itisrequiredtodumptheoriginaldatawiththe2.8versionofarangodumpandrestoreitwiththe3.0versionofarangorestore.
Afterthatthe3.0instanceofArangoDBwillcontainthedatabasesandcollectionsthatwerepresentinthe2.8instance.
Adjustingauthenticationinfo
AuthenticationinformationwasstoredperdatabaseinArangoDB2.8,meaningtherecouldbedifferentusersandaccesscredentialsperdatabase.In3.0,theusersarestoredinacentrallocationinthe_systemdatabase.Tousethesameusersetupasin2.8,itmayberequiredtocreateextrausersand/oradjusttheirpermissions.
Inordertodothat,pleaseconnecttothe3.0instancewithanArangoShell(thiswillconnecttothe_systemdatabasebydefault):
arangosh--server.usernamemyuser--server.passwordmypasswd
Upgradingto3.0
500
Usethefollowingcommandstocreateanewuserwithsomepasswordandgrantthemaccesstoaspecificdatabase
require("@arangodb/users").save(username,password,true);
require("@arangodb/users").grantDatabase(username,databaseName,"rw");
Forexample,tocreateausermyuserwithpasswordmypasswdandgivethemaccesstodatabasesmydb1andmydb2,thecommandswouldlookasfollows:
require("@arangodb/users").save("myuser","mypasswd",true);
require("@arangodb/users").grantDatabase("myuser","mydb1","rw");
require("@arangodb/users").grantDatabase("myuser","mydb2","rw");
Existinguserscanalsobeupdated,removedorlistedusingthefollowingcommands:
/*updateusermyuserwithpasswordmypasswd*/
require("@arangodb/users").update("myuser","mypasswd",true);
/*removeusermyuser*/
require("@arangodb/users").remove("myuser");
/*listallusers*/
require("@arangodb/users").all();
Foxxapplications
Thedump/restoreproceduredescribedabovewillnotexportandre-importFoxxapplications.Inordertomovethesefrom2.8to3.0,Foxxapplicationsshouldbeexportedaszipfilesviathe2.8webinterface.
Thezipfilescanthenbeuploadedinthe"Services"sectionintheArangoDB3.0webinterface.Applicationsmayneedtobeadjustedmanuallytorunin3.0.PleaseconsultthemigrationguideforFoxxapps.
AnalternativewayofmovingFoxxappsinto3.0istocopythesourcedirectoryofa2.8Foxxapplicationmanuallyintothe3.0Foxxappsdirectoryforthetargetdatabase(whichisnormally/var/lib/arangodb3-apps/_db/<dbname>/buttheexactlocationisplatform-specific).
Upgradingto3.0
501
UpgradingtoArangoDB2.8PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.8.Pleasebesurethatyouhavecheckedthelistofchangesin2.8beforeupgrading.
PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.8cannotbeusedwithearlierversions(e.g.ArangoDB2.7)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.
DatabaseDirectoryVersionCheckandUpgrade
ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.8encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.
Theoutputwillthenlooklikethis:
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Databasedirectoryversion(20702)islowerthancurrentversion(208
00).
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':---------------------------------------------------------------------
-
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':--upgrade
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbstop
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbstart
2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':---------------------------------------------------------------------
-
2015-12-04T17:11:17Z[31432]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption
TomakeArangoDB2.8startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.
Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).
Forexample,ifyouregularlystartyourArangoDBserverwith
unix>arangodmydatabasefolder
thenrunning
unix>arangodmydatabasefolder--upgrade
willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.
Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.
Thelastlineoftheoutputshouldlooklikethis:
2015-12-04T17:12:15Z[31558]INFOdatabaseupgradepassed
Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.8regularly.
Upgradingaclusterplannedinthewebinterface
Upgradingto2.8
502
AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.
Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:
1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.
2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.
3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".
4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.
ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute
arangosh>require("org/arangodb/cluster").Upgrade("root","");
Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.
UpgradingFoxxappsgeneratedbyArangoDB2.7andearlier
TheimplementationoftherequirefunctionusedtoimportmodulesinArangoDBandFoxxhaschangedinordertoimprovecompatibilitywithNode.jsmodules.
Givenanapp/servicewiththefollowinglayout:
manifest.jsoncontrollers/
todos.jsmodels/
todo.jsrepositories/
todos.jsnode_modules/
models/todo.js
Thefilecontrollers/todos.jswouldpreviouslycontainthefollowingrequirecalls:
var_=require('underscore');
varjoi=require('joi');
varFoxx=require('org/arangodb/foxx');
varArangoError=require('org/arangodb').ArangoError;
varTodos=require('repositories/todos');//<--!
varTodo=require('models/todo');//<--!
Therequirepathsrepositories/todosandmodels/todowerepreviouslyresolvedlocallyasrelativetotheapproot.
Startingwith2.8thesepathswouldinsteadberesolvedasrelativetothenode_modulesfolderortheglobalArangoDBmodulepathsbeforebeingresolvedlocallyasafallback.
Inthegivenexamplelayouttheappwouldbreakin2.8becausethemodulenamemodels/todowouldalwaysresolvetonode_modules/models/todo.js(whichpreviouslywouldhavebeenignored)insteadofthelocalmodels/todo.js.
Inordertomakesuretheappstillworksin2.8,therequirecallsincontrollers/todos.jswouldneedtobeadjustedtolooklikethis:
Upgradingto2.8
503
var_=require('underscore');
varjoi=require('joi');
varFoxx=require('org/arangodb/foxx');
varArangoError=require('org/arangodb').ArangoError;
varTodos=require('../repositories/todos');//<--!
varTodo=require('../models/todo');//<--!
Notethattheold"global"stylerequirecallsmaystillworkin2.8butmaybreakunexpectedlyifmoduleswithmatchingnamesareinstalledglobally.
Upgradingto2.8
504
UpgradingtoArangoDB2.6PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.6.Pleasebesurethatyouhavecheckedthelistofchangesin2.6beforeupgrading.
PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.6cannotbeusedwithearlierversions(e.g.ArangoDB2.5)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.
DatabaseDirectoryVersionCheckandUpgrade
ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.6encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.
Theoutputwillthenlooklikethis:
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Databasedirectoryversion(20501)islowerthancurrentversion(2060
0).
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':--upgrade
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbstop
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbstart
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------
2015-02-17T09:43:11Z[8302]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption
TomakeArangoDB2.6startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.
Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).
Forexample,ifyouregularlystartyourArangoDBserverwith
unix>arangodmydatabasefolder
thenrunning
unix>arangodmydatabasefolder--upgrade
willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.
Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.
Thelastlineoftheoutputshouldlooklikethis:
2014-12-22T12:03:31Z[12026]INFOdatabaseupgradepassed
Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.6regularly.
Upgradingaclusterplannedinthewebinterface
Upgradingto2.6
505
AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.
Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:
1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.
2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.
3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".
4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.
ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute
arangosh>require("org/arangodb/cluster").Upgrade("root","");
Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.
Upgradingto2.6
506
UpgradingtoArangoDB2.5PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.5.Pleasebesurethatyouhavecheckedthelistofchangesin2.5beforeupgrading.
PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.5cannotbeusedwithearlierversions(e.g.ArangoDB2.4)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.
In2.5wehavealsochangedthepathsforFoxxapplications.PleasealsomakesurethatyouhaveabackupofallFoxxappsinyourjavascript.app-pathandjavascript.dev-app-path.ItissufficienttohavethesourcefilesforFoxxsomewhereelsesoyoucanreinstallthemonerror.Tocheckthateverythinghasworkedduringupgradeyoucouldusetheweb-interfaceApplicationstabor
unix>foxx-managerlist
forallyourdatabases.Thelistedappsshouldbeidenticalbeforeandaftertheupgrade.
DatabaseDirectoryVersionCheckandUpgrade
ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.5encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.
Theoutputwillthenlooklikethis:
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Databasedirectoryversion(20401)islowerthancurrentversion(2050
0).
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':--upgrade
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbstop
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':/etc/init.d/arangodbstart
2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------
2015-02-17T09:43:11Z[8302]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption
TomakeArangoDB2.5startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.Note:WehavechangedFoxxfolderstructureandimplementedanupgradetasktomoveyourapplicationstothenewstructure.InordertotellthisupgradetasktoalsomoveyourdevelopmentFoxxappspleasemakesureyougivethedev-app-pathaswell.IfyouhavenotuseddevelopmentmodeforFoxxappsyoucandropthe--javascript.dev-app-path.Itisonlypossibletoupgradeonedev-app-pathtogetherwithonedatafolder.
unix>arangoddata--upgrade--javascript.dev-app-pathdevapps
wheredataisArangoDB'smaindatadirectoryanddevappsisthedirectorywhereyoudevelopFoxxapps.
Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).
Forexample,ifyouregularlystartyourArangoDBserverwith
unix>arangodmydatabasefolder
thenrunning
unix>arangodmydatabasefolder--upgrade
Upgradingto2.5
507
willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.
Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.
Thelastlineoftheoutputshouldlooklikethis:
2014-12-22T12:03:31Z[12026]INFOdatabaseupgradepassed
Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.5regularly.
UpgradingaclusterplannedinthewebinterfaceAclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.
Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:
1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.
2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.
3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".
4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.
ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute
arangosh>require("org/arangodb/cluster").Upgrade("root","");
Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.
Upgradingto2.5
508
UpgradingtoArangoDB2.4PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.4.Pleasebesurethatyouhavecheckedthelistofchangesin2.4beforeupgrading.
PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.4cannotbeusedwithearlierversions(e.g.ArangoDB2.3)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.
DatabaseDirectoryVersionCheckandUpgrade
ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.4encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.
Theoutputwillthenlooklikethis:
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':Databasedirectoryversion(20302)islowerthancurrentversion(204
00).
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':---------------------------------------------------------------------
-
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':--upgrade
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':/etc/init.d/arangodbstop
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':/etc/init.d/arangodbstart
2014-12-22T12:02:28Z[12001]ERRORIndatabase'_system':---------------------------------------------------------------------
-
2014-12-22T12:02:28Z[12001]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption
TomakeArangoDB2.4startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption:
unix>arangoddata--upgrade
wheredataisArangoDB'smaindatadirectory.
Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).
Forexample,ifyouregularlystartyourArangoDBserverwith
unix>arangodmydatabasefolder
thenrunning
unix>arangodmydatabasefolder--upgrade
willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.
Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.
Thelastlineoftheoutputshouldlooklikethis:
2014-12-22T12:03:31Z[12026]INFOdatabaseupgradepassed
Upgradingto2.4
509
Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.4regularly.
Upgradingaclusterplannedinthewebinterface
AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.
Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:
1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.
2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.
3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".
4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.
ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute
arangosh>require("org/arangodb/cluster").Upgrade("root","");
Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.
Upgradingto2.4
510
UpgradingtoArangoDB2.3PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.3.Pleasebesurethatyouhavecheckedthelistofchangesin2.3beforeupgrading.
PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.3cannotbeusedwithearlierversions(e.g.ArangoDB2.2)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.
DatabaseDirectoryVersionCheckandUpgrade
ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.3encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.
Theoutputwillthenlooklikethis:
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':Databasedirectoryversion(2.2)islowerthancurrentversion(20300)
.
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':----------------------------------------------------------------------
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':--upgrade
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':/etc/init.d/arangodbstop
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':/etc/init.d/arangodbstart
2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':----------------------------------------------------------------------
2014-11-03T15:48:06Z[2694]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption
TomakeArangoDB2.3startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption:
unix>arangoddata--upgrade
wheredataisArangoDB'smaindatadirectory.
Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).
Forexample,ifyouregularlystartyourArangoDBserverwith
unix>arangodmydatabasefolder
thenrunning
unix>arangodmydatabasefolder--upgrade
willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.
Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.
Theoutputshouldlooklikethis:
2014-11-03T15:48:47Z[2708]INFOIndatabase'_system':Found24definedtask(s),5task(s)torun
2014-11-03T15:48:47Z[2708]INFOIndatabase'_system':stateprod/standalone/upgrade,tasksupdateUserModel,createStatistics,
upgradeClusterPlan,setupQueues,setupJobs
2014-11-03T15:48:48Z[2708]INFOIndatabase'_system':upgradesuccessfullyfinished
2014-11-03T15:48:48Z[2708]INFOdatabaseupgradepassed
Upgradingto2.3
511
Pleasechecktheoutputthe--upgraderun.Itmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolved,youcanstartArangoDB2.3regularly.
Upgradingaclusterplannedinthewebinterface
AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.
Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:
1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.
2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.
3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".
4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.
ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute
arangosh>require("org/arangodb/cluster").Upgrade("root","");
Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.
Upgradingto2.3
512
UpgradingtoArangoDB2.2PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.2.
PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.2cannotbeusedwithearlierversions(e.g.ArangoDB2.1)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.
DatabaseDirectoryVersionCheckandUpgrade
ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.2encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.
Theoutputwillthenlooklikethis:
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':Databasedirectoryversion(2.1)islowerthanserverversion(2.2).
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':---------------------------------------------------------------------
-
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':--upgrade
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':/etc/init.d/arangodbstop
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':/etc/init.d/arangodbstart
2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':---------------------------------------------------------------------
-
2014-07-07T22:04:53Z[18675]FATALDatabaseversioncheckfailedfor'_system'.Pleasestarttheserverwiththe--upgradeopti
on
TomakeArangoDB2.2startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption:
unix>arangoddata--upgrade
wheredataisArangoDB'smaindatadirectory.
Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).
Forexample,ifyouregularlystartyourArangoDBserverwith
unix>arangodmydatabasefolder
thenrunning
unix>arangodmydatabasefolder--upgrade
willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.
Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.
Theoutputshouldlooklikethis:
2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':startingupgradefromversion2.1to2.2.0
2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':Found19definedtask(s),2task(s)torun
2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':upgradesuccessfullyfinished
2014-07-07T22:11:30Z[18867]INFOdatabaseupgradepassed
Upgradingto2.2
513
Pleasechecktheoutputthe--upgraderun.Itmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolved,youcanstartArangoDB2.2regularly.
Upgradingaclusterplannedinthewebinterface
AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.
Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:
1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.
2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.
3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".
4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.
ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute
arangosh>require("org/arangodb/cluster").Upgrade("root","");
Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.
Upgradingto2.2
514
ArangodIftheArangoDBserverdoesnotstartorifyoucannotconnecttoitusingarangoshorotherclients,youcantrytofindtheproblemcausebyexecutingthefollowingsteps.Iftheserverstartsupwithoutproblemsyoucanskipthissection.
Checktheserverlogfile:Iftheserverhaswrittenalogfileyoushouldcheckitbecauseitmightcontainrelevanterrorcontextinformation.
Checktheconfiguration:Theserverlooksforaconfigurationfilenamedarangod.confonstartup.Thecontentsofthisfilewillbeusedasabaseconfigurationthatcanoptionallybeoverriddenwithcommand-lineconfigurationparameters.Youshouldchecktheconfigfileforthemostrelevantparameterssuchas:
server.endpoint:WhatIPaddressandporttobindtologparameters:Ifandwheretologdatabase.directory:Paththedatabasefilesarestoredin
Iftheconfigurationrevealsthatsomethingisnotconfiguredrighttheconfigfileshouldbeadjustedandtheserverberestarted.
Starttheservermanuallyandcheckitsoutput:Startingtheservermightfailevenbeforeloggingisactivatedsotheserverwillnotproducelogoutput.Thiscanhappeniftheserverisconfiguredtowritethelogstoafilethattheserverhasnopermissionson.Inthiscasetheservercannotloganerrortothespecifiedlogfilebutwillwriteastartuperroronstderrinstead.Startingtheservermanuallywillalsoallowyoutooverridespecificconfigurationoptions,e.g.toturnon/offfileorscreenloggingetc.
ChecktheTCPport:Iftheserverstartsupbutdoesnotacceptanyincomingconnectionsthismightbeduetofirewallconfigurationbetweentheserverandanyclient(s).TheserverbydefaultwilllistenonTCPport8529.PleasemakesurethisportisactuallyaccessiblebyotherclientsifyouplantouseArangoDBinanetworksetup.
Whenusinghostnamesintheconfigurationorwhenconnecting,pleasemakesurethehostnameisactuallyresolvable.ResolvinghostnamesmightinvokeDNS,whichcanbeasourceoferrorsonitsown.
ItisgenerallygoodadvicetonotuseDNSwhenspecifyingtheendpointsandconnectionaddresses.UsingIPaddressesinsteadwillruleoutDNSasasourceoferrors.Anotheralternativeistouseahostnamespecifiedinthelocal/etc/hostsfile,whichwillthenbypassDNS.
Testifcurlcanconnect:Oncetheserverisstarted,youcanquicklyverifyifitrespondstorequestsatall.Thischeckallowsyoutodeterminewhetherconnectionerrorsareclient-specificornot.Ifatleastoneclientcanconnect,itislikelythatconnectionproblemsofotherclientsarenotduetoArangoDB'sconfigurationbutduetoclientorin-betweennetworkconfigurations.
Youcantestconnectivityusingasimplecommandsuchas:
curl--dump--XGEThttp://127.0.0.1:8529/_api/version&&echo
ThisshouldreturnaresponsewithanHTTP200statuscodewhentheserverisrunning.Ifitdoesitalsomeanstheserverisgenerallyacceptingconnections.Alternativetoolstocheckconnectivityarelynxorab.
arangod
516
EmergencyConsoleTheArangoDBdatabaseserverhastwomodesofoperation:Asaserver,whereitwillanswertoclientrequestsandasanemergencyconsole,inwhichyoucanaccessthedatabasedirectly.Thelatter-asthenamesuggests-shouldonlybeusedincaseofanemergency,forexample,acorruptedcollection.Usingtheemergencyconsoleallowsyoutoissueallcommandsnormallyavailableinactionsandtransactions.Whenstartingtheserverinemergencyconsolemode,theservercannothandleanyclientrequests.
Youshouldneverstartmorethanoneserverusingthesamedatabasedirectory,independentofthemodeofoperation.Normally,ArangoDBwillpreventyoufromdoingthisbyplacingalockfileinthedatabasedirectoryandnotallowingasecondArangoDBinstancetousethesamedatabasedirectoryifalockfileisalreadypresent.
InCaseOfDisaster
Thefollowingcommandstartsanemergencyconsole.
Note:Neverstarttheemergencyconsoleforadatabasewhichalsohasaserverattachedtoit.Ingeneral,theArangoDBshelliswhatyouwant.
>./arangod--console--logerror/tmp/vocbase
ArangoDBshell[V8version5.0.71.39,DBversion3.x.x]
arango>1+2;
3
arango>vardb=require("@arangodb").db;db.geo.count();
703
TheemergencyconsoleprovidesaJavaScriptconsoledirectlyrunninginthearangodserverprocess.ThisallowstodebugandexaminecollectionsanddocumentsaswiththenormalArangoDBshell,butwithoutclient/servercommunication.
However,itisverylikelythatyouwillneverneedtheemergencyconsoleunlessyouareanArangoDBdeveloper.
EmergencyConsole
517
DatafileDebugger
InCaseOfDisaster
AranagoDBusesappend-onlyjournals.Datacorruptionshouldonlyoccurwhenthedatabaseserveriskilled.Inthiscase,thecorruptionshouldonlyoccurinthelastobject(s)thathavebeingwrittentothejournal.
Ifacorruptionoccurswithinanormaldatafile,thenthiscanonlyhappenifahardwarefaultoccurred.
Ifajournalordatafileiscorrupt,shutdownthedatabaseserverandstarttheprogram
unix>arango-dfdb
inordertochecktheconsistencyofthedatafilesandjournals.Thisbringsup
_________________
/\___||____/_(_)|___/\/__\/_\
//\/_`|__/_`||_|||/_\//\/__\////_\/
//_//(_||||(_||_|||__///_//\/\//_\\
/___,'\__,_|\__\__,_|_||_|_|\___|/___,'\_____/\____/
Availablecollections:
0:_structures
1:_users
2:_routing
3:_modules
4:_graphs
5:products
6:prices
*:all
Collectiontocheck:
Youcannowselectwhichdatabaseandcollectionyouwanttocheck.Afteryouselectedoneorallofthecollections,aconsistencycheckwillbeperformed.
Checkingcollection#1:_users
Database
path:/usr/local/var/lib/arangodb
Collection
name:_users
identifier:82343
Datafiles
#ofjournals:1
#ofcompactors:1
#ofdatafiles:0
Datafile
path:/usr/local/var/lib/arangodb/collection-82343/journal-1065383.db
type:journal
currentsize:33554432
maximalsize:33554432
totalused:256
#ofentries:3
status:OK
Ifthereisaproblemwithoneofthedatafiles,thenthedatabasedebuggerwillprintitandpromptforwhethertoattempttofixit.
WARNING:Thejournalwasnotclosedproperly,thelastentriesarecorrupted.
ThismighthappenArangoDBwaskilledandthelastentrieswerenot
fullywrittentodisk.
DatafileDebugger
518
Wipethelastentries(Y/N)?
IfyouanswerY,thecorruptedentrywillberemoved.
Ifyouseeacorruptioninadatafile(andnotajournal),thensomethingisterriblywrong.ThesefilesareimmutableandneverchangedbyArangoDB.Acorruptioninsuchfileisanindicationofahard-diskfailure.
DatafileDebugger
519
ArangobenchArangobenchisArangoDB'sbenchmarkandtesttool.Itcanbeusedtoissuetestrequeststothedatabaseforperformanceandserverfunctiontesting.Itsupportsparallelqueryingandbatchrequests.
Relatedblogposts:
MeasuringArangoDBinsertperformanceGainfactorof5usingbatchrequests
Startupoptions--async:Sendasynchronousrequests.Thedefaultvalueisfalse.
--batch-size:Numberofoperationstosendperbatch.Use0todisablebatching(thisisthedefault).
--collection:Nameofcollectiontouseintest(onlyrelevantforteststhatinvokecollections).
--replication-factor:Incaseofacluster,thereplicationfactorofthecreatedcollections.
--number-of-shards:Incaseofacluster,thenumberofshardsofthecreatedcollections.
--wait-for-sync:ThevalueofwaitForSyncforcreatedcollections.
--complexity:Complexityvaluefortestcase(default:1).Meaningdependsontestcase.
--concurrency:Numberofparallelthreadsthatwillissuerequests(default:1).
--configuration:Readconfigurationfromfile.
--delay:Useastartupdelay.Thisisonlynecessarywhenruninseries.Thedefaultvalueisfalse.
--keep-alive:UseHTTPkeep-alive(default:true).
--progress:Showprogressofbenchmark,onevery20threquest.Settofalsetodisableintermediatelogging.Thedefaultvalueistrue.
--requests:Totalnumberofrequeststoperform(default:1000).
--server.endpoint:Serverendpointtoconnectto,consistingofprotocol,IPaddressandport.Defaultstotcp://localhost:8529.
--server.database:Databasenametousewhenconnecting(default:"_system").
--server.username:Usernametousewhenconnecting(default:"root").
--server.password:Passwordtousewhenconnecting.Don'tspecifythisoptiontogetapasswordprompt.
--server.authentication:Wetherornottoshowthepasswordpromptanduseauthenticationwhenconnectingtotheserver(default:true).
--test-case:Nameoftestcasetoperform(default:"version").Possiblevalues:
version:requests/_api/versiondocument:createsdocumentscollection:createscollectionsimport-document:createsdocumentsviatheimportAPIhash:Create/Read/Update/Readdocumentsindexedbyahashindexskiplist:Create/Read/Update/Readdocumentsindexedbyaskiplistedge:Create/Read/Updateedgedocumentsshapes:Create&Deletedocumentswithheterogeneousattributenamesshapes-append:Createdocumentswithheterogeneousattributenamesrandom-shapes:Create/Read/Deleteheterogeneousdocumentswithrandomvaluescrud:Create/Read/Update/Deletecrud-append:Create/Read/Update/Readagain
Arangobench
520
crud-write-read:Create/ReadDocumentsaqltrx:AQLTransactionswithdeepnestedAQLFOR-loopscounttrx:usesJStransactionstocountthedocumentsandinserttheresultagainmultitrx:multipletransactionscombiningreads&writesfromjsmulti-collection:multipletransactionscombiningreads&writesfromjsonmultiplecollectionsaqlinsert:insertdocumentsviaAQLaqlv8:executeAQLwithV8functionstoinsertrandomdocuments
--verbose:PrintoutrepliesiftheHTTPheaderindicatesDBerrors.(default:false).
Examples
arangobench
StartsArangobenchwiththedefaultuserandserverendpoint.
--test-caseversion--requests1000--concurrency1
Runsthe'version'testcasewith1000requests,withoutconcurrency.
--test-casedocument--requests1000--concurrency2
Runsthe'document'testcasewith2000requests,withtwoconcurrentthreads.
--test-casedocument--requests1000--concurrency2--asynctrue
Runsthe'document'testcasewith2000requests,withconcurrency2,withasyncrequests.
--test-casedocument--requests1000--concurrency2--batch-size10
Runsthe'document'testcasewith2000requests,withconcurrency2,usingbatchrequests.
Arangobench
521
Architecture
AppendOnly/MVCC
Insteadofoverwritingexistingdocuments,ArangoDBwillcreateanewversionofmodifieddocuments.Thisiseventhecasewhenadocumentgetsdeleted.Thetwobenefitsare:
Objectscanbestoredcoherentlyandcompactlyinthemainmemory.Objectsarepreserved,isolatedwritingandreadingtransactionsallowaccessingtheseobjectsforparalleloperations.
Thesystemcollectsobsoleteversionsasgarbage,recognizingthemasforsaken.Garbagecollectionisasynchronousandrunsparalleltootherprocesses.
MostlyMemory/Durability
Databasedocumentsarestoredinmemory-mappedfiles.Perdefault,thesememory-mappedfilesaresyncedregularlybutnotinstantly.Thisisoftenagoodtradeoffbetweenstorageperformanceanddurability.Ifthislevelofdurabilityistoolowforanapplication,theservercanalsosyncallmodificationstodiskinstantly.ThiswillgivefulldurabilitybutwillcomewithaperformancepenaltyaseachdatamodificationwilltriggerasyncI/Ooperation.
Architecture
522
Write-aheadlogBothstorageenginesuseaformofwriteaheadlogging(WAL).Startingwithversion2.2ArangoDBstoresalldata-modificationoperationinitswrite-aheadlog.Thewrite-aheadlogissequenceofappend-onlyfilescontainingallthewriteoperationsthatwereexecutedontheserver.
Itisusedtorundatarecoveryafteraservercrash,andcanalsobeusedinareplicationsetupwhenslavesneedtoreplaythesamesequenceofoperationsasonthemaster.
MMFilesWALDetails
Bydefault,eachwrite-aheadlogfileis32MiBinsize.Thissizeisconfigurableviatheoption--wal.logfile-size.Whenawrite-aheadlogfileisfull,itissettoread-only,andfollowingoperationswillbewrittenintothenextwrite-aheadlogfile.Bydefault,ArangoDBwillreservesomesparelogfilesinthebackgroundsoswitchinglogfilesshouldbefast.HowmanyreservelogfilesArangoDBwilltrytokeepavailableinthebackgroundcanbecontrolledbytheconfigurationoption--wal.reserve-logfiles.
Datacontainedinfullwrite-aheadfileswilleventuallybetransferredintothejournalsordatafilesofcollections.Onlythe"surviving"documentswillbecopiedover.Whenallremainingoperationsfromawrite-aheadlogfilehavebeencopiedoverintothejournalsordatafilesofthecollections,thewrite-aheadlogfilecansafelyberemovedifitisnotusedforreplication.
Long-runningtransactionspreventwrite-aheadlogfilesfrombeingfullygarbage-collectedbecauseitisunclearwhetheratransactionwillcommitorabort.Long-runningtransactionscanthusblockthegarbage-collectionprogressandshouldthereforebeavoidedatallcosts.
Onasystemthatactsasareplicationmaster,itisusefultokeepafewofthealreadycollectedwrite-aheadlogfilessoreplicationslavesstillcanfetchdatafromthemifrequired.Howmanycollectedlogfileswillbekeptbeforetheygetdeletedisconfigurableviatheoption--wal.historic-logfiles.
Forallwrite-aheadlogconfigurationoptions,pleaserefertothepageWrite-aheadlogoptions.
RocksDBWALDetails
TheoptionsmentionedaboveonlyapplyforMMFiles.TheWALintherocksdbstorageengineworksslightlydifferently.Note:InrocksdbtheWALoptionsareallprefixedwith--rocksdb.*.The--wal.*optionsdohavenoeffect.
TheindividualRocksDBWALfilesareperdefaultabout64MiBbig.Thesizewillalwaysbeproportionallysizedtothevaluespecifiedvia--rocksdb.write-buffer-size.Thevaluespecifiestheamountofdatatobuildupinmemory(backedbytheunsortedWALondisk)beforeconvertingittoasortedon-diskfile.Largervaluescanincreaseperformance,especiallyduringbulkloads.Upto--rocksdb.max-write-buffer-numberwritebuffersmaybeheldinmemoryatthesametime,soyoumaywishtoadjustthisparametertocontrolmemoryusage.Alargerwritebufferwillresultinalongerrecoverytimethenexttimethedatabaseisopened.
TheRocksDBWALonlycontainscommittedtransactions.Thismeansyouwillneverseepartialtransactionsinthereplicationlog,butitalsomeanstransactionsaretrackedcompletelyin-memory.InpracticethiscausesRocksDBtransactionsizestobelimited,formoreinformationseetheRocksDBConfiguration
Write-aheadlog
523
StorageEnginesAttheverybottomoftheArangoDBdatabaseliesthestorageengine.Thestorageengineisresponsibleforpersistingthedocumentsondisk,holdingcopiesinmemory,providingindexesandcachestospeedupqueries.
Uptoversion3.1ArangoDBonlysupportedmemorymappedfiles(MMFiles)assolestorageengine.Beginningwith3.2ArangoDBhassupportforpluggablestorageengines.ThesecondsupportedengineisRocksDBfromFacebook.
MMFiles RocksDB
default optional
datasetneedstofitintomemory workwithasmuchdataasfitsondisk
indexesinmemory hotsetinmemory,dataandindexesondisk
slowrestartduetoindexrebuilding faststartup(norebuildingofindexes)
volatilecollections(onlyinmemory,optional) collectiondataalwayspersisted
collectionlevellocking(writesblockreads) concurrentreadsandwrites
Blogarticle:ComparingnewRocksDBandMMFilesstorageengines
RocksDBisanembeddablepersistentkey-valuestore.Itisalogstructuredatabaseandisoptimizedforfaststorage.
TheMMFilesengineisoptimizedfortheuse-casewherethedatafitsintothemainmemory.Itallowsforveryfastconcurrentreads.However,writesblockreadsandlockingisoncollectionlevel.Indexesarealwaysinmemoryandarerebuiltonstartup.Thisgivesbetterperformancebutimposesalongerstartuptime.
TheRocksDBengineisoptimizedforlargedata-setsandallowsforasteadyinsertperformanceevenifthedata-setismuchlargerthanthemainmemory.Indexesarealwaysstoredondiskbutcachesareusedtospeedupperformance.RocksDBusesdocument-levellocksallowingforconcurrentwrites.Writesdonotblockreads.Readsdonotblockwrites.
Theenginemustbeselectedforthewholeserver/cluster.Itisnotpossibletomixengines.Thetransactionhandlingandwrite-ahead-logformatintheindividualenginesisverydifferentandthereforecannotbemixed.
RocksDB
Advantages
RocksDBisaveryflexibleenginethatcanbeconfiguredforvarioususecases.
ThemainadvantagesofRocksDBare
document-levellockssupportforlargedata-setspersistentindexes
Caveats
RocksDBallowsconcurrentwrites.However,whentouchingthesamedocumentawriteconflictisraised.ThiscannothappenwiththeMMFilesengine,thereforeapplicationsthatswitchtoRocksDBneedtobepreparedthatsuchexceptioncanarise.ItispossibletoexclusivelylockcollectionswhenexecutingAQL.Thiswillavoidwriteconflictsbutalsoinhibitsconcurrentwrites.
Currently,anotherrestrictionisduetothetransactionhandlinginRocksDB.Transactionsarelimitedintotalsize.Ifyouhaveastatementmodifyingalotofdocumentsitisnecessarytocommitdatainbetween.ThiswillbedoneautomaticallyforAQLbydefault.
Performance
RocksDBisabasedonlog-structuredmergetree.Agoodintroductioncanbefoundin:
StorageEngines
524
http://www.benstopford.com/2015/02/14/log-structured-merge-trees/https://blog.acolyer.org/2014/11/26/the-log-structured-merge-tree-lsm-tree/
Thebasicideaisthatdataisorganizedinlevelswereeachlevelisafactorlargerthantheprevious.Newdatawillresideinsmallerlevelswhileolddataismoveddowntothelargerlevels.Thisallowstosupporthighrateofinsertsoveranextendedperiod.Inprincipleitispossiblethatthedifferentlevelsresideondifferentstoragemedia.ThesmalleronesonfastSSD,thelargeronesonbiggerspinningdisks.
RocksDBitselfprovidesalotofdifferentknobstofinetunethestorageengineaccordingtoyouruse-case.ArangoDBsupportsthemostcommononesusingtheoptionsbelow.
Performancereportsforthestorageenginecanbefoundhere:
https://github.com/facebook/rocksdb/wiki/performance-benchmarkshttps://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
ArangoDBoptions
ArangoDBhasacacheforthepersistentindexesinRocksDB.Thetotalsizeofthiscacheiscontrolledbytheoption
--cache.size
RocksDBalsohasacachefortheblocksstoredondisk.Thesizeofthiscacheiscontrolledbytheoption
--rocksdb.block-cache-size
ArangoDBdistributestheavailablememoryequallybetweenthetwocachesbydefault.
ArangoDBchoosesasizeforthevariouslevelsinRocksDBthatissuitableforgeneralpurposeapplications.
RocksDBlogstrutureddatalevelshaveincreasingsize
MEM:--
L0:--
L1:----
L2:--------
...
NeworupdatedDocumentsarefirststoredinmemory.Ifthismemtablereachesthelimitgivenby
--rocksdb.write-buffer-size
itwillconvertedtoanSSTfileandinsertedatlevel0.
Thefollowingoptioncontrolsthesizeofeachlevelandthedepth.
--rocksdb.num-levelsN
LimitsthenumberoflevelstoN.Bydefaultitis7andthereisseldomareasontochangethis.Anewlevelisonlyopenedifthereistoomuchdatainthepreviousone.
--rocksdb.max-bytes-for-level-baseB
L0willholdatmostBbytes.
--rocksdb.max-bytes-for-level-multiplierM
EachlevelisatmostMtimesasmuchbytesasthepreviousone.ThereforethemaximumnumberofbytesforlevelLcanbecalculatedas
max-bytes-for-level-base*(max-bytes-for-level-multiplier^(L-1))
StorageEngines
525
Future
RocksDBimposesalimitonthetransactionsize.Itisoptimizedtohandlesmalltransactionsveryefficiently,butiseffectivelylimitingthetotalsizeoftransactions.
ArangoDBcurrentlyusesRocksDB'stransactionstoimplementtheArangoDBtransactionhandling.ThereforethesamerestrictionsapplyforArangoDBtransactionswhenusingtheRocksDBengine.
WewillimprovethisbyintroducingdistributedtransactionsinafutureversionofArangoDB.ThiswillallowhandlinglargetransactionsasaseriesofsmallRocksDBtransactionsandhenceremovingthesizerestriction.
StorageEngines
526
ReleaseNotes
WhatsNew
WhatsNewin3.3WhatsNewin3.2WhatsNewin3.1WhatsNewin3.0WhatsNewin2.8WhatsNewin2.7WhatsNewin2.6WhatsNewin2.5WhatsNewin2.4WhatsNewin2.3WhatsNewin2.2WhatsNewin2.1
KnownIssues
KnownIssuesin3.2
Incompatiblechanges
AlsoseeUpgradingintheAdministrationchapter.
Incompatiblechangesin3.3Incompatiblechangesin3.2Incompatiblechangesin3.1Incompatiblechangesin3.0Incompatiblechangesin2.8Incompatiblechangesin2.7Incompatiblechangesin2.6Incompatiblechangesin2.5Incompatiblechangesin2.4Incompatiblechangesin2.3
Releasenotes
527
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.3.ArangoDB3.3alsocontainsseveralbugfixesthatarenotlistedhere.
Datacenter-to-datacenterreplication(DC2DC)
Everycompanyneedsadisasterrecoveryplanforallimportantsystems.Thisistruefromsmallunitslikesingleprocessesrunninginsomecontainertothelargestdistributedarchitectures.Fordatabasesinparticularthisusuallyinvolvesamixtureoffault-tolerance,redundancy,regularbackupsandemergencyplans.Thelargeradatastore,themoredifficultitistocomeupwithagoodstrategy.
Therefore,itisdesirabletobeabletorunadistributeddatabaseinonedata-centerandreplicatealltransactionstoanotherdata-centerinsomeway.Often,transactionlogsareshippedoverthenetworktoreplicateeverythinginanother,identicalsystemintheotherdata-center.Somedistributeddatastoreshavebuilt-insupportformultipledata-centerawarenessandcanreplicatebetweendata-centersinafullyautomaticfashion.
ArangoDB3.3takesanevolutionarystepforwardbyintroducingmulti-data-centersupport,whichisasynchronousdata-centertodata-centerreplication.Oursolutionisasynchronousandscalestoarbitraryclustersizes,providedyournetworklinkbetweenthedata-centershasenoughbandwidth.Itisfault-tolerantwithoutasinglepointoffailureandincludesalotofmetricsformonitoringinaproductionscenario.
DC2DCisavailableintheEnterpriseedition.
Encryptedbackups
ArangodumpcannowcreateencryptedbackupsusingAES256forencryption.Theencryptionkeycanbereadfromafileorfromageneratorprogram.Itworksinsingleserverandclustermode.
Examplefornon-encryptedbackup(everyonewithaccesstothebackupwillbeabletoreadit):
arangodump--collection"secret"dump
Inordertocreateanencryptedbackup,addthe--encryption.keyfileoptionwheninvokingarangodump:
arangodump--collection"secret"dump--encryption.keyfile~/SECRET-KEY
Thekeymustbeexactly32byteslong(requiredbytheAESblockcipher).
Notethatarangodumpwillnotstorethekeyanywhere.Itistheresponsibilityoftheusertofindasafeplaceforthekey.However,arangodumpwillstoretheusedencryptionmethodinafilenamedENCRYPTIONinthedumpdirectory.Thatwayarangorestorecanlaterfindoutwhetheritisdealingwithanencrypteddumpornot.
Tryingtorestoretheencrypteddumpwithoutspecifyingthekeywillfail:
arangorestore--collection"secret-collection"dump--create-collectiontrue
arangorestorewillcomplainwith:
thedumpdataseemstobeencryptedwithaes-256-ctr,butnokeyinformationwasspecifiedtodecryptthedumpitisrecommendedtospecifyeither--encryption.key-fileor--encryption.key-generatorwheninvokingarangorestorewithanencrypteddump
Itisrequiredtousetheexactsamekeywhenrestoringthedata.Againthisisdonebyprovidingthe--encryption.keyfileparameter:
arangorestore--collection"secret-collection"dump--create-collectiontrue--encryption.keyfile~/SECRET-KEY
WhatsNewin3.3
528
Usingadifferentkeywillleadtothebackupbeingnon-recoverable.
NotethatencryptedbackupscanbeusedtogetherwiththealreadyexistingRocksDBencryption-at-restfeature,buttheycanalsobeusedfortheMMFilesengine,whichdoesnothaveencryption-at-rest.
EncryptedbackupsareavailableintheEnterpriseedition.
Server-levelreplicationArangoDBsupportsasynchronousreplicationfunctionalitysinceversion1.4,butreplicatingfromamasterserverwithmultipledatabasesrequiredmanualsetupontheslaveforeachindividualdatabasetoreplicate.Whenanewdatabasewascreatedonthemaster,oneneededtotakeactionontheslavetoensurethatdataforthatdatabasegotactuallyreplicated.Replicationontheslavealsowasnotawareofwhenadatabasewasdroppedonthemaster.
3.3addsserver-levelreplication,whichwillreplicatethecurrentandfuturedatabasesfromthemastertotheslaveautomaticallyaftertheinitialsetup.
Inordertosetupglobalreplicationona3.3slaveforalldatabasesofagiven3.3master,thereisnowtheso-calledglobalApplier.Ithasthesameinterfaceastheexistingapplier,butitwillreplicatefromalldatabasesofthemasterandnotjustasingleone.
Inordertostartthereplicationontheslaveandmakeitreplicatealldatabasesfromagivenmaster,usethesecommandsontheslave:
varreplication=require("@arangodb/replication");
replication.setupReplicationGlobal({
endpoint:"tcp://127.0.0.1:8529",
username:"root",
password:"",
autoStart:true
});
Tocheckiftheapplierisrunning,alsousetheglobalApplierobject:
replication.globalApplier.state().state
Theserver-levelreplicationrequiresboththemasterandslaveserverstoArangoDBversion3.3orhigher.
Asynchronousfailover
Aresilientsetupcannoweasilybeachievedbyrunningapairofconnectedservers,ofwhichoneinstancebecomesthemasterandtheotheranasynchronouslyreplicatingslave,withautomaticfailoverbetweenthem.
Twoserversareconnectedviaasynchronousreplication.Oneoftheserversiselectedleader,andtheotheroneismadeafollowerautomatically.Atstartup,thetwoserversfightforleadership.Thefollowerwillautomaticallystartreplicationfromthemasterforallavailabledatabases,usingtheserver-levelreplicationintroducedin3.3.
Whenthemastergoesdown,thisisautomaticallydetectedbyanagencyinstance,whichisalsostartedinthismode.Thisinstancewillmakethepreviousfollowerstopitsreplicationandmakeitthenewleader.
Thefollowerwillautomaticallydenyallreadandwriterequestsfromclientapplications.Onlythereplicationitselfisallowedtoaccessthefollower'sdatauntilthefollowerbecomesanewleader.
Whensendingarequesttoreadorwritedataonafollower,thefollowerwillalwaysrespondwithHTTP503(Serviceunavailable)andprovidetheaddressofthecurrentleader.Clientapplicationsanddriverscanusethisinformationtothenmakeafollow-uprequesttotheproperleader:
HTTP/1.1503ServiceUnavailable
X-Arango-Endpoint:http://[::1]:8531
....
WhatsNewin3.3
529
Clientapplicationscanalsodetectwhothecurrentleaderandthefollowersarebycallingthe/_api/cluster/endpointsRESTAPI.ThisAPIisaccessibleonleadersandfollowersalike.
TheArangoDBstartersupportsstartingtwoserverswithasynchronousreplicationandfailoveroutofthebox.
ThearangojsdriverforJavaScript,theGodriverandtheJavadriverforArangoDBsupportautomaticfailoverincasethecurrentlyaccessedserverendpointrespondswithHTTP503.
Blogarticles:
UsingAutomaticfailoverforresilientsingleserversIntroducingthenewArangoDBJavadriverwithloadbalancingandadvancedfallback
RocksDBthrottlingArangoDB3.3allowswriteoperationstotheRocksDBenginebethrottled,inordertopreventlongerwritestalls.Thethrottlingisadaptive,meaningthatitautomaticallyadaptstotheactualwriterate.Thisresultsinmuchmorestableresponsetimes,whichisbetterforclientapplicationsandclusterhealthtests,becausetimeoutscausedbywritestallsarelesslikelytooccurandtheserverthusnotmistakenlyassumedtobedown.
Blogarticle:RocksDBsmoothingforArangoDBcustomers
Fastershardcreationincluster
Whenusingacluster,onenormallywantsresilience,soreplicationFactorissettoatleast2.Thenumberofshardsisoftensettoratherhighvalueswhencreatingcollections.
Creatingacollectionintheclusterwillmakethecoordinatorstorethesetupmetadataofthenewcollectionintheagencyfirst.Subsequentiallyalldatabaseserversoftheclusterwilldetectthatthereisworktodoandwillbegincreatingtheshards.Thiswillfirsthappenfortheshardleaders.Foreachshardleaderthatfinisheswiththesetup,thesynchronousreplicationwithitsfollowersisthenestablished.Thatwillmakesurethateveryfuturedatamodificationwillnotbecomeeffectiveontheleaderonly,butalsoonallthefollowers.
In3.3thissetupprotocolhasgotsomeshortcutsfortheinitialshardcreation,whichspeedsupcollectioncreationbyroughly50to60percent.
LDAPauthenticationTheLDAPauthenticationmoduleintheEnterpriseeditionhasbeenenhanced.Thefollowingoptionshavebeenaddedtoit:
theoption--server.local-authenticationcontrolswhetherthelocal_userscollectionisalsousedforlookingupusers.Thisisalsothedefaultbehavior.IftheauthenticationshallberestrictedtojusttheLDAPdirectory,theoptioncanbesettotrue,andarangodwillthennotmakeanyqueriestoits_userscollectionwhenlookingupusers.
theoption--server.authentication-timeoutcontrolstheexpirationtimeforcachedLDAPuserinformationentriesinarangod.
basicrolesupporthasbeenaddedfortheLDAPmoduleintheEnterpriseedition.NewconfigurationoptionsforLDAPin3.3are:
--ldap.roles-attribute-name
--ldap.roles-transformation
--ldap.roles-search
--ldap.roles-include
--ldap.roles-exclude
--ldap.superuser-role
PleaserefertoLDAPforadetailedexplanation.
Miscellaneousfeatures
WhatsNewin3.3
530
whencreatingacollectioninthecluster,thereisnowanoptionalparameterenforceReplicationFactor:whenset,thisparameterenforcesthatthecollectionwillonlybecreatediftherearenotenoughdatabaseserversavailableforthedesiredreplicationFactor.
AQLDISTINCTisnotchangingtheorderofprevious(sorted)results
PreviouslytheimplementationofAQLdistinctstoredallencounteredvaluesinahashtableinternally.Whendone,thefinalresultswerereturnedintheorderdictatedbythehashtablethatwasusedtostorethekeys.Thisorderwasmoreorlessunpredictable.Thoughthiswasdocumentedbehavior,itwasinconvenientforendusers.
3.3nowdoesnotchangethesortorderoftheresultanymorewhenDISTINCTisused.
SeveralAQLfunctionshavebeenimplementedinC++,whichcanhelpsavememoryandCPUtimeforconvertingthefunctionargumentsandresults.Thefollowingfunctionshavebeenported:
LEFTRIGHTSUBSTRINGTRIMMATCHES
TheArangoShellpromptsubstitutioncharactershavebeenextended.Nowthefollowingextrasubstitutionscanbeusedforthearangoshprompt:
'%t':currenttimeastimestamp'%a':elpasedtimesinceArangoShellstartinseconds'%p':durationoflastcommandinseconds
Forexample,toshowtheexecutiontimeofthelastcommandexecutedinarangoshintheshell'sprompt,startarangoshusing:
arangosh--console.prompt"%E@%d%p>"
Therearenewstartupoptionsfortheloggingtoaiddebugginganderrorreporting:
--log.role:willshowone-lettercodeofserverrole(A=agent,C=coordinator,...)Thisisespeciallyusefulwhenaggregatinglogs.
Theexistingrolesusedinlogsare:
U:undefined/unclear(usedatstartup)S:singleserverC:coordinatorP:primaryA:agent
--log.line-numbertrue:thisoptionwillnowadditionallyshowthenameoftheC++functionthattriggeredthelogmessage(filenameandlinenumberwerealreadyloggedinpreviousversions)
--log.thread-nametrue:thisnewoptionwilllogthenameoftheArangoDBthreadthattriggeredthelogmessage.WillhavemeaningfuloutputonLinuxonly
maketheArangoShell(arangosh)refillitscollectioncachewhenayet-unknowncollectionisfirstaccessed.Thisfixesthefollowingproblemwhenworkingwiththeshellwhileinanothershellorbyanotherprocessanewcollectionisadded:
arangosh1>db._collections();//shell1listsallcollections
arangosh2>db._create("test");//shell2nowcreatesanewcollection'test'
arangosh1>db.test.insert({});//shell1isnotawareofthecollectioncreated
//inshell2,sotheinsertwillfail
WhatsNewin3.3
531
IncompatiblechangesinArangoDB3.3ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.3,andadjustanyclientprogramsifnecessary.
ThefollowingincompatiblechangeshavebeenmadeinArangoDB3.3:
AQL:duringatraversalifavertexisnotfound,arangodwillnotloganerrorandcontinuewithaNULLvalue,butwillinsteadregisterawarningatthequeryandcontinuewithaNULLvalue.
Ifanon-existingvertexisreferencedfromatraversal,itisnotdesirabletologerrorsasArangoDBcanstoreedgespointingtonon-existingvertices(whichisperfectlyvalidifthelow-levelinsertAPIsareused).Aslinkingtonon-existingverticesmayindicateanissuein/withthedatamodelortheclientapplication,thewarningisregisteredinthequerysoclientapplicationshaveaccesstoit.
ArangoDBusernamesmustnotstartwiththestring:role:.
Thestartupconfigurationparameter--cluster.my-iddoesnothaveanyeffectin3.3.Forcompatibilityreasons,ArangoDB3.3willnotfailonstartupiftheoptionisstillusedintheconfiguration,butitwillsilentlyignorethisoption.
Thestartupconfigurationparameter--cluster.my-local-infoisdeprecatednow.Usingitwillmakearangodlogawarningonstartup.
Serverstartup:therecommendedvaluefortheLinuxkernelsettingin/proc/sys/vm/max_map_countwasincreasedtoavalueeighttimesashighasin3.2.arangodcomparesatstartupiftheeffectivevalueofthissettingispresumablytoolow,anditwillissueawarninginthiscase,recommendingtoincreasethevalue.
Thisisnowmorelikelytohappenthaninpreviousversions,astherecommendedvalueisnoweighttimeshigherthanin3.2.Thestartupwarningswilllooklikethis(withactualnumbersvarying):
WARNING{memory}maximumnumberofmemorymappingsperprocessis65530,whichseemstoolow.itisrecommendedtosetit
toatleast512000
PleaserefertotheLinuxkerneldocumentationformoreinformationonthissetting.ThischangeonlyaffectstheLinuxversionofArangoDB.
Clienttools
Theoption--recycle-idshasbeenremovedfromthearangorestorecommand.Usingthisoptioncouldhaveledtoproblemsontherestore,withpotentialidconflictsbetweentheoriginatingserver(thesourcedumpserver)andthetargetserver(therestoreserver).
Theoption--compatparameterhasbeenremovedfromthearangodumpcommandandthe/_api/replication/dumpRESTAPIendpoint.InordertocreateadumpfromanArangoDB2.8instance,pleaseuseanolderversionoftheclienttools.OlderArangoDBversionsarenolongerbesupportedbythearangodumpandarangorestorebinariesshippedwith3.3.
Miscellaneous
TheminimumsupportedcompilerforcompilingArangoDBfromsourceisnowg++5.4(bumpedupfromg++4.9).ThischangeonlyaffectsusersthatcompileArangoDBontheirown.
Incompatiblechangesin3.3
532
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.2.ArangoDB3.2alsocontainsseveralbugfixesthatarenotlistedhere.
Storageengines
ArangoDB3.2offerstwostorageengines:
thealways-existingmemory-mappedfilesstorageengineanewstorageenginebasedonRocksDB
Memory-mappedfilesstorageengine(MMFiles)
Theformerstorageengine(namedMMFilesenginehenceforth)persistsdatainmemory-mappedfiles.
Anydatachangesaredonefirstintheengine'swrite-aheadlog(WAL).TheWALisreplayedafteracrashsotheengineoffersdurabilityandcrash-safety.DatafromtheWALiseventuallymovedtocollection-specificdatafiles.Thefilesarealwayswritteninanappend-onlyfashion,sodatainfilesisneveroverwritten.Obsoletedatainfileswilleventuallybepurgedbybackgroundcompactionthreads.
Mostofthisengine'sindexesarebuiltinRAM.Whenacollectionisloaded,thisrequiresrebuildingtheindexesinRAMfromthedatastoredondisk.TheMMFilesenginehascollection-levellocking.
Thisstorageengineisagoodchoicewhendata(includingtheindexes)canfitintheserver'savailableRAM.Ifthesizeofdataplusthein-memoryindexesexceedsthesizeoftheavailableRAM,thenthisenginemaytrytoallocatemorememorythanavailable.Thiswilleithermaketheoperatingsystemswapoutpartsofthedata(andcausediskI/O)or,whennoswapspaceisconfigured,invoketheoperatingsystem'sout-of-memoryprocesskiller.
Thelockingstrategyallowsparallelreadsandisoftengoodenoughinread-mostlyworkloads.Writesneedexclusivelocksonthecollections,sotheycanblockotheroperationsinthesamecollection.Thelockingstrategyalsoprovidestransactionalconsistencyandisolation.
RocksDBstorageengine
TheRocksDBstorageengineisnewinArangoDB3.2.Itisdesignedtostoredatasetsthatarebiggerthantheserver'savailableRAM.Itpersistsalldata(includingtheindexes)inaRocksDBinstance.
ThatmeansanydocumentreadorwriteoperationswillbeansweredbyRocksDBunderthehood.RocksDBwillservethedatafromitsownin-RAMcachesorfromdisk.TheRocksDBenginehasawrite-aheadlog(WAL)andusesbackgroundthreadsforcompaction.Itsupportsdatacompression.
TheRocksDBstorageenginehasdocument-levellocking.Readoperationsdonotblockandareneverblockedbyotheroperations.Writeoperationsonlyblockwritesonthesamedocuments/indexvalues.Becausemultiplewriterscanoperateinparallelonthesamecollection,thereisthepossibilityofwrite-writeconflicts.Ifsuchwriteconflictisdetected,oneofthewriteoperationsisabortedwitherror1200("conflict").Clientapplicationscantheneitheraborttheoperationorretry,basedontherequiredconsistencysemantics.
Storageengineselection
ThestorageenginetouseinanArangoDBclusterorasingle-serverinstancemustbeselectedinitially.ThedefaultstorageengineinArangoDB3.2istheMMFilesengineifnostorageengineisselectedexplicitly.Thisensuresallusersupgradingfromearlierversionscancontinuewiththewell-knownMMFilesengine.
Toselectthestorage-engine,thereistheconfigurationoption--server.storage-engine.Itcanbesettoeithermmfiles,rocksdborauto.Whilethefirsttwovalueswillexplicitlyselectastorageengine,theautooptionwillautomaticallychoosethestorageenginebasedonwhichstorageenginewaspreviouslyselected.Ifnoenginewasselectedpreviously,autowillselecttheMMFilesengine.Ifanenginewaspreviouslyselected,theselectionwillbewrittentoafileENGINEintheserver'sdatabasedirectoryandwillbereadfromthereatanysubsequentserverstarts.
WhatsNewin3.2
533
Oncethestorageenginewasselected,theselectioncannotbechangedbyadjusting--server.storage-engine.Inordertoswitchtoanotherstorageengine,itisrequiredtore-starttheserverwithanother(empty)databasedirectory.Inordertousedatacreatedwiththeotherstorageengine,itisrequiredtodumpthedatafirstwiththeoldengineandrestoreitusingthenewstorageengine.Thiscanbeachievedviainvokingarangodumpandarangorestore.
UnlikeinMySQL,thestorageengineselectioninArangoDBisforanentireclusteroranentiresingle-serverinstance.Alldatabasesandcollectionswillusethesamestorageengine.
RocksDBstorageengine:supportedindextypes
TheexistingindexesintheRocksDBengineareallpersistent.Thefollowingindexesaresupportedthere:
primary:thistypeofindexisautomaticallycreated.Itindexes_id/_key
edge:thisindexisautomaticallycreatedforedgecollections.Itindexes_fromand_to
hash,skiplist,persistent:theseareuser-definedindexes,Despitetheirnames,theyareneitherhashnorskiplistindexes.TheseindextypesmaptothesameRocksDB-basedsortedindeximplementation.Thesameistrueforthe"persistent"index.Thenames"hash","skiplist"and"persistent"areonlyusedforcompatibilitywiththeMMFilesenginewheretheseindexesexistedinpreviousandthecurrentversionofArangoDB.
geo:user-definedindexforproximitysearches
fulltext:user-definedsortedrevertedindexonwordsoccurringindocuments
SatelliteCollections
WithSatelliteCollections,youcandefinecollectionstoshardtoaclusterandcollectionstoreplicatetoeachmachine.TheArangoDBqueryoptimizerknowswhereeachshardislocatedandsendstherequeststotheDBServersinvolved,whichthenexecutesthequery,locally.Withthisapproach,networkhopsduringjoinoperationsonshardedcollectionscanbeavoidedandresponsetimescanbeclosetothatofasingleinstance.
SatellitecollectionsareavailableintheEnterpriseedition.
MemorymanagementmakearangodstartwithlessV8JavaScriptcontexts
Thisspeedsuptheserverstartandmakesarangoduselessmemoryatstart.WheneveraV8contextisneededbyaFoxxactionorsomeotherJavaScriptoperationandthereisnousableV8context,anewcontextwillbecreateddynamicallynow.
Upto--javascript.v8-contextsV8contextswillbecreated,sothisoptionwillchangeitsmeaning.PreviouslyasmanyV8contextsasspecifiedbythisoptionwerecreatedatserverstart,andthenumberofV8contextsdidnotchangeatruntime.NowuptothisnumberofV8contextswillbeinuseatthesametime,buttheactualnumberofV8contextsisdynamic.
ThegarbagecollectorthreadwillautomaticallydeleteunusedV8contextsafterawhile.Thenumberofsparecontextswillgodowntoasfewasconfiguredinthenewoption--javascript.v8-contexts-minimum.ActuallythatmanyV8contextsarealsocreatedatserverstart.
ThefirstfewrequestsinnewV8contextsmaytakelongerthanincontextsthathavebeentherealready.PerformancemaythereforesufferabitfortheinitialrequestssenttoArangoDBorwhenthereareonlyfewbutperformance-criticalsituationsinwhichnewV8contextsneedtobecreated.Ifthisisaconcern,itcaneasilybefixedbysetting--javascipt.v8-contexts-minimumand--javascript.v8-contextstoarelativelyhighvalue,whichwillguaranteethatmanynumberofV8contextstobecreatedatstartupandkeptaroundevenwhenunused.
WaitingforanunusedV8contextwillnowalsoabortandwritealogmessageincasenoV8contextcanbeacquired/createdafter60seconds.
thenumberofpendingoperationsinarangodcannowbelimitedtoaconfigurablenumber.Ifthisnumberisexceeded,theserverwillnowrespondwithHTTP503(serviceunavailable).Themaximumsizeofpendingoperationsiscontrolledviathestartupoption--server.maximal-queue-size.Settingitto0means"nolimit".
WhatsNewin3.2
534
thein-memorydocumentrevisionscachewasremovedentirelybecauseitdidnotprovidetheexpectedbenefits.The3.1implementationshadoweddocumentdatainRAM,whichincreasedtheserver'sRAMusagebutdidnotspeedupdocumentlookupstoomuch.
Thisalsoobsoletesthestartupoptions--database.revision-cache-chunk-sizeand--database.revision-cache-target-size.
TheMMFilesenginenowdoesnotuseadocumentrevisionscachebuthasin-memoryindexesandmapsdocumentstoRAMautomaticallyviammapwhendocumentsareaccessed.TheRocksDBenginehasitsownmechanismforcachingaccesseddocuments.
CommunicationLayerHTTPresponsesreturnedbyarangodwillnowincludetheextraHTTPheaderx-content-type-options:nosnifftoworkaroundacross-sitescriptingbuginMSIE
thedefaultvaluefor--ssl.protocolwaschangedfromTLSv1toTLSv1.2.Whennotexplicitlyset,arangodandallclienttoolswillnowuseTLSv1.2.
theJSONdatainallincomingHTTPrequestsinnowvalidatedforduplicateattributenames.
IncomingJSONdatawithduplicateattributenameswillnowberejectedasinvalid.PreviousversionsofArangoDBonlyvalidatedtheuniquenessofattributenamesinsideincomingJSONforsomeAPIendpoints,butnotconsistentlyforallAPIs.
InternalJavaScriptRESTactionswillnowhidetheirstacktracestotheclientunlessinHTTPresponses.Insteadtheywillalwayslogtothelogfile.
JavaScript
updatedV8versionto5.7.0.0
changeundocumentedbehaviourincaseofinvalidrevisionidsinIf-MatchandIf-None-Matchheadersfrom400(BAD)to412(PRECONDITIONFAILED).
changedefaultstringtruncationlengthfrom80charactersto256charactersforprint/printShellfunctionsinArangoShellandarangod.Thiswillemitlongerprefixesofstringvaluesbeforetruncatingthemwith...,whichishelpfulfordebugging.ThischangeismostlyusefulwhenusingtheArangoShell(arangosh).
the@arangodbmodulenowprovidesatimefunctionwhichreturnsthecurrenttimeinsecondsasafloatingpointvaluewithmicrosecondprecision.
FoxxThereisnowanofficialHTTPAPIformanagingservices,allowingservicestobeinstalled,modified,uninstalledandreconfiguredwithouttheadministrativewebinterface.
ItisnowpossibletouploadasingleJavaScriptfileinsteadofaziparchiveifyourservicerequiresnoconfiguration,additionalfilesorsetup.Aminimalmanifestwillbegeneratedautomaticallyuponinstallationandtheuploadedfilewillbeusedastheservice'smainentrypoint.
DistributedGraphProcessing
WeaddedsupportforexecutingdistributedgraphalgorithmsakaPregel.Userscanrunarbitraryalgorithmsonanentiregraph,includinginclustermode.Weimplementedanumberofalgorithmsforvariouswell-knowngraphmeasures:
ConnectedComponentsPageRankShortestPathsCentralityMeasures(CentralityandBetweeness)
WhatsNewin3.2
535
CommunityDetection(viaLabelPropagation,Speakers-ListenersLabelPropagationorDMID)Userscancontributetheirownalgorithms
AQL
Optimizerimprovements
GeoindexesarenowimplicitlyandautomaticallyusedwhenusingappropriateSORT/FILTERstatementsinAQL,withouttheneedtousethesomewhatlimitedspecial-purposegeoAQLfunctionsNEARorWITHIN.
ComparedtousingthespecialpurposeAQLfunctionsthisapproachhastheadvantagethatitismorecomposable,andwillalsohonoranyLIMITvaluesusedintheAQLquery.
ThespecialpurposeNEARAQLfunctioncannowbesubstitutedwiththefollowingAQL(providedthereisageoindexpresentonthedoc.latitudeanddoc.longitudeattributes):
FORdocingeoSort
SORTDISTANCE(doc.latitude,doc.longitude,0,0)
LIMIT5
RETURNdoc
WITHINcanbesubstitutedwiththefollowingAQL:
FORdocingeoFilter
FILTERDISTANCE(doc.latitude,doc.longitude,0,0)<2000
RETURNdoc
Miscellaneousimprovements
addedREGEX_REPLACEAQLfunction
REGEX_REPLACE(text,search,replacement,caseInsensitive)→string
Replacethepatternsearchwiththestringreplacementinthestringtext,usingregularexpressionmatching.
text(string):thestringtosearchinsearch(string):aregularexpressionsearchpatternreplacement(string):thestringtoreplacethesearchpatternwithreturnsstring(string):thestringtextwiththesearchregexpatternreplacedwiththereplacementstringwhereverthepatternexistsintext
addednewstartupoption--query.fail-on-warningtomakeAQLqueriesabortinsteadofcontinuingwithwarnings.
Whensettotrue,thiswillmakeanAQLquerythrowanexceptionandabortincaseawarningoccurs.Thisoptionshouldbeusedindevelopmenttocatcherrorsearly.Ifsettofalse,warningswillnotbepropagatedtoexceptionsandwillbereturnedwiththequeryresults.Thestartupoptioncanalsobeoverridenonaperquery-level.
theslowquerylistnowcontainsthevaluesofbindvariablesusedintheslowqueries.Bindvariablesarealsoprovidedforthecurrentlyrunningqueries.Thishelpsdebuggingsloworblockingqueriesthatusedynamiccollectionnamesviabindparameters.
AQLbreakingchangeincluster:TheSHORTEST_PATHstatementusingedgecollectionnamesinsteadofagraphnamesnowrequirestoexplicitlynamethevertexcollectionnameswithintheAQLqueryinthecluster.ItcanbedonebyaddingWITH<name>atthebeginningofthequery.
Example:
FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]
Nowhastobe:
WITHvertices
FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]
WhatsNewin3.2
536
Thischangeisduetoavoiddeadlocksitationsinclusteredcase.Anerrorstatingtheaboveisincluded.
Clienttools
addeddataexporttool,arangoexport.
arangoexportcanbeusedtoexportcollectionstojson,jsonlorxmlandexportagraphorcollectionstoxgmml.
added"jsonl"asinputfiletypeforarangoimp
added--translateoptionforarangoimptotranslateattributenamesfromtheinputfilestoattriubtenamesexpectedbyArangoDB
The--translateoptioncanbespecifiedmultipletimes(oncepertranslationtobeexecuted).Thefollowingexamplerenamesthe"id"columnfromtheinputfileto"_key",andthe"from"columnto"_from",andthe"to"columnto"_to":
arangoimp--typecsv--filedata.csv--translate"id=_key"--translate"from=_from"--translate"to=_to"
--translateworksforCSVandTSVinputsonly.
added--threadsoptiontoarangoimptospecifythenumberofparallelimportthreads
changeddefaultvalueforclienttoolsoption--server.max-packet-sizefrom128MBto256MB.thisallowstransferringbiggerresultsetsfromtheserverwithouttheclienttoolsrejectingthemasinvalid.
Authentication
addedLDAPauthentication(Enterpriseonly)
Authorization
addedreadonlymodeforuserscollectionlevelauthorizationrights
Readmoreintheoverview.
Foxxthecookiesessiontransportnowsupportsalloptionssupportedbythecookiemethodoftheresponseobject.
it'snowpossibletoprovideyourownversionofthegraphql-syncmodulewhenusingtheGraphQLextensionsforFoxxbypassingacopyofthemoduleusingthenewgraphqloption.
customAPIendpointscannowbetaggedusingthetagmethodtogenerateacleanerSwaggerdocumentation.
MiscellaneousChanges
arangodnowvalidatesseveralOS/environmentsettingsonstartupandwarnsifthesettingsarenon-ideal.Itadditionallywillprintoutwaystoremedytheoptions.
MostofthechecksareexecutedonLinuxsystemsonly.
added"deduplicate"attributeforarrayindexes,whichcontrolswhetherinsertingduplicateindexvaluesfromthesamedocumentintoauniquearrayindexwillleadtoanerrorornot:
//withdeduplicate=true,whichisthedefaultvalue:
db._create("test");
db.test.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:true});
db.test.insert({tags:["a","b"]});
WhatsNewin3.2
537
db.test.insert({tags:["c","d","c"]});//willwork,becausededuplicate=true
db.test.insert({tags:["a"]});//willfail
//withdeduplicate=false
db._create("test");
db.test.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:false});
db.test.insert({tags:["a","b"]});
db.test.insert({tags:["c","d","c"]});//willnotwork,becausededuplicate=false
db.test.insert({tags:["a"]});//willfail
WhatsNewin3.2
538
KnownIssuesThefollowingknownissuesarepresentinthisversionofArangoDBandwillbefixedinfollow-upreleases:
RocksDBstorageengine
TheRocksDBstorageengineisintentionallymissingthefollowingfeaturesthatarepresentintheMMFilesengine:
thedatafiledebugger(arango-dfdb)cannotbeusedwiththisstorageengine
RocksDBhasitsowncrashrecoverysousingthedfdbwillnotmakeanysensehere.
APIsthatreturncollectionpropertiesorfigureswillreturnslightlydifferentattributesfortheRocksDBenginethanfortheMMFilesengine.Forexample,theattributesjournalSize,doCompact,indexBucketsandisVolatilearepresentintheMMFilesenginebutnotintheRocksDBengine.ThememoryusagefiguresreportedforcollectionsintheRocksDBengineareestimatevalues,whereastheyareexactfortheMMFilesengine.
theRocksDBenginedoesnotsupportsomeoperationswhichonlymakesenseinthecontextoftheMMFilesengine.Theseare:
therotatemethodoncollectionstheflushmethodforWALfiles
theRocksDBstorageenginedoesnotsupportvolatilecollections
transactionsarelimitedinsize.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatamodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.
Thethresholdvaluesfortransactionsizescanbeconfiguredgloballyusingthestartupoptions
--rocksdb.intermediate-commit-size:ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.
--rocksdb.intermediate-commit-count:ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.
--rocksdb.max-transaction-size:thisisanupperlimitforthetotalnumberofbytesofalloperationsinatransaction.Iftheoperationsinatransactionconsumemorethanthisthresholdvalue,thetransactionwillautomaticallyabortwitherror32("resourcelimitexceeded").
Itisalsopossibletooverridethesethresholdspertransaction.
Thefollowingknownissueswillberesolvedinfuturereleases:
theRocksDBengineisnotyetperformance-optimizedandpotentiallynotwellconfigured
collectionsforwhichageoindexispresentwillusecollection-levelwritelocksevenwiththeRocksDBengine.Readsfromthesecollectionscanstillbedoneinparallelbutnowrites
modifyingdocumentsinacollectionwithageoindexwillcausemultipleadditionalwritestoRocksDBformaintainingtheindexstructures
thenumberofdocumentsreportedforcollections(db.<collection>.count())maybeslightlywrongduringtransactionsifthereareparalleltransactionsongoingforthesamecollectionthatalsomodifythenumberofdocuments
theanyoperationtoprovidearandomdocumentfromacollectionissupportedbytheRocksDBenginebuttheoperationhasmuchhigheralgorithmiccomplexitythanintheMMFilesengine.Itisthereforediscouragedtocallitforcasesotherthanmanualinspectionofafewdocumentsinacollection
AQLqueriesintheclusterstillissueanextralockingHTTPrequestpershardthoughthiswouldnotbenecessaryfortheRocksDBengineinmostcases
Installer
KnownIssuesin3.2
539
Upgradingfrom3.1to3.2onWindowsrequirestheusertomanuallycopythedatabasedirectorytothenewlocationandrunanupgradeonthedatabase.PleaseconsulttheDocumentationfordetailedinstructions.
SystemIntegration
OnsomeLinuxsystemssystemdandsystemvmightreportthatthearangodbserviceisingoodconditionwhenitcouldnotbestarted.Inthiscasetheuserneedstocheck/var/log/arangodb3forfurtherinformationaboutthefailedstartup.
MacOSX
Storageengineisnotchangeableonanexistingdatabase.Currentlyonlytheinitialselectionofthestorageengineissupported.Inordertouseanotherstorageengine,youhavetodeleteyourArangoDBapplication(MacApplicationFolder)and/Users/<your_user_name>/Library/ArangoDBfolder.
OpenSSL1.1
ArangoDBhasbeentestedwithOpenSSL1.0onlyandwon'tbuildagainst1.1whencompilingonyourown.SeehereforhowtocompileonsystemsthatshipOpenSSL1.1bydefault.
KnownIssuesin3.2
540
IncompatiblechangesinArangoDB3.2ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.2,andadjustanyclientprogramsifnecessary.
AQL
AQLbreakingchangeincluster:TheSHORTEST_PATHstatementusingedge-collectionnamesinsteadofagraphnamenowrequirestoexplicitlynamethevertex-collectionnameswithintheAQLqueryinthecluster.ItcanbedonebyaddingWITH<name>atthebeginningofthequery.
Example:
FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]
Nowhastobe:
WITHvertices
FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]
Thischangeisduetoavoiddead-locksitationsinclusteredcase.Anerrorstatingtheaboveisincluded.
RESTAPIRemovedundocumentedinternalHTTPAPI:
PUT/_api/edgesThedocumentedGET/_api/edgesandtheundocumentedPOST/_api/edgesremainsunmodified.
changeundocumentedbehaviourincaseofinvalidrevisionidsinIf-MatchandIf-None-MatchheadersfromreturningHTTPstatuscode400(badrequest)toreturningHTTPstatuscode412(preconditionfailed).
theRESTAPIforfetchingthelistofcurrentlyrunningAQLqueriesandtheRESTAPIforfetchingthelistofslowAQLqueriesnowreturnanextrabindVarsattributewhichcontainsthebindparametersusedbythequeries.
ThisaffectsthereturnvaluesofthefollowingAPIendpoints:
GET/_api/query/currentGET/_api/query/slow
TheRESTAPIforretrievingindexes(GET/_api/index)nowreturnsthededuplicateattributeforeachindex
TheRESTAPIforcreatingindexes(POST/_api/index)nowacceptstheoptionaldeduplicateattribute
JavaScriptAPI
changeundocumentedbehaviourincaseofinvalidrevisionidsinJavaScriptdocumentoperationsfromreturningerrorcode1239("illegaldocumentrevision")toreturningerrorcode1200("conflict").
thecollection.getIndexes()functionnowreturnsthededuplicateattributeforeachindex
thecollection.ensureIndex()functionnowacceptstheoptionaldeduplicateattribute
Foxx
JWTtokensissuedbythebuilt-inJWTsessionstoragenowcorrectlyspecifytheiatandexpvaluesinsecondsratherthanmillisecondsasspecifiedintheJSONWebTokenstandard.
Incompatiblechangesin3.2
541
Thismayresultinpreviouslyexpiredtokensusingmillisecondsbeingincorrectlyaccepted.ForthisreasonitisrecommendedtoreplacethesigningsecretorsetthenewmaxExpoptiontoareasonablevaluethatissmallerthantheoldestissuedexpirationtimestamp.
ForexamplesettingmaxExpto10**12wouldinvalidateallincorrectlyissuedtokensbefore9September2001withoutimpairingnewtokensuntiltheyear33658(atwhichpointthesetokensarehopefullynolongerrelevant).
ArangoDBrunninginstandalonemodewillcommitallservicesinthejavascript.app-pathtothedatabaseonstartup.ThismayresultinuninstalledservicesshowingupinArangoDBiftheywerenotproperlyremovedfromthefilesystem.
ArangoDBcoordinatorsinaclusternowperformaself-healingstepduringstartuptoensureinstalledservicesareconsistentaccrossallcoordinators.WerecommendbackingupyourservicesandconfigurationbeforeupgradingtoArangoDB3.2,especiallyifyouhavemadeuseofthedevelopmentmode.
Servicesinstalledbeforeupgradingto3.2(includingservicesinstalledonalphareleasesofArangoDB3.2)areNOTpickedupbythecoordinatorself-healingwatchdog.Thiscanbesolvedbyeitherupgrading/replacingtheseservicesorbyusingthe"commit"routeoftheFoxxservicemanagementHTTPAPI,whichcommitstheexactservicesinstalledonagivencoordinatortothecluster.Newserviceswillbepickedupautomatically.
TheformatusedbyFoxxtostoreinternalservicemetadatainthedatabasehasbeensimplifiedandexistingdocumentswillbeupdatedtothenewformat.Ifyouhavemadeanychangestothedatastoredinthe_appssystemcollection,youmaywishtoexportthesechangesastheywillbeoverwritten.
ThereisnowanofficialHTTPAPIformanagingservices.IfyouwerepreviouslyusinganyoftheundocumentedAPIsortheroutesusedbytheadministrativewebinterfacewehighlyrecommendmigratingtothenewAPI.TheoldundocumentedHTTPAPIformananagingservicesisdeprecatedandwillberemovedinafutureversionofArangoDB.
Althoughchangestothefilesystemoutsideofdevelopmentmodewerealreadystronglydiscouraged,thisisareminderthattheyarenolongersupported.Allfilesgeneratedbyservices(whetherbyasetupscriptorduringnormaloperationsuchasuploads)shouldeitherbestoredoutsidetheservicedirectoryorbeconsideredextremelyvolatile.
IntroduceddistinctionbetweenarangoUserandauthorizedinFoxxrequests.ClusterinternalrequestswillneverhaveanarangoUserbutareauthorized.InearlierversionsofArangoDBpartsofthestatisticswerenotaccessiblebythecoordinatorsbecausetheunderlyingFoxxservicecouldn'tauthorizetherequests.Itnowcorrectlychecksthenewreq.authorizedproperty.req.arangoUserstillworksasbefore.Endusersmayusethisnewpropertyaswelltoeasilycheckifarequestisauthorizedornotregardlessofaspecificuser.
Command-lineoptionschanged
--server.maximal-queue-sizeisnowanabsolutemaximum.Ifthequeueisfull,then503isreturned.Settingitto0means"nolimit".Thedefaultvalueforthisoptionisnow0.
thedefaultvaluefor--ssl.protocolhasbeenchangedfrom4(TLSv1)to5(TLSv1.2).
thestartupoptions--database.revision-cache-chunk-sizeand--database.revision-cache-target-sizearenowobsoleteanddonothing
thestartupoption--database.index-threadsoptionisnowobsolete
theoption--javascript.v8-contextsisnowanabsolutemaximum.TheservermaystartlessV8contextsforJavaScriptexecutionatstartup.IfatsomepointtheserverneedsmoreV8contextsitmaystartthemdynamically,untilthenumberofV8contextsreachesthevalueof--javascript.v8-contexts.
theminimumnumberofV8contextstocreateatstartupcanbeconfiguredviathenewstartupoption--javascript.v8-contexts-minimum.
addedcommand-lineoption--javascript.allow-admin-execute
Thisoptioncanbeusedtocontrolwhetheruser-definedJavaScriptcodeisallowedtobeexecutedonserverbysendingviaHTTPtotheAPIendpoint/_admin/executewithanauthenticateduseraccount.Thedefaultvalueisfalse,whichdisablestheexecutionofuser-definedcode.Thisisalsotherecommendedsettingforproduction.Intestenvironments,itmaybeconvenienttoturntheoptiononinordertosendarbitrarysetuporteardowncommandsforexecutionontheserver.
Incompatiblechangesin3.2
542
TheintroductionofthisoptionchangesthedefaultbehaviorofArangoDB3.2:3.2nowbydefaultdisablestheexecutionofJavaScriptcodeviathisAPI,whereasearlierversionsallowedit.Torestoretheoldbehavior,itisnecessarytosettheoptiontotrue.
UsersManagement
Itisnolongersupportedtoaccessthe_userscollecctioninanywaydirectly,exceptthroughtheofficial@arangodb/usersmoduleorthe_apit/usersRESTAPI.
Theaccesstothe_userscollectionfromoutsideofthearangodserverprocessisnowforbidden(Throughdrivers,arangoshortheRESTAPI).Foxxservicesarestillbeabletoaccessthe_userscollectionfornow,butthismightchangeinfutureminorreleases.
Theinternalformatofthedocumentsinthe_userscollectionhaschangedfrompreviousversions
The_queuescollectiononlyallowsread-onlyaccessfromoutsideofthearangodserverprocess.
Accessing_queuesisonlysupportedthroughtheofficial@arangodb/queuesmoduleforFoxxapps.
Incompatiblechangesin3.2
543
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.1.ArangoDB3.1alsocontainsseveralbugfixesthatarenotlistedhere.
SmartGraphs
ArangoDB3.1addsafirstmajorenterpriseonlyfeaturecalledSmartGraphs.SmartGraphsformanadditiontothealreadyexistinggraphfeaturesandallowtoscalegraphsbeyondasinglemachinewhilekeepingalmostthesamequeryperformance.TheSmartGraphfeatureissuggestedforallgraphdatabaseusecasesthatrequireaclusterofdatabaseserversforwhateverreason.Youcaneitherhaveagraphthatistoolargetobestoredonasinglemachineonly.Oryoucanhaveasmallgraph,butatthesametimeneedadditionaldatawithhastobeshardedandyouwanttokeepalloftheminthesameenvirenment.Oryousimplyusetheclusterforhigh-availability.InalltheabovecasesSmartGraphswillsignificantlyincreasetheperformanceofgraphoperations.Formoredetailedinformationreadthismanualsection.
Dataformat
Theformatoftherevisionvaluesstoredinthe_revattributeofdocumentshasbeenchangedin3.1.Upto3.0theywerestringscontaininglargishdecimalnumbers.With3.1,revisionvaluesarestillstrings,butareactuallyencodedtimestampsofthecreationdateoftherevisionofthedocument.Thetimestampsareacquiredusingahybridlogicalclock(HLC)ontheDBserverthatholdstherevision(fortheconceptofahybridlogicalclockseethispaper).Seethismanualsectionfordetails.
ArangoDB>=3.1canArangoDB3.0databasedirectoriesandwillsimplycontinuetousetheold_revattributevalues.Newrevisionswillbewrittenwiththenewtimestamps.
ItishighlyrecommendedtobackupallyourdatabeforeloadingadatabasedirectorythatwaswrittenbyArangoDB<=3.0intoanArangoDB>=3.1.
CommunicationLayerArangoDBupto3.0usedlibevforthecommunicationlayer.ArangoDBstartingfrom3.1usesBoostASIO.
StartingwithArangoDB3.1webegintoprovidetheVelocyStreamProtocol(vst)asaadditiontotheestablishedhttpprotocol.
Afewoptionshavechangedconcerningcommunication,pleasecheckoutIncompatiblechangesin3.1.
ClusterForitsinternalclustercommunicationa(bundledversion)ofcurlisnowbeingused.Thisenablesasynchronousoperationthroughouttheclusterandshouldimprovegeneralperformanceslightly.
Authenticationisnowsupportedwithinthecluster.
Documentrevisionscache
TheArangoDBservernowprovidesanin-memorycacheforfrequentlyaccesseddocumentrevisions.Documentsthatareaccessedduringread/writeoperationsareloadedintotherevisionscacheautomatically,andsubsequentlyservedfromthere.
Thecachehasatotaltargetsize,whichcanbecontrolledwiththestartupoption--database.revision-cache-target-size.Oncethecachereachesthetargetsize,olderentriesmaybeevictedfromthecachetofreememory.Notethatthetargetsizecurrentlyisahighwatermarkthatwilltriggercachememorygarbagecollectionifexceeded.However,ifallcachechunksarestillinusewhenthehighwatermarkisreached,thecachemaystillgrowandallocatemorechunksuntilcacheentriesbecomeunusedandareallowedtobegarbage-collected.
WhatsNewin3.1
544
Thecacheismaintainedonaper-collectionbasis,thatis,memoryforthecacheisallocatedonaper-collectionbasisinchunks.Thesizeforthecachememorychunkscanbecontrolledviathestartupoption--database.revision-cache-chunk-size.Thedefaultvalueis4MBperchunk.Biggerchunksizesallowsavingmoredocumentsperchunk,whichcanleadtomoreefficientchunkallocationandlookups,butwillalsoleadtomemorywasteifmanychunksareallocatedandnotfullyused.Thelatterwillbethecaseifthereexistmanysmallcollectionswhichallallocatetheirownchunksbutnotfullyutilizethembecauseofthelownumberofdocuments.
AQL
Functionsadded
ThefollowingAQLfunctionshavebeenaddedin3.1:
OUTERSECTION(array1,array2,...,arrayn):returnsthevaluesthatoccuronlyonceacrossallarraysspecified.
DISTANCE(lat1,lon1,lat2,lon2):returnsthedistancebetweenthetwocoordinatesspecifiedby(lat1,lon1)and(lat2,lon2).Thedistanceiscalculatedusingthehaversineformula.
JSON_STRINGIFY(value):returnsaJSONstringrepresentationofthevalue.
JSON_PARSE(value):convertsaJSON-encodedstringintoaregularobject
Indexusageintraversals
3.1allowsAQLtraversalstouseotherindexesthanjusttheedgeindex.Traversalswithfiltersonedgescannowmakeuseofmorespecificindexes.Forexample,thequery
FORv,e,pIN2OUTBOUND@start@@edge
FILTERp.edges[0].foo=="bar"
RETURN[v,e,p]
mayuseahashindexon["_from","foo"]insteadoftheedgeindexonjust["_from"].
Optimizerimprovements
MaketheAQLqueryoptimizerinjectfilterconditionexpressionsreferredtobyvariablesduringfilterconditionaggregation.Forexample,inthefollowingquery
FORdocINcollection
LETcond1=(doc.value==1)
LETcond2=(doc.value==2)
FILTERcond1||cond2
RETURN{doc,cond1,cond2}
theoptimizerwillnowinjecttheconditionsforcond1andcond2intothefilterconditioncond1||cond2,expandingitto(doc.value==1)||(doc.value==2)andmakingtheseconditionsavailableforindexsearching.
Notethattheoptimizerpreviouslyalreadyinjectedsomeconditionsintootherconditions,butonlyifthevariablethatdefinedtheconditionwasnotusedelsewhere.Forexample,thefilterconditioninthequery
FORdocINcollection
LETcond=(doc.value==1)
FILTERcond
RETURN{doc}
alreadygotoptimizedbeforebecausecondwasonlyusedonceinthequeryandtheoptimizerdecidedtoinjectitintotheplacewhereitwasused.
Thisonlyworkedforvariablesthatwerereferredtoonceinthequery.Whenavariablewasusedmultipletimes,theconditionwasnotinjectedasinthefollowingquery
FORdocINcollection
WhatsNewin3.1
545
LETcond=(doc.value==1)
FILTERcond
RETURN{doc,cond}
3.1allowsusingthisconditionsothatthequerycanuseanindexondoc.value(ifsuchindexexists).
Miscellaneousimprovements
Theperformanceofthe[*]operatorwasimprovedforcasesinwhichthisoperatordidnotuseanyfilters,projectionsand/oroffset/limits.
TheAQLqueryexecutorcannowreportthetimerequiredforloadingandlockingthecollectionsusedinanAQLquery.Whenprofilingisenabled,itwillreportthetotalloadingandlockingtimeforthequeryintheloadingcollectionssub-attributeoftheextra.profilevalueoftheresult.TheloadingandlockingtimecanalsobeviewintheAQLqueryeditorinthewebinterface.
AuditLogAuditlogginghasbeenadded,seeAuditing.
ClienttoolsAddedoption--skip-linesforarangoimpThisallowsskippingthefirstfewlinesfromtheimportfileincasetheCSVorTSVimportareusedandsomeinitiallinesshouldbeskippedfromtheinput.
WebAdminInterfaceTheusabilityoftheAQLeditorsignificantlyimproved.InadditiontothestandardJSONoutput,theAQLEditorisnowabletorenderqueryresultsasagraphprevieworatable.FurthermoretheAQLeditordisplaysqueryprofilinginformation.
AddedanewGraphViewerinordertoexchangethetechnicallyobsoleteversion.ThenewGraphViewerisbasedonCanvasbutdoesalsoincludeafirstWebGLimplementation(limitedfunctionality-willchangeinthefuture).ThenewGraphVieweroffersasmoothwaytodiscoverandvisualizeyourgraphs.
Theshardviewinclustermodenowdisplaysaprogressindicatorwhilemovingshards.
AuthenticationUptoArangoDB3.0authenticationofclientrequestswasonlypossiblewithHTTPbasicauthentication.
Startingwith3.1itisnowpossibletoalsouseaJSONWebTokens(JWT)forauthenticatingincomingrequests.
FordetailschecktheHTTPauthenticationchapter.Bothauthenticationmethodsarevalidandwillbesupportedinthenearfuture.Usewhateversuitsyoubest.
Foxx
GraphQL
ItisnoweasytogetstartedwithprovidingGraphQLAPIsinFoxx,seeFoxxGraphQL.
OAuth2
FoxxnowofficiallyprovidesamoduleforimplementingOAuth2clients,seeFoxxOAuth2.
Per-routemiddleware
WhatsNewin3.1
546
It'snowpossibletospecifymiddlewarefunctionsforaroutewhendefiningaroutehandler.Thesemiddlewarefunctionsonlyapplytothesinglerouteandsharetheroute'sparameterdefinitions.CheckouttheFoxxRouterdocumentationformoreinformation.
WhatsNewin3.1
547
IncompatiblechangesinArangoDB3.1ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.1,andadjustanyclientprogramsifnecessary.
CommunicationLayer
TheinternalcommicationlayerisnowbasedonBoostASIO.Afewoptionsregardingthreadsandcommunicationhavebeenchanged.
Therearenolongertwodifferentthreadspools(--scheduler.threadsand--server.threads).Theoption--scheduler.threadshasbeenremoved.Thenumberofthreadsisnowcontrolledbytheoption--server.threadsonly.Bydefault--server.threadsissettothenumberofhyper-cores.
Asaconsequenceofthechange,thefollowing(hidden)startupoptionshavebeenremoved:
--server.extra-threads
--server.aql-threads
--server.backend
--server.show-backends
--server.thread-affinity
AQL
ThebehavioroftheAQLarraycomparisonoperatorshaschangedforemptyarrays:
ALLandANYnowalwaysreturnfalsewhentheleft-handoperandisanemptyarray.Thebehaviorfornon-emptyarraysdoesnotchange:
[]ALL==1willreturnfalse[1]ALL==1willreturntrue[1,2]ALL==1willreturnfalse[2,2]ALL==1willreturnfalse[]ANY==1willreturnfalse[1]ANY==1willreturntrue[1,2]ANY==1willreturntrue[2,2]ANY==1willreturnfalse
NONEnowalwaysreturnstruewhentheleft-handoperandisanemptyarray.Thebehaviorfornon-emptyarraysdoesnotchange:
[]NONE==1willreturntrue[1]NONE==1willreturnfalse[1,2]NONE==1willreturnfalse[2,2]NONE==1willreturntrue
WITHinclustertraversalsisnowmandatoryinordertoavoiddeadlocks.
Dataformatchanges
TheattributemaximalSizehasbeenrenamedtojournalSizeincollectionmeta-datafiles("parameter.json").FilescontainingthemaximalSizeattributewillstillbepickedupcorrectlyfornot-yetadjustedcollections.
Theformatoftherevisionvaluesstoredinthe_revattributeofdocumentshasbeenchangedin3.1.Upto3.0theywerestringscontaininglargishdecimalnumbers.With3.1,revisionvaluesarestillstrings,butareactuallyencodedtimestampsofthecreationdateoftherevisionofthedocument.Thetimestampsareacquiredusingahybridlogicalclock(HLC)ontheDBserverthatholdstherevision(fortheconceptofahybridlogicalclockseethispaper).Seethismanualsectionfordetails.
Incompatiblechangesin3.1
548
ArangoDB>=3.1canArangoDB3.0databasedirectoriesandwillsimplycontinuetousetheold_revattributevalues.Newrevisionswillbewrittenwiththenewtimestamps.
ItishighlyrecommendedtobackupallyourdatabeforeloadingadatabasedirectorythatwaswrittenbyArangoDB<=3.0intoanArangoDB>=3.1.
Tochangeallyourold_revattributesintonewstyletimestampsyouhavetousearangodumptodumpalldataout(usingArangoDB3.0),andusearangorestoreintothenewArangoDB3.1,whichisthesafestwaytoupgrade.
Thechangealsoaffectsthereturnformatof_revvaluesandotherrevisionvaluesinHTTPAPIs(seebelow).
HTTPAPIchanges
APIsadded
ThefollowingHTTPRESTAPIshavebeenaddedforonlinelogleveladjustmentoftheserver:
GET/_admin/log/levelreturnsthecurrentloglevelsettingsPUT/_admin/log/levelmodifiesthecurrentloglevelsettings
APIschanged
thefollowingRESTAPIsthatreturnrevisionidsnowmakeuseofthenewrevisionidformatintroducedin3.1.Allrevisionidsreturnedwillbestringsasin3.0,buthaveadifferentinternalformat.
ThefollowingAPIsareaffected:
GET/_api/collection/{collection}/checksum:revisionattributeGET/_api/collection/{collection}/revision:revisionattributeallotherAPIsthatreturndocuments,whichmayincludethedocuments'_revattribute
Clientapplicationsshouldnottrytointerprettheinternalsofrevisionvalues,butonlyuserevisionvaluesforcheckingwhethertworevisionstringsareidentical.
thereplicationRESTAPIswillnowusetheattributenamejournalSizeinsteadofmaximalSizewhenreturninginformationaboutcollections.
thedefaultvalueforkeepNullhasbeenchangedfromfalsetotrueforthefollowingpartialupdateoperationsforverticesandedgesin/_api/gharial:
PATCH/_api/gharial/{graph}/vertex/{collection}/{key}PATCH/_api/gharial/{graph}/edge/{collection}/{key}
ThevalueforkeepNullcanstillbesetexplicitlytofalsebysettingtheURLparameterkeepNulltoavalueoffalse.
theRESTAPIfordroppingcollections(DELETE/_api/collection)nowacceptsanoptionalquerystringparameterisSystem,whichcansettotrueinordertodropsystemcollections.Iftheparameterisnotsetornotsettotrue,theRESTAPIwillrefusetodropsystemcollections.InpreviousversionsofArangoDB,theisSystemparameterdidnotexist,andtherewasnodistinctionbetweensystemandnon-systemcollectionswhendroppingcollections.
theRESTAPIforretrievingAQLqueryresults(POST/_api/cursor)willnowreturnanadditionalsub-attributeloadingcollectionsthatwillcontainthetotaltimerequiredforloadingandlockingcollectionsduringtheAQLquerywhenprofilingisenabled.Theattributecanbefoundintheextraresultattributeinsub-attributeloadingcollections.Theattributewillonlybesetifprofilingwasenabledforthequery.
FoxxTesting
TheQUnitinterfacetoMochahasbeenremoved.Thisaffectsthebehaviourofthesuite,test,before,after,beforeEachandafterEachfunctionsinFoxxtestsuites.ThesuiteandtestfunctionsarenowprovidedbytheTDDinterface.Thebefore,after,beforeEachandafterEachfunctionsarenowprovidedbytheBDDinterface.
Thisshouldnotcauseanyproblemswithexistingtestsbutmayresultinfailuresintestcasesthatpreviouslypassedforthewrongreasons.Specificallytheexecutionorderofthebefore,after,etcfunctionsnowfollowstheintendedorderandisnolongerarbitrary.
Incompatiblechangesin3.1
549
FordetailsontheexpectedbehaviourofthesefunctionsseethetestingchapterintheFoxxdocumentation.
Incompatiblechangesin3.1
550
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.0.ArangoDB3.0alsocontainsseveralbugfixesthatarenotlistedhere.
Internaldataformatchanges
ArangoDBnowusesVelocyPackforstoringdocuments,queryresultsandtemporarilycomputedvalues.Usingasingledataformatremovedtheneedforsomedataconversionsinthecorethatslowedoperationsdownpreviously.
TheVelocyPackformatisalsoquitecompact,andreducesstoragespacerequirementsfor"small"valuessuchasboolean,integers,shortstrings.ThiscanspeedupseveraloperationsinsideAQLqueries.
VelocyPackdocumententriesstoredondiskarealsoself-contained,inthesensethateachstoreddocumentwillcontainallofitsdatatypeandattributenamedescriptions.Whilethismayrequireabitmorespaceforstoringthedocuments,itremovestheoverheadoffetchingattributenamesanddocumentlayoutfromsharedstructuresasinpreviousversionsofArangoDB.Italsosimplifiesthecodepathsforstoringandreadingdocuments.
AQLimprovements
Syntaximprovements
LIKEstring-comparisonoperator
AQLnowprovidesaLIKEoperatorandcanbeusedtocomparestringslikethis,forexampleinsidefilterconditions:
valueLIKEsearch
ThischangemakesLIKEanAQLkeyword.UsingLIKEasanattributeorcollectionnameinAQLthusrequiresquotingthenamefromnowon.
TheLIKEoperatoriscurrentlyimplementedbycallingthealreadyexistingAQLfunctionLIKE,whichalsoremainsoperationalin3.0.UsetheLIKEfunctionincaseyouwanttosearchcase-insensitive(optionalparameter),astheLIKEoperatoralwayscomparescase-sensitive.
AQLarraycomparisonoperators
AllAQLcomparisonoperatorsnowalsoexistinanarrayvariant.Inthearrayvariant,theoperatorisprecededwithoneofthekeywordsALL,ANYorNONE.Usingoneofthesekeywordschangestheoperatorbehaviortoexecutethecomparisonoperationforall,any,ornoneofitslefthandargumentvalues.Itisthereforeexpectedthatthelefthandargumentofanarrayoperatorisanarray.
Examples:
[1,2,3]ALLIN[2,3,4]//false
[1,2,3]ALLIN[1,2,3]//true
[1,2,3]NONEIN[3]//false
[1,2,3]NONEIN[23,42]//true
[1,2,3]ANYIN[4,5,6]//false
[1,2,3]ANYIN[1,42]//true
[1,2,3]ANY==2//true
[1,2,3]ANY==4//false
[1,2,3]ANY>0//true
[1,2,3]ANY<=1//true
[1,2,3]NONE<99//false
[1,2,3]NONE>10//true
[1,2,3]ALL>2//false
[1,2,3]ALL>0//true
[1,2,3]ALL>=3//false
["foo","bar"]ALL!="moo"//true
WhatsNewin3.0
551
["foo","bar"]NONE=="bar"//false
["foo","bar"]ANY=="foo"//true
Regularexpressionstring-comparisonoperators
AQLnowsupportstheoperators=~and!~fortestingstringsagainstregularexpressions.=~testsifastringvaluematchesaregularexpression,and!~testsifastringvaluedoesnotmatcharegularexpression.
Thetwooperatorsexpecttheirleft-handoperandstobestrings,andtheirright-handoperandstobestringscontainingvalidregularexpressionsasspecifiedbelow.
Theregularexpressionsmayconsistofliteralcharactersandthefollowingcharactersandsequences:
.–thedotmatchesanysinglecharacterexceptlineterminators.Toincludelineterminators,use[\s\S]insteadtosimulate.withDOTALLflag.\d–matchesasingledigit,equivalentto[0-9]\s–matchesasinglewhitespacecharacter\S–matchesasinglenon-whitespacecharacter\t–matchesatabcharacter\r–matchesacarriagereturn\n–matchesaline-feedcharacter[xyz]–setofcharacters.matchesanyoftheenclosedcharacters(i.e.x,yorzinthiscase[^xyz]–negatedsetofcharacters.matchesanyothercharacterthantheenclosedones(i.e.anythingbutx,yorzinthiscase)[x-z]–rangeofcharacters.Matchesanyofthecharactersinthespecifiedrange,e.g.[0-9A-F]tomatchanycharacterin0123456789ABCDEF[^x-z]–negatedrangeofcharacters.Matchesanyothercharacterthantheonesspecifiedintherange(xyz)–definesandmatchesapatterngroup(x|y)–matcheseitherxory –matchesthebeginningofthestring(e.g. xyz)$–matchestheendofthestring(e.g.xyz$)
Notethatthecharacters.,*,?,[,],(,),{,}, ,and$haveaspecialmeaninginregularexpressionsandmayneedtobeescapedusingabackslash(\\).Aliteralbackslashshouldalsobeescapedusinganotherbackslash,i.e.\\\\.
Charactersandsequencesmayoptionallyberepeatedusingthefollowingquantifiers:
x*–matcheszeroormoreoccurrencesofxx+–matchesoneormoreoccurrencesofxx?–matchesoneorzerooccurrencesofxx{y}–matchesexactlyyoccurrencesofxx{y,z}–matchesbetweenyandzoccurrencesofxx{y,}–matchesatleastyoccurencesofx
Enclosingidentifiersinforwardticks
AQLidentifierscannowoptionallybeenclosedinforwardticksinadditiontousingbackwardticks.ThisallowsconvenientwritingofAQLqueriesinJavaScripttemplatestrings(whicharedelimitedwithbackticksthemselves),e.g.
varq=`FORdocIN´collection´RETURNdoc.´name´`;
Functionsadded
ThefollowingAQLfunctionshavebeenaddedin3.0:
REGEX_TEST(value,regex):testswhetherthestringvaluematchestheregularexpressionspecifiedinregex.Returnstrueifitmatches,andfalseotherwise.
Thesyntaxforregularexpressionsisthesameasfortheregularexpressionoperators=~and!~.
WhatsNewin3.0
552
HASH(value):Calculatesahashvalueforvalue.valueisnotrequiredtobeastring,butcanhaveanydatatype.Thecalculatedhashvaluewilltakethedatatypeofvalueintoaccount,soforexamplethenumber1andthestring"1" willhavedifferenthashvalues.Forarraysthehashvalueswillbecrearedifthearrayscontainexactlythesamevalues(includingvaluetypes)inthesameorder.Forobjectsthesamehashvalueswillbecreatediftheobjectshaveexactlythesameattributenamesandvalues(includingvaluetypes).Theorderinwhichattributesappearinsideobjectsisnotimportantforhashing.Thehashvaluereturnedbythisfunctionisanumber.ThehashalgorithmisnotguaranteedtoremainthesameinfutureversionsofArangoDB.Thehashvaluesshouldthereforebeusedonlyfortemporarycalculations,e.g.tocompareiftwodocumentsarethesame,orforgroupingvaluesinqueries.
TYPENAME(value):Returnsthedatatypenameofvalue.Thedatatypenamecanbeeithernull,bool,number,string,arrayorobject.
LOG(value):Returnsthenaturallogarithmofvalue.ThebaseisEuler'sconstant(2.71828...).
LOG2(value):Returnsthebase2logarithmofvalue.
LOG10(value):Returnsthebase10logarithmofvalue.
EXP(value):ReturnsEuler'sconstant(2.71828...)raisedtothepowerofvalue.
EXP2(value):Returns2raisedtothepowerofvalue.
SIN(value):Returnsthesineofvalue.
COS(value):Returnsthecosineofvalue.
TAN(value):Returnsthetangentofvalue.
ASIN(value):Returnsthearcsineofvalue.
ACOS(value):Returnsthearccosineofvalue.
ATAN(value):Returnsthearctangentofvalue.
ATAN2(y,x):Returnsthearctangentofthequotientofyandx.
RADIANS(value):Returnstheangleconvertedfromdegreestoradians.
DEGREES(value):Returnstheangleconvertedfromradianstodegrees.
Optimizerimprovements
"inline-subqueries"rule
TheAQLoptimizerrule"inline-subqueries"hasbeenadded.ThisrulecanpulloutcertainsubqueriesthatareusedasanoperandtoaFORlooponelevelhigher,eliminatingthesubquerycompletely.Thisreducescomplexityofthequery'sexecutionplanandwilllikelyenablefurtheroptimizations.Forexample,thequery
FORiIN(
FORjIN[1,2,3]
RETURNj
)
RETURNi
willbetransformedbytheruleto:
FORiIN[1,2,3]
RETURNi
Thequery
FORnameIN(
FORdocIN_users
FILTERdoc.status==1
RETURNdoc.name
)
LIMIT2
WhatsNewin3.0
553
RETURNname
willbetransformedinto
FORtmpIN_users
FILTERtmp.status==1
LIMIT2
RETURNtmp.name
TherulewillonlyfirewhenthesubqueryisusedasanoperandtoaFORloop,andifthesubquerydoesnotcontainaCOLLECTwithanINTOvariable.
"remove-unnecessary-calculations"rule
TheAQLoptimizerrule"remove-unnecessary-calculations"nowfiresinmorecasesthaninpreviousversions.Thisruleremovescalculationsfromexecutionplans,andbyhavinglesscalculationsdone,aquerymayexecutefasterorrequireslessmemory.
Therulewillnowremovecalculationsthatareusedexactlyonceinotherexpressions(e.g.LETa=docRETURNa.value)andcalculations,orcalculationsthatarejustreferencestoothervariables(e.g.LETa=b).
"optimize-traversals"rule
TheAQLoptimizerrule"merge-traversal-filter"wasrenamedto"optimize-traversals".TherulewillremoveunusededgeandpathresultvariablesfromthetraversalincasetheyarespecifiedintheFORsectionofthetraversal,butnotreferencedlaterinthequery.Thissavesconstructingedgesandpathsresultsthatarenotusedlater.
AQLnowusesVelocyPackinternallyforstoringintermediatevalues.Formanyvaluetypesitcannowgetawaywithoutextramemoryallocationsandlessinternalconversions.ValuescanbepassedintointernalAQLfunctionswithoutcopyingthem.ThiscanleadtoreducedqueryexecutiontimesforqueriesthatuseC++-basedAQLfunctions.
"replace-or-with-in"and"use-index-for-sort"rules
Theserulesnowfireinsomeadditionalcases,whichallowssimplifyingindexlookupconditionsandremovingSortNodesfromexecutionplans.
Clusterstatemanagement
Thecluster'sinternalstateinformationisnowalsomanagedbyArangoDBinstances.Earlierversionsreliedonthirdpartysoftwarebeinginstalledforthestoringtheclusterstate.ThestateismanagedbydedicatedArangoDBinstances,whichcanbestartedinaspecialagencymode.Theseinstancescanoperateinadistributedfashion.Theywillautomaticallyelectoneofthemtobecometheirleader,beingresponsibileforstoringthestatechangessentfromserversinthecluster.Theotherinstanceswillautomaticallyfollowtheleaderandwilltransparentlystandinshoulditbecomeunavailable.Theagencyinstancesarealsoself-organizing:theywillcontinuouslyprobeeachotherandre-electleaders.Thecommunicationbetweentheagencyinstancesusetheconsensus-basedRAFTprotocol.
TheoperationsforstoringandretrievingclusterstateinformationarenowmuchlessexpensivefromanArangoDBclusternodeperspective,whichinturnallowsforfasterclusteroperationsthatneedtofetchorupdatetheoverallclusterstate.
_fromand _toattributesofedgesareupdatableandusableinindexes
InArangoDBpriorto3.0theattributes_fromand_toofedgesweretreatedspeciallywhenloadingorstoringedges.Thatspecialhandlingledtotheseattributesbeingnotasflexibleasregulardocumentattributes.Forexample,the_fromand_toattributevaluesofanexistingedgecouldnotbeupdatedoncetheedgewascreated.Nowthisispossibleviathesingle-documentAPIsandviaAQL.
Additionally,the_fromand_toattributescouldnotbeindexedinuser-definedindexes,e.g.tomakeeachcombinationof_fromand_tounique.Finally,as_fromand_toreferencedthelinkedcollectionsbycollectionidandnotbycollectionname,theirmeaningbecameunclearonceareferencedcollectionwasdropped.Thecollectionidstoredinedgesthenbecameunusable,andwhenaccessing
WhatsNewin3.0
554
suchedgethecollectionnamepartofitwasalwaystranslatedto_undefined.
InArangoDB3.0,the_fromand_tovaluesofedgesaresavedasregularstrings.Thisallowsusing_fromand_toinuser-definedindexes.Additionally,thisallowstoupdatethe_fromand_tovaluesofexistingedges.Furthermore,collectionsreferencedby_fromand_tovaluesmaybedroppedandre-createdlater.Any_fromand_tovaluesofedgespointingtosuchdroppedcollectionareunaffectedbythedropoperationnow.
UnifiedAPIsforCRUDoperations
TheCRUDAPIsfordocumentsandedgehavebeenunified.EdgescannowbeinsertedandmodifiedviathesameAPIsasdocuments._fromand_toattributevaluescanbepassedasregulardocumentattributesnow:
db.myedges.insert({_from:"myvertices/some",_to:"myvertices/other",...});
Passing_fromand_toseparatelyasitwasrequiredinearlierversionsisnotnecessaryanymorebutwillstillwork:
db.myedges.insert("myvertices/some","myvertices/other",{...});
TheCRUDoperationsnowalsosupportbatchvariantsthatworksonarraysofdocuments/edges,e.g.
db.myedges.insert([
{_from:"myvertices/some",_to:"myvertices/other",...},
{_from:"myvertices/who",_to:"myvertices/friend",...},
{_from:"myvertices/one",_to:"myvertices/two",...},
]);
ThebatchvariantsarealsoavailableinArangoDB'sHTTPAPI.Theycanbeusedtomoreefficientlycarryoutoperationswithmultipledocumentsthantheirsingle-documentequivalents,whichrequiredoneHTTPrequestperoperation.Withthebatchoperations,theHTTPrequest/responseoverheadcanbeamortizedacrossmultipleoperations.
Persistentindexes
ArangoDB3.0providesanexperimentalpersistentindexfeature.Persistentindexesstoretheindexvaluesondiskinsteadofin-memoryonly.Thismeanstheindexesdonotneedtoberebuiltin-memorywhenacollectionisloadedorreloaded,whichshouldimprovecollectionloadingtimes.
ThepersistentindexesinArangoDBarebasedontheRocksDBengine.Tocreateapersistentindexforacollection,createanindexoftype"rocksdb"asfollows:
db.mycollection.ensureIndex({type:"rocksdb",fields:["fieldname"]});
Thepersistentindexesaresorted,sotheyallowequalitylookupsandrangequeries.Notethatthefeatureisstillhighlyexperimentalandhassomeknowndeficiencies.Itwillbefinalizeduntilthereleaseofthe3.0stableversion.
UpgradedV8version
TheV8enginethatisusedinsideArangoDBtoexecuteJavaScriptcodehasbeenupgradedfromversion4.3.61to5.0.71.39.ThenewversionmakesseveralmoreES6featuresavailablebydefault,including
arrowfunctionscomputedpropertynamesrestparametersarraydestructuringnumericandobjectliterals
WhatsNewin3.0
555
WebAdminInterface
TheArangoDB3.0webinterfaceissignificantlyimproved.Itnowcomeswithamoreresponsivedesign,makingiteasiertouseondifferentdevices.Navigationandmenushavebeensimplified,andrelateditemshavebeenregroupedtostayclosertogetherandallowtighterworkflows.
TheAQLqueryeditorisnowmucheasiertouse.Multiplequeriescanbestartedandtrackedinparallel,whileresultsofearlierqueriesarestillpreserved.Queriesstillrunningcanbecanceleddirectlyfromtheeditor.TheAQLqueryeditornowallowstheusageofbindparameterstoo,andprovidesahelperforfindingcollectionnames,AQLfunctionnamesandkeywordsquickly.
Thewebinterfacenowkeepstrackofwhethertheserverisofflineandofwhichserver-sideoperationshavebeenstartedandarestillrunning.Itnowremainsusablewhilesuchlonger-runningoperationsareongoing.Italsokeepsmorestateaboutuser'schoices(e.g.windowssizes,whetherthetreeorthecodeviewwaslastusedinthedocumenteditor).
Clusterstatisticsarenowintegratedintothewebinterfaceaswell.Additionally,amenuitem"Helpus"hasbeenaddedtoeasilyprovidetheArangoDBteamfeedbackabouttheproduct.
Thefrontendmaynowbemountedbehindareverseproxyonadifferentpath.ForthistoworktheproxyshouldsendaX-Script-Nameheadercontainingthepath.
Abackendconfigurationforhaproxymightlooklikethis:
reqaddX-Script-Name:\/arangodb
Thefrontendwillrecognizethesubpathandproduceappropriatelinks.ArangoDBwillonlyacceptpathsfromtrustedfrontendproxies.Trustedproxiesmaybeaddedonstartup:
--frontend.proxy-request-checktrue--frontend.trusted-proxy192.168.1.117
--frontend.trusted-proxymaybeanyaddressornetmask.
Todisablethecheckandblindlyacceptanyx-script-nameset--frontend.proxy-request-checktofalse.
Foxximprovements
TheFoxxframeworkhasbeencompletelyrewrittenfor3.0withanew,simplerandmorefamiliarAPI.Themostnotablechangesare:
Legacymodefor2.8services
Stuckwitholdcode?Youcancontinueusingyour2.8-compatibleFoxxserviceswith3.0byadding"engines":{"arangodb":"^2.8.0"}(orsimilarversionrangesthatexclude3.0andup)totheservicemanifest.
Nomoreglobalvariablesandmagicalcomments
TheapplicationContextisnowmodule.context.Insteadofmagicalcommentsjustusethesummaryanddescriptionmethodstodocumentyourroutes.
RepositoryandModelhavebeenremoved
InsteadofrepositoriesjustuseArangoDBcollectionsdirectly.Forvalidationsimplyusethejoischemas(butwrappedinjoi.object())thatpreviouslylivedinsidethemodel.CollectionsandqueriesreturnplainJavaScriptobjects.
Controllershavebeenreplacedwithnestablerouters
Createrouterswithrequire('@arangodb/foxx/router')(),attachthemtoyourservicewithmodule.context.use(router).Becauseroutersarenolongermountedautomagically,youcanexportandimportthemlikeanyotherobject.Userouter.use('/path',subRouter)tonestroutersasdeeplyasyouwant.
Routescanbenamedandreversed
NomorememorizingURLs:addanametoyourroutelikerouter.get('/hello/:name',function(){...},'hello')andredirecttothefullURLwithres.redirect(req.resolve('hello',{name:'world'})).
WhatsNewin3.0
556
Simplerexpress-likemiddleware
Ifyoualreadyknowexpress,thisshouldbefamiliar.Here'sarequestloggerinthreelinesofcode:
router.use(function(req,res,next){
varstart=Date.now();
try{next();}
finally{console.log(`${req.method}${req.url}${res.statusCode}${Date.now()-start}ms`);}
});
Sessionsandauthwithoutdependencies
Tomakeiteasiertogetstarted,thefunctionalitypreviouslyprovidedbythesimple-auth,oauth2,sessions-localandsessions-jwtserviceshavebeenmovedintoFoxxasthe@arangodb/foxx/auth,@arangodb/foxx/oauth2and@arangodb/foxx/sessionsmodules.
Logging
ArangoDB'sloggingisnowgroupedintotopics.Thelogverbosityandoutputfilescanbeadjustedperlogtopic.Forexample
--log.levelstartup=trace--log.levelqueries=trace--log.levelinfo
willlogmessagesconcerningstartupattracelevel,AQLqueriesattracelevelandeverythingelseatinfolevel.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.
Somerelevantlogtopicsavailablein3.0are:
collector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperations(includingmsync)queries:executedAQLqueries,slowqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads
Thisalsoallowsdirectinglogoutputtodifferentfilesbasedontopics.Forexample,tologallAQLqueriestoafile"queries.log"onecanusetheoptions:
--log.levelqueries=trace--log.outputqueries=file:///path/to/queries.log
ToadditionallylogHTTPrequesttoafilenamed"requests.log"addtheoptions:
--log.levelrequests=info--log.outputrequests=file:///path/to/requests.log
Buildsystem
ArangoDBnowusesthecross-platformbuildsystemCMakeforallitsbuilds.Previousversionsusedtwodifferentbuildsystems,makingdevelopmentandcontributionsharderthannecessary.Nowthebuildsystemisunified,andalltargets(Linux,Windows,MacOS)arebuiltfromthesamesetofbuildinstructions.
Documentation
Thedocumentationhasbeenenhancedandre-organizedtobemoreintuitive.
WhatsNewin3.0
557
AnewintroductionforbeginnersshouldbringyouuptospeedwithArangoDBinlessthananhour.Additionaltopicshavebeenintroducedandwillbeextendedwithupcomingreleases.
ThetopicsAQLandHTTPAPIarenowseparatedfromthemanualforbettersearchabilityandlessconfusion.Aversionswitchermakesiteasiertojumptotheversionofthedocsyouareinterestedin.
WhatsNewin3.0
558
IncompatiblechangesinArangoDB3.0ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.0,andadjustanyclientprogramsifnecessary.
Buildsystem
BuildingArangoDB3.0fromsourcenowrequiresCMake.
Thepre-3.0buildsystemusedaconfigure-basedapproach.ThestepstobuildArangoDB2.8fromsourcecodewere:
makesetup
./configure<options>
make
Thesestepswillnotworkanymore,asArangoDB3.0doesnotcomewithaconfigurescript.
Tobuild3.0onLinux,createaseparatebuilddirectoryfirst:
mkdir-pbuild
andthencreatetheinitialbuildscriptsonceusingCMake:
(cdbuild&&cmake<options>..)
Theabovecommandwillconfigurethebuildandcheckfortherequireddependencies.Ifeverythingworkswelltheactualbuildcanbestartedwith
(cdbuild&&make)
ThebinariesfortheArangoDBserverandallclienttoolswillthenbecreatedinsidethebuilddirectory.TostartArangoDBlocallyfromthebuilddirectory,use
build/bin/arangod<options>
Datafilesanddatafilenames
ArangoDB3.0usesanewVelocyPack-basedformatforstoringdatainWALlogfilesandcollectiondatafiles.ThefileformatisnotcompatiblewiththefilesusedinpriorversionsofArangoDB.ThatmeansdatafileswrittenbyArangoDB3.0cannotbeusedinearlierversionsandviceversa.
Thepatternforcollectiondirectorynameswaschangedin3.0toincludearandomidcomponentattheend.Thenewpatterniscollection-<id>-<random>,where<id>isthecollectionidand<random>isarandomnumber.PreviousversionsofArangoDBusedapatterncollection-<id>withouttherandomnumber.
UserManagementUnlikeArangoDB2.x,ArangoDB3.0usersarenowseparatedfromdatabases,andyoucangrantoneormoredatabasepermissionstoauser.
IfyouwanttomimicthebehaviorofArangoDB,youshouldnameyouruserslikeusername@dbname.
Usersthatcanaccessthe_systemdatabaseareallowedtomanageusersandpermissionsforalldatabases.
Incompatiblechangesin3.0
559
Edgesandedgesattributes
InArangoDBpriorto3.0theattributes_fromand_toofedgesweretreatedspeciallywhenloadingorstoringedges.Thatspecialhandlingledtotheseattributesbeingnotasflexibleasregulardocumentattributes.Forexample,the_fromand_toattributevaluesofanexistingedgecouldnotbeupdatedoncetheedgewascreated.Additionally,the_fromand_toattributescouldnotbeindexedinuser-definedindexes,e.g.tomakeeachcombinationof_fromand_tounique.Finally,as_fromand_toreferencedthelinkedcollectionsbycollectionidandnotbycollectionname,theirmeaningbecameunclearonceareferencedcollectionwasdropped.Thecollectionidstoredinedgesthenbecameunusable,andwhenaccessingsuchedgethecollectionnamepartofitwasalwaystranslatedto_undefined.
InArangoDB3.0,the_fromand_tovaluesofedgesaresavedasregularstrings.Thisallowsusing_fromand_toinuser-definedindexes.Additionallythisallowsupdatingthe_fromand_tovaluesofexistingedges.Furthermore,collectionsreferencedby_fromand_tovaluesmaybedroppedandre-createdlater.Any_fromand_tovaluesofedgespointingtosuchdroppedcollectionareunaffectedbythedropoperationnow.Alsonotethatrenamingthecollectionreferencedin_fromand_toinArangoDB2.8alsorelinkedtheedges.In3.0theedgesareNOTautomaticallyrelinkedtothenewcollectionanymore.
DocumentsDocuments(incontrasttoedges)cannotcontaintheattributes_fromor_toonthemainlevelinArangoDB3.0.Theseattributeswillbeautomaticallyremovedwhensavingdocuments(i.e.non-edges)._fromand_tocanbestillusedinsub-objectsinsidedocuments.
The_fromand_toattributeswillofcoursebepreservedandarestillrequiredwhensavingedges.
AQL
Edgeshandling
WhenupdatingorreplacingedgesviaAQL,anymodificationstothe_fromand_toattributesofedgeswereignoredbypreviousversionsofArangoDB,withoutsignalinganyerrors.Thiswasduetothe_fromand_toattributesbeingimmutableinearlierversionsofArangoDB.
From3.0on,the_fromand_toattributesofedgesaremutable,soanyAQLqueriesthatmodifythe_fromor_toattributevaluesofedgeswillattempttoactuallychangetheseattributes.Clientsshouldbeawareofthischangeandshouldreviewtheirqueriesthatmodifyedgestoruleoutunintendedside-effects.
Additionally,whencompletelyreplacingthedataofexistingedgesviatheAQLREPLACEoperation,itisnowrequiredtospecifyvaluesforthe_fromand_toattributes,asREPLACErequirestheentirenewdocumenttobespecified.Ifeither_fromor_toaremissingfromthereplacementdocument,anREPLACEoperationwillfail.
Graphfunctions
Inversion3.0allformergraphrelatedfunctionshavebeenremovedfromAQLtobereplacedbynativeAQLconstructs.Theseconstructsallowformorefine-grainedfilteringonseveralgraphlevels.AlsothisallowstheAQLoptimizertoautomaticallyimprovethesequeriesbyenhancingthemwithappropriateindexes.Wehavecreatedrecipestoupgradefrom2.8to3.0whenusingthesefunctions.
Thefunctions:
GRAPH_COMMON_NEIGHBORSGRAPH_COMMON_PROPERTIESGRAPH_DISTANCE_TOGRAPH_EDGESGRAPH_NEIGHBORSGRAPH_TRAVERSALGRAPH_TRAVERSAL_TREEGRAPH_SHORTEST_PATHGRAPH_PATHSGRAPH_VERTICES
Incompatiblechangesin3.0
560
arecoveredinMigratingGRAPH_*Functionsfrom2.8orearlierto3.0
GRAPH_ABSOLUTE_BETWEENNESSGRAPH_ABSOLUTE_CLOSENESSGRAPH_ABSOLUTE_ECCENTRICITYGRAPH_BETWEENNESSGRAPH_CLOSENESSGRAPH_DIAMETERGRAPH_ECCENTRICITYGRAPH_RADIUS
arecoveredinMigratingGRAPH_*Measurementsfrom2.8orearlierto3.0
EDGESNEIGHBORSPATHSTRAVERSALTRAVERSAL_TREE
arecoveredinMigratinganonymousgraphfunctionsfrom2.8orearlierto3.0
Typecastingfunctions
ThetypecastingappliedbytheTO_NUMBER()AQLfunctionhaschangedasfollows:
stringvaluesthatdonotcontainavalidnumericvaluearenowconvertedtothenumber0.InpreviousversionsofArangoDBsuchstringvalueswereconvertedtothevaluenull.arrayvalueswithmorethan1memberarenowconvertedtothenumber0.InpreviousversionsofArangoDBsucharrayswereconvertedtothevaluenull.objects/documentsarenowconvertedtothenumber0.InpreviousversionsofArangoDBobjects/documentswereconvertedtothevaluenull.
Additionally,theTO_STRING()AQLfunctionnowconvertsnullvaluesintoanemptystring("")insteadofthestring"null",whichismoreinlinewithLENGTH(null)returning0andnot4sincev2.6.
TheoutputofTO_STRING()hasalsochangedforarraysandobjectsasfollows:
arraysarenowconvertedintotheirJSON-stringifyequivalents,e.g.
[]isnowconvertedto[][1,2,3]isnowconvertedto[1,2,3]["test",1,2]isnowconvertedto["test",1,2]`
PreviousversionsofArangoDBconvertedarrayswithnomembersintotheemptystring,andnon-emptyarraysintoacomma-separatedlistofmembervalues,withoutthesurroundingangularbrackets.Additionally,stringarraymemberswerenotenclosedinquotesintheresultstring:
[]wasconvertedto``[1,2,3]wasconvertedto1,2,3["test",1,2]wasconvertedtotest,1,2`
objectsarenowconvertedtotheirJSON-stringifyequivalents,e.g.
{}isconvertedto{}{a:1,b:2}isconvertedto{"a":1,"b":2}{"test":"foobar"}isconvertedto{"test":"foobar"}
PreviousversionsofArangoDBalwaysconvertedobjectsintothestring[objectObject]
ThischangealsoaffectsotherpartsinAQLthatusedTO_STRING()toimplicitlycastoperandstostrings.ItalsoaffectstheAQLfunctionsCONCAT()andCONCAT_SEPARATOR()whichtreatedarrayvaluesdifferently.PreviousversionsofArangoDBautomaticallyflattenedarrayvaluesinthefirstlevelofthearray,e.g.CONCAT([1,2,3,[4,5,6]])produced1,2,3,4,5,6.Nowthiswillproduce[1,2,3,[4,5,6]].Toflattenarraymembersonthetoplevel,youcannowusethemoreexplicitCONCAT(FLATTEN([1,2,3,[4,5,6]],1)).
Incompatiblechangesin3.0
561
Arithmeticoperators
AsthearithmeticoperationsinAQLimplicitlyconverttheiroperandstonumericvaluesusingTO_NUMBER(),theircastingbehaviorhasalsochangedasdescribedabove.
Someexamplesofthechangedbehavior:
"foo"+1produces1now.Inpreviousversionsthisproducednull.[1,2]+1produces1.Inpreviousversionsthisproducednull.1+"foo"+1´produces2now.Inpreviousversionthisproduced1`.
Attributenamesandparameters
PreviousversionsofArangoDBhadsometroublewithattributenamesthatcontainedthedotsymbol(.).SomecodepartsinAQLusedthedotsymboltosplitanattributenameintosub-components,soanattributenameda.bwasnotcompletelydistinguishablefromanattributeawithasub-attributeb.Thisinconsistentbehaviorsometimesallowed"hacks"toworksuchaspassingsub-attributesinabindparameterasfollows:
FORdocINcollection
FILTERdoc.@name==1
RETURNdoc
Ifthebindparameter@namecontainedthedotsymbol(e.g.@bind=a.b,itwasunclearwhetherthisshouldtriggersub-attributeaccess(i.e.doc.a.b)oraaccesstoanattributewithexactlythespecifiedname(i.e.doc["a.b"]).
ArangoDB3.0nowhandlesattributenamescontainingthedotsymbolproperly,andsendingabindparameter@name=a.bwillnowalwaystriggeranaccesstotheattributedoc["a.b"],notthesub-attributebofaindoc.
Forusersthatusedthe"hack"ofpassingbindparameterscontainingdotsymboltoaccesssub-attributes,ArangoDB3.0allowsspecifyingtheattributenamepartsasanarrayofstrings,e.g.@name=["a","b"],whichwillberesolvedtothesub-attributeaccessdoc.a.bwhenthequeryisexecuted.
Keywords
LIKEisnowakeywordinAQL.UsingLIKEineithercaseasanattributeorcollectionnameinAQLqueriesnowrequiresquoting.
SHORTEST_PATHisnowakeywordinAQL.UsingSHORTEST_PATHineithercaseasanattributeorcollectionnameinAQLqueriesnowrequiresquoting.
Subqueries
Queriesthatcontainsubqueriesthatcontaindata-modificationoperationssuchasINSERT,UPDATE,REPLACE,UPSERTorREMOVEwillnowrefusetoexecuteifthecollectionaffectedbythesubquery'sdata-modificationoperationisread-accessedinanouterscopeofthequery.
Forexample,thefollowingquerywillrefusetoexecuteasthecollectionmyCollectionismodifiedinthesubquerybutalsoread-accessedintheouterscope:
FORdocINmyCollection
LETchanges=(
FORwhatINmyCollection
FILTERwhat.value==1
REMOVEwhatINmyCollection
)
RETURNdoc
Itisstillpossibletowritetocollectionsfromwhichdataisreadinthesamequery,e.g.
FORdocINmyCollection
FILTERdoc.value==1
REMOVEdocINmyCollection
Incompatiblechangesin3.0
562
andtomodifydataindifferentcollectionviasubqueries.
Otherchanges
TheAQLoptimizerrule"merge-traversal-filter"thatalreadyexistedin3.0wasrenamedto"optimize-traversals".ThisshouldbeofnorelevancetoclientapplicationsexceptiftheyprogramaticallylookforappliedoptimizerrulesintheexplainoutofAQLqueries.
TheorderofresultscreatedbytheAQLfunctionsVALUES()andATTRIBUTES()wasneverguaranteedanditonlyhadthe"correct"orderingbyaccidentwheniteratingoverobjectsthatwerenotloadedfromthedatabase.Assomeofthefunctioninternalshavechanged,the"correct"orderingwillnotappearanymore,andstillnoresultorderisguaranteedbythesefunctionsunlessthesortparameterisspecified(fortheATTRIBUTES()function).
UpgradedV8version
TheV8enginethatisusedinsideArangoDBtoexecuteJavaScriptcodehasbeenupgradedfromversion4.3.61to5.0.71.39.Thenewversionshouldbemostlycompatibletotheoldversion,buttheremaybesubtledifferences,includingchangesoferrormessagetextsthrownbytheengine.Furthermore,someV8startupparametershavechangedtheirmeaningorhavebeenremovedinthenewversion.ThisisonlyrelevantwhenArangoDBorArangoShellarestartedwithacustomvalueforthe--javascript.v8-optionsstartupoption.
Amongothers,thefollowingV8optionschangeinthenewversionofArangoDB:
--es_staging:in2.8ithadthemeaningenableallcompletedharmonyfeatures,in3.0theoptionmeansenabletest-worthyharmonyfeatures(forinternaluseonly)
--strong_this:thisoptionwasn'tpresentin2.8.In3.0itmeansdon'tallow'this'toescapefromconstructorsanddefaultstotrue.
--harmony_regexps:thisoptionsmeansenable"harmonyregularexpressionextensions"andchangesitsdefaultvaluefromfalsetotrue
--harmony_proxies:thisoptionsmeansenable"harmonyproxies"andchangesitsdefaultvaluefromfalsetotrue
--harmony_reflect:thisoptionsmeansenable"harmonyReflectAPI"andchangesitsdefaultvaluefromfalsetotrue
--harmony_sloppy:thisoptionsmeansenable"harmonyfeaturesinsloppymode"andchangesitsdefaultvaluefromfalsetotrue
--harmony_tostring:thisoptionsmeansenable"harmonytoString"andchangesitsdefaultvaluefromfalsetotrue
--harmony_unicode_regexps:thisoptionsmeansenable"harmonyunicoderegexps"andchangesitsdefaultvaluefromfalsetotrue
--harmony_arrays,--harmony_array_includes,--harmony_computed_property_names,--harmony_arrow_functions,--harmony_rest_parameters,--harmony_classes,--harmony_object_literals,--harmony_numeric_literals,--harmony_unicode:theseoptionhavebeenremovedinV85.
AsaconsequenceoftheupgradetoV8version5,theimplementationoftheJavaScriptBufferobjecthadtobechanged.JavaScriptBufferobjectsinArangoDBnowalwaysstoretheirdataontheheap.ThereisnosharedpoolforsmallBuffervalues,andnopointingintoexistingBufferdatawhenextractingslices.ThischangemayincreasethecostofcreatingBufferswithshortcontentsorwhenpeekingintoexistingBuffers,butwasrequiredforsafermemorymanagementandtopreventleaks.
JavaScriptAPIchanges
ThefollowingincompatiblechangeshavebeenmadetotheJavaScriptAPIinArangoDB3.0:
Foxx
TheFoxxframeworkhasbeencompletelyrewrittenfor3.0withanew,simplerandmorefamiliarAPI.TomakeFoxxservicesdevelopedfor2.8orearlierArangoDBversionsrunin3.0,theservice'smanifestfileneedstobeedited.
ToenablethelegacymodeforaFoxxservice,add"engines":{"arangodb":"^2.8.0"}(orsimilarversionrangesthatexclude3.0andup)totheservicemanifestfile(named"manifest.json",locatedintheservice'sbasedirectory).
Incompatiblechangesin3.0
563
Require
ModulesshippedwithArangoDBcannowberequiredusingthepattern@arangodb/<module>insteadoforg/arangodb/<module>,e.g.
varcluster=require("@arangodb/cluster");
Theoldformatcanstillbeusedforcompatibility:
varcluster=require("org/arangodb/cluster");
ArangoDBpriortoversion3.0allowedatransparentuseofCoffeeScriptsourcefileswiththerequire()function.FileswithafilenameextensionofcoffeewereautomaticallysentthroughaCoffeeScriptparserandtranspiledintoJavaScripton-the-fly.ThissupportisgonewithArangoDB3.0.TorunanyCoffeeScriptsourcefiles,theymustbeconvertedtoJavaScriptbytheclientapplication.
Responseobject
The@arangodb/requestresponseobjectnowstorestheparsedJSONresponsebodyinapropertyjsoninsteadofbodywhentherequestwasmadeusingthejsonoption.Thebodyinsteadcontainstheresponsebodyasastring.
EdgesAPI
Whencompletelyreplacinganedgeviaacollection'sreplace()functionthereplacingedgedatanowneedstocontainthe_fromand_toattributesforthenewedge.PreviousversionsofArangoDBdidnotrequiretheedgedatatocontain_fromand_toattributeswhenreplacinganedge,since_fromand_tovalueswereimmutableforexistingedges.
Forexample,thefollowingcallworkedinArangoDB2.8butwillfailin3.0:
db.edgeCollection.replace("myKey",{value:"test"});
TomakethisworkinArangoDB3.0,_fromand_toneedtobeaddedtothereplacementdata:
db.edgeCollection.replace("myKey",{_from:"myVertexCollection/1",_to:"myVertexCollection/2",value:"test"});
Notethatthisonlyaffectsthereplace()functionbutnotupdate(),whichwillonlyupdatethespecifiedattributesoftheedgeandleaveallothersintact.
Additionally,thefunctionsedges(),outEdges()andinEdges()withanarrayofedgeidswillnowmaketheedgeidsuniquebeforereturningtheconnectededges.Thisisprobablydesiredanyway,asresultswillbereturnedonlyonceperdistinctinputedgeid.However,itmaybreakclientapplicationsthatrelyontheoldbehavior.
DatabasesAPI
The_listDatabases()functionofthedbobjecthasbeenrenamedto_databases(),makingitconsistentwiththe_collections()function.Alsothe_listEndpoints()functionhasbeenrenamedto_endpoints().
CollectionAPI
Examplematching
ThecollectionfunctionbyExampleHash()andbyExampleSkiplist()havebeenremovedin3.0.Theirfunctionalityisprovidedbycollection'sbyExample()function,whichwillautomaticallyuseasuitableindexifpresent.
ThecollectionfunctionbyConditionSkiplist()hasbeenremovedin3.0.ThesamefunctionalitycanbeachievedbyissuinganAQLquerywiththetargetcondition,whichwillautomaticallyuseasuitableindexifpresent.
Revisionidhandling
Incompatiblechangesin3.0
564
Theexists()methodofacollectionnowthrowsanexceptionwhenthespecifieddocumentexistsbutitsrevisioniddoesnotmatchtherevisionidspecified.PreviousversionsofArangoDBsimplyreturnedfalseifeithernodocumentexistedwiththespecifiedkeyorwhentherevisioniddidnotmatch.Itwasthereforeimpossibletodistinguishthesetwocasesfromthereturnvaluealone.3.0correctsthis.Additionally,exists()inpreviousversionsalwaysreturnedabooleanifonlythedocumentkeywasgiven.3.0nowreturnsthedocument'smeta-data,whichincludesthedocument'scurrentrevisionid.
GiventhereisadocumentwithkeytestincollectionmyCollection,thenthebehaviorof3.0isasfollows:
/*testifdocumentexists.thisreturnedtruein2.8*/
db.myCollection.exists("test");
{
"_key":"test",
"_id":"myCollection/test",
"_rev":"9758059"
}
/*testifdocumentexists.thisreturnedtruein2.8*/
db.myCollection.exists({_key:"test"});
{
"_key":"test",
"_id":"myCollection/test",
"_rev":"9758059"
}
/*testifdocumentexists.thisalsoreturnedfalsein2.8*/
db.myCollection.exists("foo");
false
/*testifdocumentwithagivenrevisionidexists.thisreturnedtruein2.8*/
db.myCollection.exists({_key:"test",_rev:"9758059"});
{
"_key":"test",
"_id":"myCollection/test",
"_rev":"9758059"
}
/*testifdocumentwithagivenrevisionidexists.thisreturnedfalsein2.8*/
db.myCollection.exists({_key:"test",_rev:"1234"});
JavaScriptexception:ArangoError1200:conflict
Capconstraints
Thecapconstraintsfeaturehasbeenremoved.Thischangehasledtotheremovalofthecollectionoperationsfirst()andlast(),whichwereinternallybasedondatafromcapconstraints.
AscapconstraintshavebeenremovedinArangoDB3.0itisnotpossibletocreateanindexoftype"cap"withacollection'sensureIndex()function.ThededicatedfunctionensureCapConstraint()hasalsobeenremovedfromthecollectionAPI.
GraphBlueprintsJSModule
Thedeprecatedmodulegraph-blueprintshasbeendeleted.Allit'sfeaturesarecoveredbythegeneral-graphmodule.
GeneralGraphFluentAQLinterface
ThefluentinterfacehasbeenremovedfromArangoDB.It'sfeatureswerecompletelyoverlappingwith"aqb"whichcomespreinstalledaswell.PleaseswitchtoAQBinstead.
UndocumentedAPIs
TheundocumentedfunctionsBY_EXAMPLE_HASH()andBY_EXAMPLE_SKIPLIST(),BY_CONDITION_SKIPLIST,CPP_NEIGHBORSandCPP_SHORTEST_PATHhavebeenremoved.ThesefunctionswerealwayshiddenandnotintendedtobepartofthepublicJavaScriptAPIforcollections.
HTTPAPIchanges
Incompatiblechangesin3.0
565
CRUDoperations
ThefollowingincompatiblechangeshavebeenmadetotheHTTPAPIinArangoDB3.0:
General
TheHTTPinsertoperationsforsingledocumentsandedges(POST/_api/document)donotsupporttheURLparameter"createCollection"anymore.InpreviousversionsofArangoDBthisparametercouldbeusedtoautomaticallycreateacollectionuponinsertionofthefirstdocument.ItisnowrequiredthatthetargetcollectionalreadyexistswhenusingthisAPI,otherwiseitwillreturnanHTTP404error.ThesameistruefortheimportAPIatPOST/_api/import.
CollectionscanstillbecreatedeasilyviaaseparatecalltoPOST/_api/collectionasbefore.
The"location"HTTPheaderreturnedbyArangoDBwheninsertinganewdocumentoredgenowalwayscontainsthedatabasename.ThiswasalsothedefaultbehaviorinpreviousversionsofArangoDB,butitcouldbeoverriddenbyclientssendingtheHTTPheaderx-arango-version:1.4intherequest.ClientscancontinuetosendthisheadertoArangoDB3.0,buttheheaderwillnotinfluencethelocationresponseheadersproducedbyArangoDB3.0anymore.
AdditionallytheCRUDoperationsAPIsdonotreturnanattribute"error"intheresponsebodywithanattributevalueof"false"incaseanoperationsucceeded.
Revisionidhandling
Theoperationsforupdating,replacingandremovingdocumentscanoptionallychecktherevisionnumberofthedocumenttobeupdated,replacedorremovedsothecallercanensuretheoperationworksonaspecificversionofthedocumentandtherearenolostupdates.
PreviousversionsofArangoDBallowedpassingtherevisionidofthepreviousdocumenteitherintheHTTPheaderIf-MatchorintheURLparameterrev.Forexample,removingadocumentwithaspecificrevisionidcouldbeachievedasfollows:
curl-XDELETE\
"http://127.0.0.1:8529/_api/document/myCollection/myKey?rev=123"
ArangoDB3.0doesnotsupportpassingtherevisionidviathe"rev"URLparameteranymore.InsteadthepreviousrevisionidmustbepassedintheHTTPheaderIf-Match,e.g.
curl-XDELETE\
--header"If-Match:'123'"\
"http://127.0.0.1:8529/_api/document/myCollection/myKey"
TheURLparameter"policy"wasalsousableinpreviousversionsofArangoDBtocontrolrevisionhandling.Usingitwasredundanttospecifyingtheexpectedrevisionidviathe"rev"parameteror"If-Match"HTTPheaderandthereforesupportforthe"policy"parameterwasremovedin3.0.
Inordertocheckforapreviousrevisionidwhenupdating,replacingorremovingdocumentspleaseusetheIf-MatchHTTPheaderasdescribedabove.WhennorevisioncheckifrequiredtheHTTPheadercanbeomitted,andtheoperationswillworkonthecurrentrevisionofthedocument,regardlessofitsrevisionid.
AlldocumentsAPI
TheHTTPAPIforretrievingtheids,keysorURLsofalldocumentsfromacollectionwaspreviouslylocatedatGET/_api/document?collection=....ThisAPIwasmovedtoPUT/_api/simple/all-keysandisnowexecutedasanAQLquery.ThenameofthecollectionmustnowbepassedintheHTTPrequestbodyinsteadofintherequestURL.Thesameistrueforthe"type"parameter,whichcontrolsthetypeoftheresulttobecreated.
CallstothepreviousAPIcanbetranslatedasfollows:
old:GET/_api/document?collection=<collection>&type=<type>withoutHTTPrequestbody3.0:PUT/_api/simple/all-keyswithHTTPrequestbody{"collection":"<collection>","type":"id"}
TheresultformatofthisAPIhasalsochangedslightly.InpreviousversionscallstotheAPIreturnedaJSONobjectwithadocumentsattribute.AsthefunctionalityisbasedonAQLinternallyin3.0,theAPInowreturnsaJSONobjectwitharesultattribute.
Incompatiblechangesin3.0
566
EdgesAPI
CRUDoperations
TheAPIfordocumentsandedgeshavebeenunifiedinArangoDB3.0.TheCRUDoperationsfordocumentsandedgesarenowhandledbythesameendpointat/_api/document.ForCRUDoperationsthereisnodistinctionanymorebetweendocumentsandedgesAPI-wise.
ThatmeansCRUDoperationsconcerningedgesneedtobesenttotheHTTPendpoint/_api/documentinsteadof/_api/edge.Sendingrequeststo/_api/edgewillresultinanHTTP404errorin3.0.Thefollowingmethodsareavailableat/_api/documentfordocumentsandedge:
HTTPPOST:insertnewdocumentoredgeHTTPGET:fetchanexistingdocumentoredgeHTTPPUT:replaceanexistingdocumentoredgeHTTPPATCH:partiallyupdateanexistingdocumentoredgeHTTPDELETE:removeanexistingdocumentoredge
WhencompletelyreplacinganedgeviaHTTPPUTpleasenotethatthereplacingedgedatanowneedstocontainthe_fromand_toattributesfortheedge.PreviousversionsofArangoDBdidnotrequiresending_fromand_towhenreplacingedges,as_fromand_tovalueswereimmutableforexistingedges.
The_fromand_toattributesofedgesnowalsoneedtobepresentinsidetheedgesobjectssenttotheserver:
curl-XPOST\
--data'{"value":1,"_from":"myVertexCollection/1","_to":"myVertexCollection/2"}'\
"http://127.0.0.1:8529/_api/document?collection=myEdgeCollection"
PreviousversionsofArangoDBrequiredthe_fromand_toattributesofedgesbesentseparatelyinURLparameterfromandto:
curl-XPOST\
--data'{"value":1}'\
"http://127.0.0.1:8529/_api/edge?collection=e&from=myVertexCollection/1&to=myVertexCollection/2"
Queryingconnectededges
TheRESTAPIforqueryingconnectededgesatGET/_api/edges/<collection>willnowmaketheedgeidsuniquebeforereturningtheconnectededges.Thisisprobablydesiredanywayasresultswillnowbereturnedonlyonceperdistinctinputedgeid.However,itmaybreakclientapplicationsthatrelyontheoldbehavior.
GraphAPI
Somedata-modificationoperationsinthenamedgraphsAPIat/_api/gharialnowreturneitherHTTP202(Accepted)orHTTP201(Created)iftheoperationsucceeds.WhichstatuscodeisreturneddependsonthewaitForSyncattributeoftheaffectedcollection.InpreviousversionssomeoftheseoperationsreturnHTTP200regardlessofthewaitForSyncvalue.
ThedeprecatedgraphAPI/_api/graphhasbeenremoved.Allit'sfeaturescanbereplacedusing/_api/gharialandAQLinstead.
SimplequeriesAPI
TheRESTroutesPUT/_api/simple/firstand/_api/simple/lasthavebeenremovedentirely.TheseAPIswereresponsibleforreturningthefirst-insertedandlast-inserteddocumentsinacollection.Thisfeaturewasbuiltoncapconstraintsinternally,whichhavebeenremovedin3.0.
Callingoneoftheseendpointsin3.0willresultinanHTTP404error.
IndexesAPI
Incompatiblechangesin3.0
567
Itisnotsupportedin3.0tocreateanindexwithtypecap(capconstraint)in3.0asthecapconstraintsfeaturehasbeeremoved.CallingtheindexcreationendpointHTTPAPIPOST/_api/index?collection=...withanindextypecapwillthereforeresultinanHTTP400error.
LogentriesAPI
TheRESTrouteHTTPGET/_admin/logisnowaccessiblefromwithinalldatabases.InpreviousversionsofArangoDB,thisroutewasaccessiblefromwithinthe_systemdatabaseonly,andanHTTP403(Forbidden)wasthrownbytheserverforanyaccessfromwithinanotherdatabase.
FiguresAPI
TheRESTrouteHTTPGET/_api/collection/<collection>/figureswillnotreturnthefollowingresultattributesastheybecamemeaninglessin3.0:
shapefiles.countshapes.fileSizeshapes.countshapes.sizeattributes.countattributes.size
DatabasesandCollectionsAPIs
WhencreatingadatabaseviatheAPIPOST/_api/database,ArangoDBwillnowalwaysreturntheHTTPstatuscode202(created)iftheoperationsucceeds.PreviousversionsofArangoDBreturnedHTTP202aswell,butthisbehaviorwaschangablebysendinganHTTPheaderx-arango-version:1.4.Whensendingthisheader,previousversionsofArangoDBreturnedanHTTPstatuscode200(ok).ClientscanstillsendthisheadertoArangoDB3.0butthiswillnotinfluencetheHTTPstatuscodeproducedbyArangoDB.
The"location"headerproducedbyArangoDB3.0willnowalwayscontainthedatabasename.ThiswasalsothedefaultinpreviousversionsofArangoDB,butthebehaviorcouldbeoverriddenbysendingtheHTTPheaderx-arango-version:1.4.Clientscanstillsendtheheader,butthiswillnotmakethedatabasenameinthe"location"responseheaderdisappear.
TheresultformatforqueryingallcollectionsviatheAPIGET/_api/collectionhasbeenchanged.
PreviousversionsofArangoDBreturnedanobjectwithanattributenamedcollectionsandanattributenamednames.Bothcontainedallavailablecollections,butcollectionscontainedthecollectionsasanarray,andnamescontainedthecollectionsagain,containedinanobjectinwhichtheattributenameswerethecollectionnames,e.g.
{
"collections":[
{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},
{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},
...
],
"names":{
"test":{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},
"something":{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},
...
}
}
Thisresultstructurewasredundant,andthereforehasbeensimplifiedtojust
{
"result":[
{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},
{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},
...
]
}
inArangoDB3.0.
Incompatiblechangesin3.0
568
ReplicationAPIs
TheURLparameter"failOnUnknown"wasremovedfromtheRESTAPIGET/_api/replication/dump.Thisparametercontrolledwhetherdumpingorreplicatingedgesshouldfailifoneofthevertexcollectionslinkedintheedge's_fromor_toattributeswasnotpresentanymore.Inthiscasethe_fromand_tovaluescouldnotbetranslatedintomeaningfulidsanymore.
Thereweretwowaysforhandlingthis:
settingfailOnUnknowntotruecausedtheHTTPrequesttofail,leavingerrorhandlingtotheusersettingfailOnUnknowntofalsecausedtheHTTPrequesttocontinue,translatingthecollectionnamepartinthe_fromor_tovalueto_unknown.
InArangoDB3.0thisparameterisobsolete,as_fromand_toarestoredasself-containedstringvaluesallthetime,sotheycannotgetinvalidwhenreferencedcollectionsaredropped.
TheresultformatoftheAPIGET/_api/replication/logger-followhaschangedslightlyinthefollowingaspects:
documentsandedgesarereportedinthesameway.Thetypefordocumentinsertions/updatesandedgeinsertions/updatesisnowalways2300.PreviousversionsofArangoDBreturnedatypevalueof2300fordocumentsand2301foredges.recordsaboutinsertions,updatesorremovalsofdocumentsandedgesdonothavethekeyandrevattributesonthetop-levelanymore.Instead,keyandrevcanbeaccessedbypeekingintothe_keyand_revattributesofthedatasub-attributesofthechangerecord.
Thesameistrueforthecollection-specificchangesAPIGET/_api/replication/dump.
UsermanagementAPIs
TheRESTAPIendpointPOST/_api/userforaddingnewusersnowrequirestherequesttocontainaJSONobjectwithanattributenameduser,containingthenameoftheusertobecreated.PreviousversionsofArangoDBalsocheckedthisattribute,butadditionallylookedforanattributeusernameiftheuserattributedidnotexist.
UndocumentedAPIs
ThefollowingundocumentedHTTPRESTendpointshavebeenremovedfromArangoDB'sRESTAPI:
/_open/cerberusand/_system/cerberus:theseendpointswereintendedforsomeArangoDB-internalapplicationsonlyPUT/_api/simple/by-example-hash,PUT/_api/simple/by-example-skiplistandPUT/_api/simple/by-condition-skiplist:thesemethodsweredocumentedinearlyversionsofArangoDBbuthavebeenmarkedasnotintendedtobecalledbyenduserssinceArangoDBversion2.3.ThesemethodsshouldnothavebeenpartofanyArangoDBmanualsinceversion2.4./_api/structure:anolderunfinishedandunpromotedAPIfordataformatandtypechecks,supersededbyFoxxapplications.
AdministrationAPIs
/_admin/shutdownnowneedstobecalledwiththeHTTPDELETEmethod
HandlingofCORSrequests
ItcannowbecontrolledindetailforwhichoriginhostsCORS(Cross-originresourcesharing)requestswithcredentialswillbeallowed.ArangoDB3.0providesthestartupoption--http.trusted-originthatcanbeusedtospecifyoneormanyoriginsfromwhichCORSrequestsaretreatedas"trustworthy".
Theoptioncanbespecifiedmultipletimes,oncepertrustedorigin,e.g.
--http.trusted-originhttp://127.0.0.1:8529--http.trusted-originhttps://127.0.0.1:8599
ThiswillmaketheArangoDBserverrespondtoCORSrequestsfromtheseoriginswithanAccess-Control-Allow-CredentialsHTTPheaderwithavalueoftrue.WebbrowserscaninspectthisheaderandcanallowpassingArangoDBwebinterfacecredentials(ifstoredinthebrowser)totherequestingsite.ArangoDBwillnotforwardorprovideanycredentials.
Incompatiblechangesin3.0
569
SettingthisoptionisonlyrequiredifapplicationsonotherhostsneedtoaccesstheArangoDBwebinterfaceorotherHTTPRESTAPIsfromawebbrowserwiththesamecredentialsthattheuserhasenteredwhenloggingintothewebinterface.WhenawebbrowserfindstheAccess-Control-Allow-CredentialsHTTPresponseheader,itmayforwardthecredentialsenteredintothebrowserfortheArangoDBwebinterfacelogintotheothersite.
Thisisapotentialsecurityissue,sotherearenotrustedoriginsbydefault.Itmayberequiredtosetsometrustedoriginsifyou'replanningtoissueAJAXrequeststoArangoDBfromothersitesfromthebrowser,withthecredentialsenteredduringtheArangoDBinterfacelogin(i.e.singlesign-on).Ifsuchfunctionalityisnotused,theoptionshouldnotbeset.
Tospecifyatrustedorigin,specifytheoptiononcepertrustedoriginasshownabove.NotethatthetrustedoriginvaluesspecifiedinthisoptionwillbecomparedbytewisewiththeOriginHTTPheadervaluesentbyclients,andonlyexactmatcheswillpass.
ThereisalsothewildcardallforenablingCORSaccessfromalloriginsinatestordevelopmentsetup:
--http.trusted-originall
SettingthisoptionwillleadtotheArangoDBserverrespondingwithanAccess-Control-Allow-Credentials:trueHTTPheadertoallincomingCORSrequests.
Command-lineoptionsQuiteafewstartupoptionsinArangoDB2weredoublenegations(like--server.disable-authenticationfalse).InArangoDB3thesearenowexpressedaspositives(e.g.--server.authentication).AlsotheoptionsbetweentheArangoDBserveranditsclienttoolshavebeingunified.Forexample,theloggeroptionsarenowthesamefortheserverandtheclienttools.Additionallymanyoptionshavebeenmovedintomoreappropriatetopicsections.
Renamedoptions
Thefollowingoptionshavebeenavailablebefore3.0andhavechangedtheirnamein3.0:
--server.disable-authenticationwasrenamedto--server.authentication.Notethatthemeaningoftheoption--server.authenticationistheoppositeoftheprevious--server.disable-authentication.--server.disable-authentication-unix-socketswasrenamedto--server.authentication-unix-sockets.Notethatthemeaningoftheoption--server.authentication-unix-socketsistheoppositeoftheprevious--server.disable-authentication-unix-sockets.--server.authenticate-system-onlywasrenamedto--server.authentication-system-only.Themeaningoftheoptioninunchanged.--server.disable-statisticswasrenamedto--server.statistics.Notethatthemeaningoftheoption--server.statisticsistheoppositeoftheprevious--server.disable-statistics.--server.cafilewasrenamedto--ssl.cafile.Themeaningoftheoptionisunchanged.--server.keyfilewasrenamedto--ssl.keyfile.Themeaningoftheoptionisunchanged.--server.ssl-cachewasrenamedto--ssl.session-cache.Themeaningoftheoptionisunchanged.--server.ssl-cipher-listwasrenamedto--ssl.cipher-list.Themeaningoftheoptionisunchanged.--server.ssl-optionswasrenamedto--ssl.options.Themeaningoftheoptionisunchanged.--server.ssl-protocolwasrenamedto--ssl.protocol.Themeaningoftheoptionisunchanged.--server.backlog-sizewasrenamedto--tcp.backlog-size.Themeaningoftheoptionisunchanged.--server.reuse-addresswasrenamedto--tcp.reuse-address.Themeaningoftheoptionisunchanged.--server.disable-replication-applierwasrenamedto--database.replication-applier.Themeaningoftheoption--database.replication-applieristheoppositeoftheprevious--server.disable-replication-applier.--server.allow-method-overridewasrenamedto--http.allow-method-override.Themeaningoftheoptionisunchanged.--server.hide-product-headerwasrenamedto--http.hide-product-header.Themeaningoftheoptionisunchanged.--server.keep-alive-timeoutwasrenamedto--http.keep-alive-timeout.Themeaningoftheoptionisunchanged.--server.foxx-queueswasrenamedto--foxx.queues.Themeaningoftheoptionisunchanged.--server.foxx-queues-poll-intervalwasrenamedto--foxx.queues-poll-interval.Themeaningoftheoptionisunchanged.--no-serverwasrenamedto--server.rest-server.Notethatthemeaningoftheoption--server.rest-serveristheoppositeoftheprevious--no-server.--database.query-cache-modewasrenamedto--query.cache-mode.Themeaningoftheoptionisunchanged.--database.query-cache-max-resultswasrenamedto--query.cache-entries.Themeaningoftheoptionisunchanged.
Incompatiblechangesin3.0
570
--database.disable-query-trackingwasrenamedto--query.tracking.Themeaningoftheoption--query.trackingistheoppositeoftheprevious--database.disable-query-tracking.--log.ttywasrenamedto--log.foreground-tty.Themeaningoftheoptionisunchanged.--upgradehasbeenrenamedto--database.auto-upgrade.Incontrastto2.8thisoptionnowrequiresabooleanparameter.Toactuallyperformanautomaticdatabaseupgradeatstartupuse--database.auto-upgradetrue.Tonotperformit,use--database.auto-upgradefalse.--check-versionhasbeenrenamedto--database.check-version.--temp-pathhasbeenrenamedto--temp.path.
Logverbosity,topicsandoutputfiles
Loggingnowsupportslogtopics.Youcancontrolthesebyspecifyingalogtopicinfrontofalogleveloranoutput.Forexample
--log.levelstartup=trace--log.levelinfo
willlogmessagesconcerningstartupattracelevel,everythingelseatinfolevel.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.
Somerelevantlogtopicsavailablein3.0are:
collector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperationsperformance:someperformance-relatedinformationqueries:executedAQLqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads
Thenewlogoption--log.output<definition>allowsdirectingtheglobalorper-topiclogoutputtodifferentoutputs.Theoutputdefinition""canbeoneof
"-"forstdin"+"forstderr"syslog://""syslog:///""file://"
Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputfordifferentlogtopics.Tosetupaper-topicoutputconfiguration,use--log.output<topic>=<definition>,e.g.
queries=file://queries.txt
logsallqueriestothefile"queries.txt".
Theoldoption--log.fileisstillavailablein3.0forconveniencereasons.In3.0itisashortcutforthemoregeneraloption--log.outputfile://filename.
Theoldoption--log.requests-fileisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.outputrequests=file://....
Theoldoption--log.performanceisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.levelperformance=trace.
Removedoptionsforlogging
Theoptions--log.content-filterand--log.source-filterhavebeenremoved.TheyhavemostbeenusedduringArangoDB'sinternaldevelopment.
Incompatiblechangesin3.0
571
Thesyslog-relatedoptions--log.applicationand--log.facilityhavebeenremoved.Theyaresupersededbythemoregeneral--log.outputoptionwhichcanalsohandlesyslogtargets.
Removedotheroptions
Theoption--server.default-api-compatibilitywaspresentinearlierversionofArangoDBtocontrolvariousaspectsoftheserverbehavior,e.g.HTTPreturncodesortheformatofHTTP"location"headers.ClientapplicationscouldsendanHTTPheader"x-arango-version"withaversionnumbertorequesttheserverbehaviorofacertainArangoDBversion.
Thisoptionwasonlyhonoredinahandfulofcases(describedabove)andwasremovedin3.0becausethechangesinserverbehaviorcontrolledbythisoptionwerechangedevenbeforeArangoDB2.0.Thisshouldhaveleftenoughtimeforclientapplicationstoadapttothenewbehavior,makingtheoptionsuperfluousin3.0.
Threadoptions
Theoptions--server.threadsand--scheduler.threadsnowhaveadefaultvalueof0.When--server.threadsissetto0onstartup,thesuitablenumberofthreadswillbedeterminedbyArangoDBbyaskingtheOSforthenumberofavailableCPUsandusingthatasabaseline.IfthenumberofCPUsislowerthan4,ArangoDBwillstillstart4dispatcherthreads.When--scheduler.threadsissetto0,thenArangoDBwillautomaticallydeterminethenumberofschedulerthreadstostart.Thiswillnormallycreate2schedulerthreads.
Iftheexactnumberofthreadsneedstobesetbytheadmin,thenitisstillpossibletoset--server.threadsand--scheduler.threadstonon-zerovalues.ArangoDBwillusethesevaluesandstartthatmanythreads(notethatsomethreadsmaybecreatedlazilysotheymaynotbepresentdirectlyafterstartup).
ThenumberofV8JavaScriptcontextstobecreated(--javascript.v8-contexts)nowhasadefaultvalueof0too,meaningthatArangoDBwillcreateasmanyV8contextsastherewillbedispatcherthreads(controlledbythe--server.threadsoption).Settingthisoptiontoanon-zerovaluewillcreateexactlyasmanyV8contextsasspecified.
Settingtheseoptionsexplicitlytonon-zerovaluesmaybebeneficialinenvironmentsthathavefewresources(processingtime,maximumthreadcount,availablememory).
AuthenticationThedefaultvaluefor--server.authenticationisnowtrueintheconfigurationfilesshippedwithArangoDB.Thismeanstheserverwillbestartedwithauthenticationenabledbydefault,requiringallclientconnectionstoprovideauthenticationdatawhenconnectingtoArangoDBAPIs.PreviousArangoDBversionsusedthesetting--server.disable-authenticationtrue,effectivelydisablingauthenticationbydefault.
Thedefaultvaluefor--server.authentication-system-onlyisnowtrueinArangoDB.ThatmeansthatFoxxapplicationsrunninginArangoDBwillbepublicaccessible(atleasttheywillnotuseArangoDB'sbuiltinauthenticationmechanism).OnlyrequeststoArangoDBAPIsatURLpathprefixes/_api/and/_adminwillrequireauthentication.Tochangethat,andusethebuiltinauthenticationmechanismforFoxxapplicationstoo,set--server.authentication-system-onlytofalse,andmakesuretohavetheoption--server.authenticationsettotrueaswell.
Thoughenablingtheauthenticationisrecommendedforproductionsetups,itmaybeoverkillinadevelopmentenvironment.Toturnoffauthentication,theoption--server.authenticationcanbesettofalseinArangoDB'sconfigurationfileoronthecommand-line.
WebAdminInterface
TheJavaScriptshellhasbeenremovedfromArangoDB'swebinterface.ThefunctionalitytheshellprovidedisstillfullyavailableintheArangoShell(arangosh)binaryshippedwithArangoDB.
ArangoShellandclienttools
Incompatiblechangesin3.0
572
TheArangoShell(arangosh)andtheotherclienttoolsbundledwithArangoDBcanonlyconnecttoanArangoDBserverofversion3.0orhigher.TheywillnotconnecttoanArangoDB2.8.ThisisbecausetheserverHTTPAPIshavechangedbetween2.8and3.0,andallclienttoolsusestheseAPIs.
InordertoconnecttoearlierversionsofArangoDBwiththeclienttools,anolderversionoftheclienttoolsneedstobekeptinstalled.
ThepreferrednameforthetemplatestringgeneratorfunctionaqlQueryisnowaqlandisautomaticallyavailableinarangosh.Elsewhere,itcanbeloadedlikeconstaql=require('@arangodb').aql.
Command-lineoptionsadded
Allclienttoolsin3.0provideanoption--server.max-packet-sizeforcontrollingthemaximumsizeofHTTPpacketstobehandledbytheclienttools.Thedefaultvalueis128MB,asinpreviousversionsofArangoDB.Incontrasttopreviousversionsinwhichthevaluewashard-coded,theoptionisnowconfigurable.ItcanbeincreasedtomaketheclienttoolshandleverylargeHTTPresultmessagessentbytheserver.
Command-lineoptionschanged
Forallclienttools,theoption--server.disable-authenticationwasrenamedto--server.authentication.Notethatthemeaningoftheoption--server.authenticationistheoppositeoftheprevious--server.disable-authentication.
Theoption--server.ssl-protocolwasrenamedto--ssl.protocol.Themeaningoftheoptionisunchanged.
Thecommand-lineoption--quietwasremovedfromallclienttoolsexceptarangoshbecauseithadnoeffectinthem.
Arangobench
Inordertomakeitspurposemoreapparenttheformerarangobclienttoolhasbeenrenamedtoarangobenchin3.0.
MiscellaneouschangesThechecksumcalculationalgorithmforthecollection.checksum()methodanditscorrespondingRESTAPIGET/_api/collection/<collection</checksumhaschangedin3.0.Checksumscalculatedin3.0willdifferfromchecksumscalculatedwith2.8orbefore.
TheArangoDBserverin3.0doesnotreadafileENDPOINTScontainingalistofadditionalendpointsonstartup.In2.8thisfilewasautomaticallyreadifpresentinthedatabasedirectory.
Thenamesofthesub-threadsstartedbyArangoDBhavechangedin3.0.ThisisrelevantonLinuxonly,wherethreadscanbenamedandthreadnamesmaybevisibletosystemtoolssuchastopormonitoringsolutions.
Incompatiblechangesin3.0
573
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.8.ArangoDB2.8alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.
AQLimprovements
AQLGraphTraversals/PatternMatching
AQLoffersanewfeaturetotraverseoveragraphwithoutwritingJavaScriptfunctionsbutwithalltheotherfeaturesyouknowfromAQL.Forthispurpose,aspecialversionofFORvariableNameINexpressionhasbeenintroduced.
Thisspecialversionhasthefollowingformat:FORvertex-variable,edge-variable,path-variableINtraversal-expression,wheretraversal-expressionhasthefollowingformat:[depth]directionstart-vertexgraph-definitionwiththefollowinginputparameters:
depth(optional):defineshowmanystepsareexecuted.Thevaluecaneitherbeanintegervalue(e.g.3)orarangeofintegervalues(e.g.1..5).Thedefaultis1.direction:defineswhichedgedirectionsarefollowed.CanbeeitherOUTBOUND,INBOUNDorANY.start-vertex:defineswherethetraversalisstarted.Mustbean_idvalueoradocument.graph-definition:defineswhichedgecollectionsareusedforthetraversal.MustbeeitherGRAPHgraph-nameforgraphscreatedwiththegraph-module,oralistofedgecollectionsedge-col1,edge-col2,..edge-colN.
Thethreeoutputvariableshavethefollowingsemantics:
vertex-variable:Thelastvisitedvertex.edge-variable:Thelastvisitededge(optional).path-variable:Thecompletepathfromstart-vertextovertex-variable(optional).
ThetraversalstatementcanbeusedinthesamewayastheoriginalFORvariableNameINexpression,andcanbecombinedwithfiltersandotherAQLconstructs.
AsanexampleonecannowfindthefriendsofafriendforacertainuserwiththisAQLstatement:
FORfoaf,e,pathIN2ANY@startUserGRAPH"relations"
FILTERpath.edges[0].type=="friend"
FILTERpath.edges[1].type=="friend"
FILTERfoaf._id!=@startUser
RETURNDISTINCTfoaf
Optimizerruleshavebeenimplementedtogainperformanceofthetraversalstatement.Theserulesmovefilterstatementsintothetraversalstatements.t.pathswhichcanneverpassthefilterarenotemittedtothevariables.
Asanexampletakethequeryaboveandassumethereareedgesthatdonothavetype=="friend".Ifinthefirstedgestepthereissuchanon-friendedgethesecondstepswillneverbecomputedfortheseedgesastheycannotfulfillthefiltercondition.
ArrayIndexes
Hashindexesandskiplistindexescannowoptionallybedefinedforarrayvaluessothattheyindexindividualarraymembersinsteadoftheentirearrayvalue.
Todefineanindexforarrayvalues,theattributenameisextendedwiththeexpansionoperator[*]intheindexdefinition.
Example:
db._create("posts");
db.posts.ensureHashIndex("tags[*]");
Whengiventhefollowingdocument
WhatsNewin2.8
574
{
"tags":[
"AQL",
"ArangoDB",
"Index"
]
}
thisindexwillnowcontaintheindividualvalues"AQL","ArangoDB"and"Index".
Nowtheindexcanbeusedforfindingalldocumentshaving"ArangoDB"somewhereintheirtagsarrayusingthefollowingAQLquery:
FORdocINposts
FILTER"ArangoDB"INdoc.tags[*]
RETURNdoc
Itisalsopossibletocreateanindexonsub-attributesofarrayvalues.Thismakessensewhentheindexattributeisanarrayofobjects,e.g.
db._drop("posts");
db._create("posts");
db.posts.ensureIndex({type:"hash",fields:["tags[*].name"]});
db.posts.insert({tags:[{name:"AQL"},{name:"ArangoDB"},{name:"Index"}]});
db.posts.insert({tags:[{name:"AQL"},{name:"2.8"}]});
Thefollowingquerywillthenusethearrayindex:
FORdocINposts
FILTER'AQL'INdoc.tags[*].name
RETURNdoc
Arrayvalueswillautomaticallybede-duplicatedbeforebeinginsertedintoanarrayindex.
PleasenotethatfilteringusingarrayindexesonlyworksfromwithinAQLqueriesandonlyifthequeryfiltersontheindexedattributeusingtheINoperator.Theothercomparisonoperators(==,!=,>,>=,<,<=)currentlydonotusearrayindexes.
Optimizerimprovements
TheAQLqueryoptimizercannowuseindexesifmultiplefilterconditionsonattributesofthesamecollectionarecombinedwithlogicalORs,andiftheusageofindexeswouldcompletelycovertheseconditions.
Forexample,thefollowingqueriescannowusetwoindependentindexesonvalue1andvalue2(thelatterqueryrequiresthattheindexesareskiplistindexesduetousageofthe<and>comparisonoperators):
FORdocINcollectionFILTERdoc.value1==42||doc.value2==23RETURNdoc
FORdocINcollectionFILTERdoc.value1<42||doc.value2>23RETURNdoc
Thenewoptimizerrule"sort-in-values"cannowpre-sorttheright-handsideoperandofINandNOTINoperatorssotheoperationcanuseabinarysearchwithlogarithmiccomplexityinsteadofalinearsearch.Therulewillbeappliedwhentheright-handsideoperandofanINorNOTINoperatorinafilterconditionisavariablethatisdefinedinadifferentloop/scopethantheoperatoritself.Additionally,thefilterconditionmustconsistofsolelytheINorNOTINoperationinordertoavoidanyside-effects.
Therulewillkickinforaqueriessuchasthefollowing:
LETvalues=/*someruntimeexpressionhere*/
FORdocINcollection
FILTERdoc.valueINvalues
RETURNdoc
Itwillnotbeappliedforthefollowigqueries,becausetheright-handsideoperandoftheINiseithernotavariable,orbecausetheFILTERconditionmayhavesideeffects:
WhatsNewin2.8
575
FORdocINcollection
FILTERdoc.valueIN/*someruntimeexpressionhere*/
RETURNdoc
LETvalues=/*someruntimeexpressionhere*/
FORdocINcollection
FILTERFUNCTION(doc.values)==23&&doc.valueINvalues
RETURNdoc
AQLfunctionsadded
ThefollowingAQLfunctionshavebeenaddedin2.8:
POW(base,exponent):returnsthebasetotheexponentexp
UNSET_RECURSIVE(document,attributename,...):recursivelyremovestheattributesattributename(canbeoneormany)fromdocumentanditssub-documents.Allotherattributeswillbepreserved.Multipleattributenamescanbespecifiedbyeitherpassingmultipleindividualstringargumentnames,orbypassinganarrayofattributenames:
UNSET_RECURSIVE(doc,'_id','_key','foo','bar')
UNSET_RECURSIVE(doc,['_id','_key','foo','bar'])
IS_DATESTRING(value):returnstrueifvalueisastringthatcanbeusedinadatefunction.Thisincludespartialdatessuchas2015or2015-10andstringscontaininginvaliddatessuchas2015-02-31.Thefunctionwillreturnfalseforallnon-stringvalues,evenifsomeofthemmaybeusableindatefunctions.
Miscellaneousimprovements
theArangoShellnowprovidestheconveniencefunctiondb._explain(query)forretrievingahuman-readableexplanationofAQLqueries.Thisfunctionisashorthandforrequire("org/arangodb/aql/explainer").explain(query).
theAQLqueryoptimizernowautomaticallyconvertsLENGTH(collection-name)toanoptimizedexpressionthatreturnsthenumberofdocumentsinacollection.PreviousversionsofArangoDBreturnedawarningwhenusingthisexpressionandalsoenumeratedalldocumentsinthecollection,whichwasinefficient.
improvedperformanceofskippingovermanydocumentsinanAQLquerywhennoindexesandnofiltersareused,e.g.
FORdocINcollection
LIMIT1000000,10
RETURNdoc
addedclusterexecutionsiteinfoinexecutionplanexplainoutputforAQLqueries
for30+AQLfunctionsthereisnowanadditionalimplementationinC++thatremovestheneedforinternaldataconversionwhenthefunctioniscalled
theAQLeditorinthewebinterfacenowsupportsusingbindparameters
Deadlockdetection
ArangoDB2.8nowhasanautomaticdeadlockdetectionfortransactions.
Adeadlockisasituationinwhichtwoormoreconcurrentoperations(usertransactionsorAQLqueries)trytoaccessthesameresources(collections,documents)andneedtowaitfortheotherstofinish,butnoneofthemcanmakeanyprogress.
Incaseofsuchadeadlock,therewouldbenoprogressforanyoftheinvolvedtransactions,andnoneoftheinvolvedtransactionscouldevercomplete.Thisiscompletelyundesirable,sothenewautomaticdeadlockdetectionmechanisminArangoDBwillautomaticallykickinandabortoneofthetransactionsinvolvedinsuchadeadlock.Abortingmeansthatallchangesdonebythetransactionwillberolledbackanderror29(deadlockdetected)willbethrown.
WhatsNewin2.8
576
Clientcode(AQLqueries,usertransactions)thataccessesmorethanonecollectionshouldbeawareofthepotentialofdeadlocksandshouldhandletheerror29(deadlockdetected)properly,eitherbypassingtheexceptiontothecallerorretryingtheoperation.
Replication
Thefollowingimprovementsforreplicationhavebeenmadein2.8(note:mostofthemhavebeenbackportedtoArangoDB2.7aswell):
addedautoResyncconfigurationparameterforcontinuousreplication.
Whensettotrue,areplicationslavewillautomaticallytriggerafulldatare-synchronizationwiththemasterwhenthemastercannotprovidethelogdatatheslavehadaskedfor.NotethatautoResyncwillonlyworkwhentheoptionrequireFromPresentisalsosettotrueforthecontinuousreplication,orwhenthecontinuoussyncerisstartedanddetectsthatnostarttickispresent.
Automaticre-synchronizationmaytransferalotofdatafromthemastertotheslaveandmaybeexpensive.Itisthereforeturnedoffbydefault.Whenturnedoff,theslavewillneverperformanautomaticre-synchronizationwiththemaster.
addedidleMinWaitTimeandidleMaxWaitTimeconfigurationparametersforcontinuousreplication.
Theseparameterscanbeusedtocontroltheminimumandmaximumwaittimetheslavewill(intentionally)idleandnotpollformasterlogchangesincasethemasterhadsentthefulllogsalready.TheidleMaxWaitTimevaluewillonlybeusedwhenadapativePollingissettotrue.WhenadaptivePollingisdisabled,onlyidleMinWaitTimewillbeusedasaconstanttimespaninwhichtheslavewillnotpollthemasterforfurtherchanges.Thedefaultvaluesare0.5secondsforidleMinWaitTimeand2.5secondsforidleMaxWaitTime,whichcorrespondtothehard-codedvaluesusedinpreviousversionsofArangoDB.
addedinitialSyncMaxWaitTimeconfigurationparameterforinitialandcontinuousreplication
Thisoptioncontrolsthemaximumwaittime(inseconds)thattheinitialsynchronizationwillwaitforaresponsefromthemasterwhenfetchinginitialcollectiondata.Ifnoresponseisreceivedwithinthistimeperiod,theinitialsynchronizationwillgiveupandfail.ThisoptionisalsorelevantforcontinuousreplicationincaseautoResyncissettotrue,asthenthecontinuousreplicationmaytriggerafulldatare-synchronizationincasethemastercannotthelogdatatheslavehadaskedfor.
HTTPrequestssentfromtheslavetothemasterduringinitialsynchronizationwillnowberetriediftheyfailwithconnectionproblems.
theinitialsynchronizationnowlogsitsprogresssoitcanbequeriedusingtheregularreplicationstatuscheckAPIs.
addedasyncattributeforsyncandsyncCollectionoperationscalledfromtheArangoShell.Setthingthisattributetotruewillmakethesynchronizationjobontheservergointothebackground,sothattheshelldoesnotblock.ThestatusofthestartedasynchronoussynchronizationjobcanbequeriedfromtheArangoShelllikethis:
/*startsinitialsynchronization*/
varreplication=require("org/arangodb/replication");
varid=replication.sync({
endpoint:"tcp://master.domain.org:8529",
username:"myuser",
password:"mypasswd",
async:true
});
/*nowquerytheidofthereturnedasyncjobandprintthestatus*/
print(replication.getSyncResult(id));
TheresultofgetSyncResult()willbefalsewhiletheserver-sidejobhasnotcompleted,anddifferenttofalseifithascompleted.Whenithascompleted,alljobresultdetailswillbereturnedbythecalltogetSyncResult().
thewebadmininterfacedashboardnowshowsaserver'sreplicationstatusatthebottomofthepage
WebAdminInterface
Thefollowingimprovementshavebeenmadeforthewebadmininterface:
theAQLeditornowhassupportforbindparameters.Thebindparametervaluescanbeeditedinthewebinterfaceandsavedwithaqueryforfutureuse.
WhatsNewin2.8
577
theAQLeditornowallowscancelingrunningqueries.Thiscanbeusedtocancellong-runningquerieswithoutswitchingtothequerymanagementsection.
thedashboardnowprovidesinformationabouttheserver'sreplicationstatusatthebottomofthepage.Thiscanbeusedtotrackeitherthestatusofaone-timesynchronizationorthecontinuousreplication.
thecompactionstatusandsomestatusinternalsaboutcollectionsarenowdisplayedinthedetailviewforacollectioninthewebinterface.Thesedatacanbeusedfordebuggingcompactionissues.
unloadingacollectionviathewebinterfacewillnowtriggergarbagecollectioninallv8contextsandforceaWALflush.Thisincreasesthechancesofperfomingtheunloadfaster.
thestatusterminologyforcollectionsforwhichanunloadrequesthasbeenissuedviathewebinterfacewaschangedfromintheprocessofbeingunloadedtowillbeunloaded.Thisismoreaccurateastheactualunloadmaybepostponeduntillateriftherearestillreferencespointingtodatainthecollection.
Foxximprovementsthemoduleresolutionusedbyrequirenowbehavesmorelikeinnode.js
theorg/arangodb/requestmodulenowreturnsresponsebodiesforerrorresponsesbydefault.Theoldbehaviorofnotreturningbodiesforerrorresponsescanbere-enabledbyexplicitlysettingtheoptionreturnBodyOnErrortofalse
Miscellaneouschanges
Thestartupoption--server.hide-product-headercanbeusedtomaketheservernotsendtheHTTPresponseheader"Server:ArangoDB"initsHTTPresponses.ThiscanbeusedtoconcealtheservermakefromHTTPclients.Bydefault,theoptionisturnedoffsotheheaderisstillsentasusual.
arangodumpandarangorestorenowhavebettererrorreporting.Additionally,arangodumpwillnowfailbydefaultwhentryingtodumpedgesthatrefertoalreadydroppedcollections.Thiscanbecircumventedbyspecifyingtheoption--forcetruewheninvokingarangodump.
arangoimpnowprovidesanoption--create-collection-typetospecifythetypeofthecollectiontobecreatedwhen--create-collectionissettotrue.Previously--create-collectionalwayscreateddocumentcollectionsandthecreationofedgecollectionswasnotpossible.
WhatsNewin2.8
578
IncompatiblechangesinArangoDB2.8ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.8,andadjustanyclientprogramsifnecessary.
AQL
Keywordsadded
ThefollowingAQLkeywordswereaddedinArangoDB2.8:
GRAPH
OUTBOUND
INBOUND
ANY
ALL
NONE
AGGREGATE
Usageofthesekeywordsforcollectionnames,variablenamesorattributenamesinAQLquerieswillnotbepossiblewithoutquoting.Forexample,thefollowingAQLquerywillstillworkasitusesaquotedcollectionnameandaquotedattributename:
FORdocIN`OUTBOUND`
RETURNdoc.`any`
Changedbehavior
TheAQLfunctionsNEARandWITHINnowhavestrictervalidationsfortheirinputparameterslimit,radiusanddistance.Theymaynowthrowexceptionswheninvalidparametersarepassedthatmayhavenotledtoexceptionsinpreviousversions.
Additionally,theexpansion([*])operatorinAQLhaschangeditsbehaviorwhenhandlingnon-arrayvalues:
InArangoDB2.8,callingtheexpansionoperatoronanon-arrayvaluewillalwaysreturnanemptyarray.PreviousversionsofArangoDBexpandednon-arrayvaluesbycallingtheTO_ARRAY()functionforthevalue,whichforexamplereturnedanarraywithasinglevalueforboolean,numericandstringinputvalues,andanarraywiththeobject'svaluesforanobjectinputvalue.Thisbehaviorwasinconsistentwithhowtheexpansionoperatorworksforthearrayindexesin2.8,sothebehaviorisnowunified:
iftheleft-handsideoperandof[*]isanarray,thearraywillbereturnedasiswhencalling[*]onitiftheleft-handsideoperandof[*]isnotanarray,anemptyarraywillbereturnedby[*]
AQLqueriesthatrelyontheoldbehaviorcanbechangedbyeithercallingTO_ARRAYexplicitlyorbyusingthe[*]atthecorrectposition.
Thefollowingexamplequerywillchangeitsresultin2.8comparedto2.7:
LETvalues="foo"RETURNvalues[*]
In2.7thequeryhasreturnedthearray["foo"],butin2.8itwillreturnanemptyarray[].Tomakeitreturnthearray["foo"]again,anexplicitTO_ARRAYfunctioncallisneededin2.8(whichinthiscaseallowstheremovalofthe[*]operatoraltogether).Thisalsoworksin2.7:
LETvalues="foo"RETURNTO_ARRAY(values)
Anotherexample:
LETvalues=[{name:"foo"},{name:"bar"}]
RETURNvalues[*].name[*]
Incompatiblechangesin2.8
579
Theabovereturned[["foo"],["bar"]]in2.7.In2.8itwillreturn[[],[]],becausethevalueofname`isnotanarray.Tochangetheresultstothe2.7style,thequerycanbechangedto
LETvalues=[{name:"foo"},{name:"bar"}]
RETURNvalues[*RETURNTO_ARRAY(CURRENT.name)]
Theabovealsoworksin2.7.Thefollowingtypesofquerieswon'tchange:
LETvalues=[1,2,3]RETURNvalues[*]
LETvalues=[{name:"foo"},{name:"bar"}]RETURNvalues[*].name
LETvalues=[{names:["foo","bar"]},{names:["baz"]}]RETURNvalues[*].names[*]
LETvalues=[{names:["foo","bar"]},{names:["baz"]}]RETURNvalues[*].names[**]
Deadlockhandling
Clientapplicationsshouldbepreparedtohandleerror29(deadlockdetected)thatArangoDBmaynowthrowwhenitdetectsadeadlockacrossmultipletransactions.Whenaclientapplicationreceiveserror29,itshouldretrytheoperationthatfailed.
TheerrorcanonlyoccurforAQLqueriesorusertransactionsthatinvolvemorethanasinglecollection.
Optimizer
TheAQLexecutionnodetypeIndexRangeNodewasreplacedwithanewmorecapableexecutionnodetypeIndexNode.ThatmeansinexecutionplanexplainoutputtherewillbenomoreIndexRangeNodesbutonlyIndexNode.Thisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.
TheoptimizerrulethatmakesAQLqueriesactuallyuseindexeswasalsorenamedfromuse-index-rangetouse-indexes.Againthisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.
Thequeryoptimizerruleremove-collect-intowasrenamedtoremove-collect-variables.Thisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.
HTTPAPI
Whenaserver-sideoperationgotcanceledduetoanexplicitclientcancelrequestviaHTTPDELETE/_api/job,previousversionsofArangoDBreturnedanHTTPstatuscodeof408(requesttimeout)fortheresponseofthecanceledoperation.
TheHTTPreturncode408hascausedproblemswithsomeclientapplications.Somebrowsers(e.g.Chrome)handleda408responsebyresendingtheoriginalrequest,whichistheoppositeofwhatisdesiredwhenajobshouldbecanceled.
ThereforeArangoDBwillreturnHTTPstatuscode410(gone)forcanceledoperationsfromversion2.8on.
Foxx
ModelandRepository
DuetocompatibilityissuestheModelandRepositorytypesarenolongerimplementedasES2015classes.
Thepre-2.7"extend"stylesubclassingissupportedagainandwillnotemitanydeprecationwarnings.
varFoxx=require('org/arangodb/foxx');
varMyModel=Foxx.Model.extend({
//...
schema:{/*...*/}
});
Moduleresolution
Incompatiblechangesin2.8
580
ThebehavioroftheJavaScriptmoduleresolutionusedbytherequirefunctionhasbeenmodifiedtoimprovecompatibilitywithmoduleswrittenforNode.js.
Specifically
absolutepaths(e.g./some/absolute/path)arenowalwaysinterpretedasabsolutefilesystempaths,relativetothefilesystemroot
globalnames(e.g.global/name)arenowfirstintepretedasreferencestomodulesresidinginarelevantnode_modulesfolder,abuilt-inmoduleoramatchingdocumentintheinternal_modulescollection,andonlyresolvedtolocalfilepathsifnoothermatchisfound
Previouslythetwoformatsweretreatedinterchangeablyandwouldberesolvedtolocalfilepathsfirst,leadingtoproblemswhenlocalfilesusedthesamenamesasothermodules(e.g.alocalfilechai.jswouldcauseproblemswhentryingtoloadthechaimoduleinstalledinnode_modules).
Formoreinformationseetheblogannouncementofthischangeandtheupgradeguide.
Moduleorg/arangodb/request
Themodulenowalwaysreturnsresponsebodies,evenforerrorresponses.Inversionspriorto2.8themodulewouldsilentlydropresponsebodiesiftheresponseheaderindicatedanerror.
TheoldbehaviorofnotreturningbodiesforerrorresponsescanberestoredbyexplicitlysettingtheoptionreturnBodyOnErrortofalse:
letresponse=request({
//...
returnBodyOnError:false
});
Garbagecollection
TheV8garbagecollectionstrategywasslightlyadjustedsothatiteventuallyhappensinallV8contextsthatholdV8externalobjects(referencestoArangoDBdocumentsandcollections).Thisenablesabettercleanupoftheseresourcesandpreventsotherprocessessuchascompactionbeingstalledwhilewaitingfortheseresourcestobereleased.
InthiscontextthedefaultvaluefortheJavaScriptgarbagecollectionfrequency(--javascript.gc-frequency)wasalsoincreasedfrom10secondsto15seconds,aslessinternaloperationsinArangoDBarecarriedoutinJavaScript.
Clienttools
arangodumpwillnowfailbydefaultwhentryingtodumpedgesthatrefertoalreadydroppedcollections.Thiscanbecircumventedbyspecifyingtheoption--forcetruewheninvokingarangodump
Incompatiblechangesin2.8
581
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.7.ArangoDB2.7alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.
Performanceimprovements
Indexbuckets
Theprimaryindexesandhashindexesofcollectionscannowbesplitintomultipleindexbuckets.ThisoptionwasavailableforedgeindexesonlyinArangoDB2.6.
Abucketcanbeconsideredacontainerforaspecificrangeofindexvalues.Forprimary,hashandedgeindexes,determiningtheresponsiblebucketforanindexvalueisdonebyhashingtheactualindexvalueandapplyingasimplearithmeticoperationonthehash.
Becauseanindexvaluewillbepresentinatmostonebucketandbucketsareindependent,usingmultiplebucketsprovidesthefollowingbenefits:
initiallybuildingthein-memoryindexdatacanbeparallelizedevenforasingleindex,withonethreadperbucket(orwiththreadsbeingresponsibleformorethanonebucketatatime).Thiscanhelpreducingtheloadingtimeforcollections.
resizinganindexwhenitisabouttorunoutofreservespaceisperformedperbucket.Aseachbucketonlycontainsafractionoftheentireindex,resizingandrehashingabucketismuchfasterandlessintrusivethanresizingandrehashingtheentireindex.
Whencreatingnewcollections,thedefaultnumberofindexbucketsis8sinceArangoDB2.7.Inpreviousversions,thedefaultvaluewas1.Thenumberofbucketscanalsobeadjustedforexistingcollectionssotheycanbenefitfromtheoptimizations.Thenumberofindexbucketscanbesetforacollectionatanytimebyusingacollection'spropertiesfunction:
db.collection.properties({indexBuckets:16});
Thenumberofindexbucketsmustbeapowerof2.
Pleasenotethatforbuildingtheindexdataformultiplebucketsinparallelitisrequiredthatacollectioncontainsasignificantamountofdocumentsbecauseforalownumberofdocumentstheoverheadofparallelizationwilloutweighitsbenefits.Thecurrentthresholdvalueis256kdocuments,butthisvaluemaychangeinfutureversionsofArangoDB.Additionally,theconfigurationoption--database.index-threadswilldeterminehowmanyparallelthreadsmaybeusedforbuildingtheindexdata.
Fasterupdateandremoveoperationsinnon-uniquehashindexes
TheuniquehashindexesinArangoDBprovidedanamortizedO(1)lookup,insert,updateandremoveperformance.Non-uniquehashindexesprovidedamortizedO(1)insertperformance,buthadworseperformanceforupdateandremoveoperationsfornon-uniquevalues.Fordocumentswiththesameindexvalue,theymaintainedalistofcollisions.Whenadocumentwasupdatedorremoved,thatexactdocumenthadtobefoundinthecollisionslistfortheindexvalue.WhilegettingtothestartofthecollisionslistwasO(1),scanningthelisthadO(n)performanceintheworstcase(withnbeingthenumberofdocumentswiththesameindexvalue).Overall,thismadeupdateandremoveoperationsinnon-uniquehashindexesslowiftheindexcontainedmanyduplicatevalues.
ThishasbeenchangedinArangoDB2.7sothatnon-uniquehashindexesnowalsoprovideupdateandremoveoperationswithanamortizedcomplexityofO(1),eveniftherearemanyduplicates.
Resizingnon-uniquehashindexesnowalsodoesn'trequirelookingintothedocumentdata(whichmayinvolveadiskaccess)becausetheindexmaintainssomeinternalcachevalueperdocument.Whenresizingandrehashingtheindex(oranindexbucket),theindexwillfirstcompareonlythecachevaluesbeforepeekingintotheactualdocuments.Thischangecanalsoleadtoreducedindexresizingtimes.
Throughputenhancements
TheArangoDB-internalimplementationsfordispatchingrequests,keepingstatisticsandassigningV8contextstothreadshavebeenimprovedinordertouselesslocks.Thesechangesallowhigherconcurrencyandthroughputinthesecomponents,whichcanalsomaketheserverhandlemorerequestsinagivenperiodoftime.
WhatsNewin2.7
582
Whatgainscanbeexpecteddependsonwhichoperationsareexecuted,buttherearereal-worldcasesinwhichthroughputincreasedbybetween25%and70%whencomparedto2.6.
Madvisehints
TheLinuxvariantforArangoDBprovidestheOSwithmadvisehintsaboutindexmemoryanddatafilememory.Thesehintscanspeedupthingswhenmemoryistight,inparticularatcollectionloadtimebutalsoforrandomaccesseslater.ThereisnoformalguaranteethattheOSactuallyusesthemadvisehintsprovidedbyArangoDB,butactualmeasurementshaveshownimprovementsforloadingbiggercollections.
AQLimprovements
Additionaldatefunctions
ArangoDB2.7providesseveralextraAQLfunctionsfordateandtimecalculationandmanipulation.ThesefunctionswerecontributedbyGitHubusers@[email protected]!
Thefollowingextradatefunctionsareavailablefrom2.7on:
DATE_DAYOFYEAR(date):Returnsthedayofyearnumberofdate.Thereturnvaluesrangefrom1to365,or366inaleapyearrespectively.
DATE_ISOWEEK(date):ReturnstheISOweekdateofdate.Thereturnvaluesrangefrom1to53.Mondayisconsideredthefirstdayoftheweek.Therearenofractionalweeks,thusthelastdaysinDecembermaybelongtothefirstweekofthenextyear,andthefirstdaysinJanuarymaybepartofthepreviousyear'slastweek.
DATE_LEAPYEAR(date):Returnswhethertheyearofdateisaleapyear.
DATE_QUARTER(date):Returnsthequarterofthegivendate(1-based):
1:January,February,March2:April,May,June3:July,August,September4:October,November,December
DATE_DAYS_IN_MONTH(date):Returnsthenumberofdaysindate'smonth(28..31).
DATE_ADD(date,amount,unit):Addsamountgiveninunittodateandreturnsthecalculateddate.
unitcanbeeitherofthefollowingtospecifythetimeunittoaddorsubtract(case-insensitive):
y,year,yearsm,month,monthsw,week,weeksd,day,daysh,hour,hoursi,minute,minutess,second,secondsf,millisecond,milliseconds
amountisthenumberofunitstoadd(positivevalue)orsubtract(negativevalue).
DATE_SUBTRACT(date,amount,unit):Subtractsamountgiveninunitfromdateandreturnsthecalculateddate.
ItworksthesameasDATE_ADD(),exceptthatitsubtracts.ItisequivalenttocallingDATE_ADD()withanegativeamount,exceptthatDATE_SUBTRACT()canalsosubtractISOdurations.NotethatnegativeISOdurationsarenotsupported(i.e.startingwith-P,like-P1Y).
DATE_DIFF(date1,date2,unit,asFloat):Calculatethedifferencebetweentwodatesingiventimeunit,optionallywithdecimalplaces.Returnsanegativevalueifdate1isgreaterthandate2.
DATE_COMPARE(date1,date2,unitRangeStart,unitRangeEnd):Comparetwopartialdatesandreturntrueiftheymatch,falseotherwise.Thepartstocomparearedefinedbyarangeoftimeunits.
WhatsNewin2.7
583
Thefullrangeis:years,months,days,hours,minutes,seconds,milliseconds.PasstheunittostartfromasunitRangeStart,andtheunittoendwithasunitRangeEnd.Allunitsinbetweenwillbecompared.LeaveoutunitRangeEndtoonlycompareunitRangeStart.
DATE_FORMAT(date,format):Formatadateaccordingtothegivenformatstring.Itsupportsthefollowingplaceholders(case-insensitive):
%t:timestamp,inmillisecondssincemidnight1970-01-01%z:ISOdate(0000-00-00T00:00:00.000Z)%w:dayofweek(0..6)%y:year(0..9999)%yy:year(00..99),abbreviated(lasttwodigits)%yyyy:year(0000..9999),paddedtolengthof4%yyyyyy:year(-009999..+009999),withsignprefixandpaddedtolengthof6%m:month(1..12)%mm:month(01..12),paddedtolengthof2%d:day(1..31)%dd:day(01..31),paddedtolengthof2%h:hour(0..23)%hh:hour(00..23),paddedtolengthof2%i:minute(0..59)%ii:minute(00..59),paddedtolengthof2%s:second(0..59)%ss:second(00..59),paddedtolengthof2%f:millisecond(0..999)%fff:millisecond(000..999),paddedtolengthof3%x:dayofyear(1..366)%xxx:dayofyear(001..366),paddedtolengthof3%k:ISOweekdate(1..53)%kk:ISOweekdate(01..53),paddedtolengthof2%l:leapyear(0or1)%q:quarter(1..4)%a:daysinmonth(28..31)%mmm:abbreviatedEnglishnameofmonth(Jan..Dec)%mmmm:Englishnameofmonth(January..December)%www:abbreviatedEnglishnameofweekday(Sun..Sat)%wwww:Englishnameofweekday(Sunday..Saturday)%&:specialescapesequenceforrareoccasions%%:literal%%:ignored
RETURNDISTINCT
Toreturnuniquevaluesfromaquery,AQLnowprovidestheDISTINCTkeyword.ItcanbeusedasamodifierforRETURNstatements,asashorteralternativetothealreadyexistingCOLLECTstatement.
Forexample,thefollowingqueryonlyreturnsdistinct(unique)statusattributevaluesfromthecollection:
FORdocINcollection
RETURNDISTINCTdoc.status
RETURNDISTINCTisnotallowedonthetop-levelofaqueryifthereisnoFORloopinfrontofit.RETURNDISTINCTisallowedinsubqueries.
RETURNDISTINCTensuresthatthevaluesreturnedaredistinct(unique),butdoesnotguaranteeanyorderofresults.Inordertohavecertainresultorder,anadditionalSORTstatementmustbeaddedtoaquery.
Shorthandobjectnotation
WhatsNewin2.7
584
AQLnowprovidesashorthandnotationforobjectliteralsinthestyleofES6objectliterals:
LETname="Peter"
LETage=42
RETURN{name,age}
Thisisequivalenttothepreviouslyavailablecanonicalform,whichisstillavailableandsupported:
LETname="Peter"
LETage=42
RETURN{name:name,age:age}
Arrayexpansionimprovements
Thealreadyexisting[*]operatorhasbeenimprovedwithoptionalfilteringandprojectionandlimitcapabilities.
Forexample,considerthefollowingexamplequerythatfiltersvaluesfromanarrayattribute:
FORuINusers
RETURN{
name:u.name,
friends:(
FORfINu.friends
FILTERf.age>u.age
RETURNf.name
)
}
Withthe[*]operator,thisquerycanbesimplifiedto
FORuINusers
RETURN{name:u.name,friends:u.friends[*FILTERCURRENT.age>u.age].name}
Thepseudo-variableCURRENTcanbeusedtoaccessthecurrentarrayelement.TheFILTERconditioncanrefertoCURRENToranyvariablesvalidintheouterscope.
Toreturnaprojectionofthecurrentelement,therecannowbeaninlineRETURN:
FORuINusers
RETURNu.friends[*RETURNCONCAT(CURRENT.name,"isafriendof",u.name)]
whichisthesimplifiedvariantfor:
FORuINusers
RETURN(
FORfriendINu.friends
RETURNCONCAT(friend.name,"isafriendof",u.name)
)
Arraycontraction
Inordertocollapse(orflatten)resultsinnestedarrays,AQLnowprovidesthe[**]operator.Itworkssimilartothe[*]operator,butadditionallycollapsesnestedarrays.Howmanylevelsarecollapsedisdeterminedbytheamountof*charactersused.
Forexample,considerthefollowingquerythatproducesanestedresult:
FORuINusers
RETURNu.friends[*].name
The[**]operatorcannowbeappliedtogetridofthenestedarrayandturnitintoaflatarray.Wesimplyapplythe[**]onthepreviousqueryresult:
WhatsNewin2.7
585
RETURN(
FORuINusersRETURNu.friends[*].name
)[**]
Templatequerystrings
AssemblingquerystringsinJavaScripthasbeenerror-pronewhenusingsimplestringconcatenation,especiallybecauseplainJavaScriptstringsdonothavemultiline-support,andbecauseofpotentialparameterinjectionissues.WhilemultilinequerystringscanbeassembledwithES6templatestringssinceArangoDB2.5,andquerybindparametersaretheresinceArangoDB1.0topreventparameterinjection,therewasnoJavaScript-ysolutiontocombinethese.
ArangoDB2.7nowprovidesanES6templatestringgeneratorfunctionthatcanbeusedtoeasilyandsafelyassembleAQLqueriesfromJavaScript.JavaScriptvariablesandexpressionscanbeusedeasilyusingregularES6templatestringsubstitutions:
letname='test';
letattributeName='_key';
letquery=aqlQuery`FORuINusers
FILTERu.name==${name}
RETURNu.${attributeName}`;
db._query(query);
ThisismorelegiblethanwhenusingaplainJavaScriptstringandalsodoesnotrequiredefiningthebindparametervaluesseparately:
letname='test';
letattributeName='_key';
letquery="FORuINusers"+
"FILTERu.name==@name"+
"RETURNu.@attributeName";
db._query(query,{
name,
attributeName
});
TheaqlQuerytemplatestringgeneratorwillalsohandlecollectionobjectsautomatically:
db._query(aqlQuery`FORuIN${db.users}RETURNu.name`);
NotethatwhiletemplatestringsareavailableintheJavaScriptfunctionsprovidedtobuildqueries,theyaren'tafeatureofAQLitself.AQLcouldalwayshandlemultilinequerystringsandprovidedbindparameters(@...)forseparatingthequerystringandtheparametervalues.TheaqlQuerytemplatestringgeneratorfunctionwilltakecareofthisseparation,too,butwilldoitbehindthescenes.
AQLqueryresultcache
TheAQLqueryresultcachecanoptionallycachethecompleteresultsofallorjustselectedAQLqueries.Itcanbeoperatedinthefollowingmodes:
off:thecacheisdisabled.Noqueryresultswillbestoredon:thecachewillstoretheresultsofallAQLqueriesunlesstheircacheattributeflagissettofalsedemand:thecachewillstoretheresultsofAQLqueriesthathavetheircacheattributesettotrue,butwillignoreallothers
Themodecanbesetatserverstartupusingthe--database.query-cache-modeconfigurationoptionandlaterchangedatruntime.Thedefaultvalueisoff,meaningthatthequeryresultcacheisdisabled.Thisisbecausethecachemayconsumeadditionalmemorytokeepqueryresults,andalsobecauseitmustbeinvalidatedwhenchangeshappenincollectionsforwhichresultshavebeencached.
Thequeryresultcachemaythereforehavepositiveornegativeeffectsonqueryexecutiontimes,dependingontheworkload:itwillnotmakemuchsenseturningonthecacheinwrite-onlyorwrite-mostlyscenarios,butthecachemaybeverybeneficialincaseworkloadsareread-onlyorread-mostly,andqueryarecomplex.
Ifthequerycacheisoperatedindemandmode,itcanbecontrolledperqueryifthecacheshouldbecheckedforaresult.
WhatsNewin2.7
586
Miscellaneouschanges
Optimizer
TheAQLoptimizerrulepatch-update-statementshasbeenadded.ThisrulecanoptimizecertainAQLUPDATEqueriesthatupdatedocumentsintheacollectionthattheyalsoiterateover.
Forexample,thefollowingqueryreadsdocumentsfromacollectioninordertoupdatethem:
FORdocINcollection
UPDATEdocWITH{newValue:doc.oldValue+1}INcollection
Inthiscase,onlyasinglecollectionisaffectedbythequery,andthereisnoindexlookupinvolvedtofindtheto-be-updateddocuments.Inthiscase,theUPDATEquerydoesnotrequiretakingafull,memory-intensivesnapshotofthecollection,butitcanbeperformedinsmallchunks.Thiscanleadtomemorysavingswhenexecutingsuchqueries.
Functioncallargumentsoptimization
ThisoptimizationwillleadtoargumentsinfunctioncallsinsideAQLqueriesnotbeingcopiedbutbeingpassedbyreference.ThismayspeedupcallstofunctionswithbiggerargumentvaluesorqueriesthatcallAQLfunctionsalotoftimes.
WebAdminInterface
Thewebinterfacenowhasanewdesign.
The"Applications"tabinthewebinterfaceshasbeenrenamedto"Services".
TheArangoDBAPIdocumentationhasbeenmovedfromthe"Tools"menutothe"Links"menu.ThenewdocumentationisbasedonSwagger2.0andopensinaseparatewebpage.
Foxximprovements
ES2015Classes
AllFoxxconstructorshavebeenreplacedwithES2015classesandcanbeextendedusingtheclasssyntax.TheextendmethodisstillsupportedatthemomentbutwillbecomedeprecatedinArangoDB2.8andremovedinArangoDB2.9.
Before:
varFoxx=require('org/arangodb/foxx');
varMyModel=Foxx.Model.extend({
//...
schema:{/*...*/}
});
After:
varFoxx=require('org/arangodb/foxx');
classMyModelextendsFoxx.Model{
//...
}
MyModel.prototype.schema={/*...*/};
Confidentialconfiguration
Itisnowpossibletospecifyconfigurationoptionswiththetypepassword.ThepasswordtypeisequivalenttothetexttypebutwillbemaskedinthewebfrontendtopreventaccidentalexposureofconfidentialoptionslikeAPIkeysandpasswordswhenconfiguringyourFoxxapplication.
WhatsNewin2.7
587
Dependencies
Thesyntaxforspecifyingdependenciesinmanifestshasbeenextendedtoallowspecifyingoptionaldependencies.Unmetoptionaldependencieswillnotpreventanappfrombeingmounted.Thetraditionalshorthandsyntaxforspecifyingnon-optionaldependencieswillstillbesupportedintheupcomingversionsofArangoDB.
Before:
{
...
"dependencies":{
"notReallyNeeded":"users:^1.0.0",
"totallyNecessary":"sessions:^1.0.0"
}
}
After:
{
"dependencies":{
"notReallyNeeded":{
"name":"users",
"version":"^1.0.0",
"required":false
},
"totallyNecessary":{
"name":"sessions",
"version":"^1.0.0"
}
}
}
Replication
TheexistingreplicationHTTPAPIhasbeenextendedwithmethodsthatreplicationclientscanusetodeterminewhetheragivendate,identifiedbyatickvalue,isstillpresentonamasterforreplication.BycallingtheseAPIs,clientscanmakeaninformeddecisionaboutwhetherthemastercanstillprovideallmissingdatastartingfromthepointuptowhichtheclienthadalreadysynchronized.Thiscanbehelpfulincaseareplicationclientisre-startedafterapause.
Masterserversnowalsotrackupthepointuptowhichtheyhavesentchangestoclientsforreplication.Thisinformationcanbeusedtodeterminethepointofdatathatreplicationclientshavereceivedfromthemaster,andifandhowfarapproximatelytheylagbehind.
Finally,restartingthereplicationapplieronaslaveserverhasbeenmademorerobustincasetheapplierwasstoppedwhiletherewerependingtransactionsonthemasterserver,andre-startingthereplicationapplierneedstorestorethestateofthesetransactions.
Clienttools
Thefilenamesindumpscreatedbyarangodumpnowcontainnotonlythenameofthedumpedcollection,butalsoanadditional32-digithashvalue.Thisisdonetopreventoverwritingdumpfilesincase-insensitivefilesystemswhenthereexistmultiplecollectionswiththesamename(butwithdifferentcases).
Forexample,ifadatabasehadtwocollectionstestandTest,previousversionsofarangodumpcreatedthefollowingfiles:
test.structure.jsonandtest.data.jsonforcollectiontestTest.structure.jsonandTest.data.jsonforcollectionTest
Thisdidnotworkincase-insensitivefilesystems,becausethefilesforthesecondcollectionwouldhaveoverwrittenthefilesofthefirst.arangodumpin2.7willcreatetheuniquefilesinthiscase,byappendingthe32-digithashvaluetothecollectionnameinallcase.Thesefilenameswillbeunambiguousevenincase-insensitivefilesystems.
Miscellaneouschanges
WhatsNewin2.7
588
Bettercontrol-Csupportinarangosh
WhenCTRL-Cispressedinarangosh,itwillnowabortthelocallyrunningcommand(ifany).Ifnocommandwasrunning,pressingCTRL-Cwillprinta Cfirst.PressingCTRL-Cagainwillthenquitarangosh.
CTRL-Ccanalsobeusedtoresetthecurrentpromptwhileenteringcomplexnestedobjectswhichspanmultipleinputlines.
CTRL-CsupporthasbeenaddedtotheArangoShellversionsbuiltwithReadline-support(LinuxandMacOSonly).TheWindowsversionofArangoDBusesadifferentlibraryforhandlinginput,andsupportforCTRL-Chasnotbeenaddedthereyet.
Start/stop
LinuxstartupscriptsandsystemdconfigurationforarangodnowtrytoadjusttheNOFILE(numberofopenfiles)limitsfortheprocess.Thelimitvalueissetto131072(128k)whenArangoDBisstartedviastart/stopcommands.
ThiswillpreventarangodrunningoutofavailablefiledescriptorsincaseofmanyparallelHTTPconnectionsorlargecollectionswithmanydatafiles.
Additionally,whenArangoDBisstarted/stoppedmanuallyviathestart/stopcommands,themainprocesswillwaitforupto10secondsafteritforksthesupervisorandarangodchildprocesses.Ifthestartupfailswithinthatperiod,thestart/stopscriptwillfailwithanon-zeroexitcode,allowinganyinvokingscriptstohandlethiserror.Previousversionsalwaysreturnedanexitcodeof0,evenwhenarangodcouldn'tbestarted.
Ifthestartupofthesupervisororarangodisstillongoingafter10seconds,themainprogramwillstillreturnwithexitcode0inordertonotblockanyscripts.Thelimitof10secondsisarbitrarybecausethetimerequiredforanarangodstartupisnotknowninadvance.
Non-sparselogfiles
WALlogfilesanddatafilescreatedbyarangodarenownon-sparse.ThispreventsSIGBUSsignalsbeingraisedwhenamemory-mappedregionbackedbyasparsedatafilewasaccessedandthememoryregionwasnotactuallybackedbydisk,forexamplebecausethediskranoutofspace.
arangodnowalwaysfullyallocatesthediskspacerequiredforalogfileordatafilewhenitcreatesone,sothememoryregioncanalwaysbebackedbydisk,andmemorycanbeaccessedwithoutSIGBUSbeingraised.
WhatsNewin2.7
589
IncompatiblechangesinArangoDB2.7ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.7,andadjustanyclientprogramsifnecessary.
AQLchanges
DISTINCTisnowakeywordinAQL.
AQLqueriesthatuseDISTINCT(inlower,upperormixedcase)asanidentifier(i.e.asavariable,acollectionnameorafunctionname)willstopworking.Tomakesuchqueriesworkingagain,eachoccurrenceofDISTINCTinanAQLqueryshouldbeenclosedinbackticks.ThiswillturnDISTINCTfromakeywordintoanidentifieragain.
TheAQLfunctionSKIPLIST()hasbeenremovedinArangoDB2.7.ThisfunctionwasdeprecatedinArangoDB2.6.Itwasaleft-overfromtimeswhenthequeryoptimizerwasn'tabletouseskiplistindexestogetherwithfilters,skipandlimitvalues.Sincethisissuebeenfixedsinceversion2.3,thereisnoAQLreplacementfunctionforSKIPLIST.QueriesthatusetheSKIPLISTfunctioncanbefixedbyusingtheusualcombinationofFOR,FILTERandLIMIT,e.g.
FORdocIN@@collection
FILTERdoc.value>=@value
SORTdoc.valueDESC
LIMIT1
RETURNdoc
Foxxchanges
Bundlingandcompilation
TheassetspropertyisnolongersupportedinFoxxmanifestsandisscheduledtoberemovedinafutureversionofArangoDB.Thefilespropertycanstillbeusedtoservestaticassetsbutitisrecommendedtouseseparatetoolingtocompileandbundleyourassets.
Manifestscripts
Thepropertiessetupandteardownhavebeenmovedintothescriptspropertymap:
Before:
{
...
"setup":"scripts/setup.js",
"teardown":"scripts/teardown.js"
}
After:
{
...
"scripts":{
"setup":"scripts/setup.js",
"teardown":"scripts/teardown.js"
}
}
FoxxQueues
Function-basedFoxxQueuejobtypesarenolongersupported.Tolearnabouthowyoucanusethenewscript-basedjobtypesfollowtheupdatedrecipeinthecookbook.
Incompatiblechangesin2.7
590
FoxxSessions
ThejwtandtypeoptionshavebeenremovedfromtheactivateSessionsAPI.
IfyouwanttoreplicatethebehaviorofthejwtoptionyoucanusetheJWTfunctionsinthecryptomodule.AJWT-basedsessionstoragethatdoesn'twritesessionstothedatabaseisavailableasthesessions-jwtappintheFoxxappstore.
Thesessiontypeisnowinferredfromthepresenceofthecookieorheaderoptions(allowingyoutoenablesupportforboth).Ifyouwanttousethedefaultsettingsforcookieorheaderyoucanpassthevaluetrueinstead.
ThesessionStorageAppoptionhasbeenremovedinfavourofthesessionStorageoption.
Before:
varFoxx=require('org/arangodb/foxx');
varctrl=newFoxx.Controller(applicationContext);
ctrl.activateSessions({
sessionStorageApp:'some-sessions-app',
type:'cookie'
});
After:
ctrl.activateSessions({
sessionStorage:applicationContext.dependencies.sessions.sessionStorage,
cookie:true
});
Requestmodule
Themoduleorg/arangodb/requestusesaninternallibraryfunctionforsendingHTTPrequests.ThislibraryfunctionallyunconditionallysetanHTTPheaderAccept-Encoding:gzipinalloutgoingHTTPrequests,withoutclientcodehavingtosetthisheaderexplicitly.
Thishasbeenfixedin2.7,soAccept-Encoding:gzipisnotsetautomaticallyanymore.AdditionallytheheaderUser-Agent:ArangoDBisnotsetautomaticallyeither.Ifclientapplicationsrelyontheseheadersbeingsent,theyarefreetoadditwhenconstructingrequestsusingtherequestmodule.
Theinternal.download()functionisalsoaffectedbythischange.Again,theheadercanbeaddedhereifrequiredbypassingitviaaheaderssub-attributeinthethirdparameter(options)tothisfunction.
arangodump/backupsThefilenamesindumpscreatedbyarangodumpnowcontainnotonlythenameofthedumpedcollection,butalsoanadditional32-digithashvalue.Thisisdonetopreventoverwritingdumpfilesincase-insensitivefilesystemswhenthereexistmultiplecollectionswiththesamename(butwithdifferentcases).
Thischangeleadstochangedfilenamesindumpscreatedbyarangodump.Ifanyclientscriptsdependonthefilenamesinthedumpoutputdirectorybeingequaltothecollectionnameplusoneofthesuffixes.structure.jsonand.data.json,theyneedtobeadjusted.
StartingwithArangoDB2.7,thefilenameswillcontainanunderscoreplusthe32-digitMD5value(representedinhexadecimalnotation)ofthecollectionname.
Forexample,whenarangodumpdumpsdataoftwocollectionstestandTest,thefilenamesinpreviousversionsofArangoDBwere:
test.structure.json(definitionsforcollectiontest)test.data.json(dataforcollectiontest)Test.structure.json(definitionsforcollectionTest)Test.data.json(dataforcollectionTest)
In2.7,thefilenameswillbe:
test_098f6bcd4621d373cade4e832627b4f6.structure.json(definitionsforcollectiontest)
Incompatiblechangesin2.7
591
test_098f6bcd4621d373cade4e832627b4f6.data.json(dataforcollectiontest)Test_0cbc6611f5540bd0809a388dc95a615b.structure.json(definitionsforcollectionTest)Test_0cbc6611f5540bd0809a388dc95a615b.data.json(dataforcollectionTest)
Starting/stopping
Whenstartingarangod,theserverwillnowdroptheprocessprivilegestothespecifiedvaluesinoptions--server.uidand--server.gidinstantlyafterparsingthestartupoptions.
Thatmeanswheneither--server.uidor--server.gidareset,theprivilegechangewillhappenearlier.Thismaypreventbindingtheservertoanendpointwithaportnumberlowerthan1024ifthearangodbuserhasnoprivilegesforthat.PreviousversionsofArangoDBchangedtheprivilegeslater,sosomestartupactionswerestillcarriedoutundertheinvokinguser(i.e.likelyrootwhenstartedviainit.dorsystemscripts)andespeciallybindingtolowportnumberswasstillpossiblethere.
Thedefaultprivilegesforuserarangodbwillnotbesufficientforbindingtoportnumberslowerthan1024.TohaveanArangoDB2.7bindtoaportnumberlowerthan1024,itneedstobestartedwitheitheradifferentprivilegeduser,ortheprivilegesofthearangodbuserhavetoraisedmanuallybeforehand.
Additionally,LinuxstartupscriptsandsystemdconfigurationforarangodnowwilladjusttheNOFILE(numberofopenfiles)limitsfortheprocess.Thelimitvalueissetto131072(128k)whenArangoDBisstartedviastart/stopcommands.Thegoalofthischangeistopreventarangodfromrunningoutofavailablefiledescriptorsforsocketconnectionsanddatafiles.
Connectionhandling
arangodwillnowactuallycloselingeringclientconnectionswhenidleforatleastthedurationspecifiedinthe--server.keep-alive-timeoutstartupoption.
InpreviousversionsofArangoDB,idleconnectionswerenotclosedbytheserverwhenthetimeoutwasreachedandtheclientwasstillconnected.Nowtheconnectionisproperlyclosedbytheserverincaseoftimeout.Clientapplicationsrelyingontheoldbehaviormaynowneedtoreconnecttotheserverwhentheiridleconnectionstimeoutandgetclosed(note:connectionsbeingidleforalongtimemaybeclosedbytheOSorfirewallsanyway-clientapplicationsshouldbeawareofthatandtrytoreconnect).
Optionchanges
Configureoptionsremoved
Thefollowingoptionsforconfigurehavebeenremovedbecausetheywereunusedorexotic:
--enable-timings
--enable-figures
Startupoptionsadded
Thefollowingconfigurationoptionshavebeenaddedin2.7:
--database.query-cache-max-results:setsthemaximumnumberofresultsinAQLqueryresultcacheperdatabase--database.query-cache-mode:setsthemodefortheAQLqueryresultscache.Possiblevaluesareon,offanddemand.Thedefaultvalueisoff
Miscellaneouschanges
Simplequeries
Manysimplequeriesprovideaskip()functionthatcanbeusedtoskipoveracertainnumberofdocumentsintheresult.ThisfunctionallowedspecifyingnegativeoffsetsinpreviousversionsofArangoDB.Specifyinganegativeoffsetledtothequeryresultbeingiteratedinreverseorder,soskippingwasperformedfromthebackoftheresult.Asmostsimplequeriesdonotprovideaguaranteedresultorder,
Incompatiblechangesin2.7
592
skippingfromthebackofaresultwithunspecificorderseemsaratherexoticusecaseandwasremovedtoincreaseconsistencywith
AQL,whichalsodoesnotprovidenegativeskipvalues.
NegativeskipvaluesweredeprecatedinArangoDB2.6.
TasksAPI
TheundocumentedfunctionaddJob()hasbeenremovedfromtheorg/arangodb/tasksmoduleinArangoDB2.7.
RuntimeendpointsmanipulationAPI
ThefollowingHTTPRESTAPImethodsforruntimemanipulationofserverendpointshavebeenremovedinArangoDB2.7:
POST/_api/endpoint:todynamicallyaddanendpointwhiletheserverwasrunningDELETE/_api/endpoint:todynamicallyremoveanendpointwhiletheserverwasrunning
ThischangealsoaffectstheequivalentJavaScriptendpointmanipulationmethodsavailableinFoxx.ThefollowingfunctionshavebeenremovedinArangoDB2.7:
db._configureEndpoint()
db._removeEndpoint()
Incompatiblechangesin2.7
593
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.6.ArangoDB2.6alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.
APIsadded
Batchdocumentremovalandlookupcommands
Thefollowingcommandshavebeenaddedforcollectionobjects:
collection.lookupByKeys(keys)collection.removeByKeys(keys)
Thesecommandscanbeusedtoperformmulti-documentlookupandremovaloperationsefficientlyfromtheArangoShell.Theargumenttotheseoperationsisanarrayofdocumentkeys.
ThesecommandscanalsobeusedviatheHTTPRESTAPI.Theirendpointsare:
PUT/_api/simple/lookup-by-keysPUT/_api/simple/remove-by-keys
CollectionexportHTTPRESTAPI
ArangoDBnowprovidesadedicatedcollectionexportAPI,whichcantakesnapshotsofentirecollectionsmoreefficientlythanthegeneral-purposecursorAPI.TheexportAPIisusefultotransferthecontentsofanentirecollectiontoaclientapplication.Itprovidesoptionalfilteringonspecificattributes.
TheexportAPIisavailableatendpointPOST/_api/export?collection=....TheAPIhasthesamereturnvaluestructureasthealreadyestablishedcursorAPI(POST/_api/cursor).
AnintroductiontotheexportAPIisgiveninthisblogpost:http://jsteemann.github.io/blog/2015/04/04/more-efficient-data-exports/
AQLimprovements
EDGESAQLFunction
TheAQLfunctionEDGESgotanewfifthoptionalparameter,whichmustbeanobjectifspecified.Rightnowonlyoneoptionisavailableforit:
includeVerticesthisisabooleanparameterthatallowstomodifytheresultofEDGES().ThedefaultvalueforincludeVerticesisfalse,whichdoesnothaveanyeffect.Settingittotruewillmodifytheresult,suchthatalsotheconnectedverticesarereturnedalongwiththeedges:
{vertex:<vertexDocument>,edge:<edgeDocument>}
SubqueryoptimizationsforAQLqueries
Thisoptimizationavoidscopyingintermediateresultsintosubqueriesthatarenotrequiredbythesubquery.
Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/04/subquery-optimizations/
ReturnvalueoptimizationforAQLqueries
Thisoptimizationavoidscopyingthefinalqueryresultinsidethequery'smainReturnNode.
Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/04/return-value-optimization-for-aql/
WhatsNewin2.6
594
SpeedupAQLqueriescontainingbigINlistsforindexlookups
INlistsusedforindexlookupshadperformanceissuesinpreviousversionsofArangoDB.Theseissueshavebeenaddressedin2.6sousingbiggerINlistsforfilteringismuchfaster.
Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/07/in-list-improvements/
AddedalternativeimplementationforAQLCOLLECT
Thealternativemethodusesahashtableforgroupinganddoesnotrequireitsinputelementstobesorted.ItwillbetakenintoaccountbytheoptimizerforCOLLECTstatementsthatdonotuseanINTOclause.
IncaseaCOLLECTstatementcanusethehashtablevariant,theoptimizerwillcreateanextraplanforitatthebeginningoftheplanningphase.Inthisplan,noextraSORTnodewillbeaddedinfrontoftheCOLLECTbecausethehashtablevariantofCOLLECTdoesnotrequiresortedinput.Instead,aSORTnodewillbeaddedafterittosortitsoutput.ThisSORTnodemaybeoptimizedawayagaininlaterstages.Ifthesortorderoftheresultisirrelevanttotheuser,addinganextraSORTnullafterahashCOLLECToperationwillallowtheoptimizertoremovethesortsaltogether.
InadditiontothehashtablevariantofCOLLECT,theoptimizerwillmodifytheoriginalplantousetheregularCOLLECTimplementation.Asthisimplementationrequiressortedinput,theoptimizerwillinsertaSORTnodeinfrontoftheCOLLECT.ThisSORTnodemaybeoptimizedawayinlaterstages.
Thecreatedplanswillthenbeshippedthroughtheregularoptimizationpipeline.Intheend,theoptimizerwillpicktheplanwiththelowestestimatedtotalcostasusual.Thehashtablevariantdoesnotrequireanup-frontsortoftheinput,andwillthusbepreferredovertheregularCOLLECTiftheoptimizerestimatesmanyinputelementsfortheCOLLECTnodeandcannotuseanindextosortthem.
TheoptimizercanbeexplicitlytoldtousetheregularsortedvariantofCOLLECTbysuffixingaCOLLECTstatementwithOPTIONS{"method":"sorted"}.ThiswilloverridetheoptimizerguessworkandonlyproducethesortedvariantofCOLLECT.
AblogpostonthenewCOLLECTimplementationcanbefoundhere:http://jsteemann.github.io/blog/2015/04/22/collecting-with-a-hash-table/
Simplifiedreturnvaluesyntaxfordata-modificationAQLqueries
ArangoDB2.4sinceversionallowstoreturnresultsfromdata-modificationAQLqueries.Thesyntaxforthiswasquitelimitedandverbose:
FORiIN1..10
INSERT{value:i}INtest
LETinserted=NEW
RETURNinserted
TheLETinserted=NEWRETURNinsertedwasrequiredliterallytoreturntheinserteddocuments.Nocalculationscouldbemadeusingtheinserteddocuments.
Thisisnowmoreflexible.Afteradata-modificationclause(e.g.INSERT,UPDATE,REPLACE,REMOVE,UPSERT)therecanfollowanynumberofLETcalculations.Thesecalculationscanrefertothepseudo-valuesOLDandNEWthatarecreatedbythedata-modificationstatements.
Thisallowsreturningprojectionsofinsertedorupdateddocuments,e.g.:
FORiIN1..10
INSERT{value:i}INtest
RETURN{_key:NEW._key,value:i}
Stillnoteveryconstructisallowedafteradata-modificationclause.Forexample,nofunctionscanbecalledthatmayaccessdocuments.
Moreinformationcanbefoundhere:http://jsteemann.github.io/blog/2015/03/27/improvements-for-data-modification-queries/
AddedAQLUPSERTstatement
WhatsNewin2.6
595
ThisaddsanUPSERTstatementtoAQLthatisacombinationofbothINSERTandUPDATE/REPLACE.TheUPSERTwillsearchforamatchingdocumentusingauser-providedexample.Ifnodocumentmatchestheexample,theinsertpartoftheUPSERTstatementwillbeexecuted.Ifthereisamatch,theupdate/replacepartwillbecarriedout:
UPSERT{page:'index.html'}/*searchexample*/
INSERT{page:'index.html',pageViews:1}/*insertpart*/
UPDATE{pageViews:OLD.pageViews+1}/*updatepart*/
INpageViews
UPSERTcanbeusedwithanUPDATEorREPLACEclause.TheUPDATEclausewillperformapartialupdateofthefounddocument,whereastheREPLACEclausewillreplacethefounddocumententirely.TheUPDATEorREPLACEpartscanrefertothepseudo-valueOLD,whichcontainsallattributesofthefounddocument.
UPSERTstatementscanoptionallyreturnvalues.Inthefollowingquery,thereturnattributefoundwillreturnthefounddocumentbeforetheUPDATEwasapplied.Ifnodocumentwasfound,foundwillcontainavalueofnull.Theupdatedresultattributewillcontaintheinserted/updateddocument:
UPSERT{page:'index.html'}/*searchexample*/
INSERT{page:'index.html',pageViews:1}/*insertpart*/
UPDATE{pageViews:OLD.pageViews+1}/*updatepart*/
INpageViews
RETURN{found:OLD,updated:NEW}
AmoredetaileddescriptionofUPSERTcanbefoundhere:http://jsteemann.github.io/blog/2015/03/27/preview-of-the-upsert-command/
Miscellaneouschanges
WhenerrorsoccurinsideAQLuserfunctions,theerrormessagewillnowcontainastacktrace,indicatingthelineofcodeinwhichtheerroroccurred.ThisshouldmakedebuggingAQLuserfunctionseasier.
WebAdminInterfaceArangoDB'sbuilt-inwebinterfacenowusessessions.Sessioninformationisstoredincookies,soclientsusingthewebinterfacemustacceptcookiesinordertouseit.
Thenewstartupoption--server.session-timeoutcanbeusedforadjustingthesessionlifetime.
TheAQLeditorinthewebinterfacenowprovidesanexplainfunctionality,whichcanbeusedforinspectingandperformance-tuningAQLqueries.ThequeryexecutiontimeisnowalsodisplayedintheAQLeditor.
Foxxappsthatrequireconfigurationoraremissingdependenciesarenowindicatedintheappoverviewanddetails.
Foxximprovements
ConfigurationandDependencies
Foxxappmanifestscannowdefineconfigurationoptions,aswellasdependenciesonotherFoxxapps.
AnintroductiontoFoxxconfigurationscanbefoundintheblog:https://www.arangodb.com/2015/05/reusable-foxx-apps-with-configurations/
AndtheblogpostonFoxxdependenciescanbefoundhere:https://www.arangodb.com/2015/05/foxx-dependencies-for-more-composable-foxx-apps/
MochaTests
YoucannowwritetestsforyourFoxxappsusingtheMochatestingframework:https://www.arangodb.com/2015/04/testing-foxx-mocha/
ArecipeforwritingtestsforyourFoxxappscanbefoundinthecookbook:https://docs.arangodb.com/2.8/Cookbook/FoxxTesting.html
WhatsNewin2.6
596
APIDocumentation
TheAPIdocumentationhasbeenupdatedtoSwagger2.YoucannowalsomountAPIdocumentationinyourownFoxxapps.
Alsoseetheblogpostintroducingthisfeature:https://www.arangodb.com/2015/05/document-your-foxx-apps-with-swagger-2/
CustomScriptsandFoxxQueue
InadditiontotheexistingsetupandteardownscriptsyoucannowdefinecustomscriptsinyourFoxxmanifestandinvoketheseusingthewebadmininterfaceortheFoxxmanagerCLI.Thesescriptscannowalsotakepositionalargumentsandexportreturnvalues.
JobtypesfortheFoxxQueuecannowbedefinedasascriptnameandappmountpathallowingtheuseofFoxxscriptsasjobtypes.Thepre-2.6jobtypesareknowntocauseissueswhenrestartingtheserverandareerror-prone;westronglyrecommendedconvertinganyexistingjobtypestothenewformat.
Clienttools
Thedefaultconfigurationvaluefortheoption--server.request-timeoutwasincreasedfrom300to1200secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).
Thedefaultconfigurationvaluefortheoption--server.connect-timeoutwasincreasedfrom3to5secondsforclienttools(arangosh,arangoimp,arangodump,arangorestore).
Arangorestore
Theoption--create-databasewasaddedforarangorestore.
Settingthisoptiontotruewillnowcreatethetargetdatabaseifitdoesnotexist.Whencreatingthetargetdatabase,theusernameandpasswordspassedtoarangorestorewillbeusedtocreateaninitialuserforthenewdatabase.
Thedefaultvalueforthisoptionisfalse.
Arangoimp
Arangoimpcannowoptionallyupdateorreplaceexistingdocuments,providedtheimportdatacontainsdocumentswith_keyattributes.
Previously,theimportcouldbeusedforinsertingnewdocumentsonly,andre-insertingadocumentwithanexistingkeywouldhavefailedwithauniquekeyconstraintviolatederror.
Thebehaviorofarangoimp(insert,update,replaceonduplicatekey)cannowbecontrolledwiththeoption--on-duplicate.Theoptioncanhaveoneofthefollowingvalues:
error:whenauniquekeyconstrainterroroccurs,donotimportorupdatethedocumentbutreportanerror.Thisisthedefault.
update:whenauniquekeyconstrainterroroccurs,tryto(partially)updatetheexistingdocumentwiththedataspecifiedintheimport.Thismaystillfailifthedocumentwouldviolatesecondaryuniqueindexes.Onlytheattributespresentintheimportdatawillbeupdatedandotherattributesalreadypresentwillbepreserved.ThenumberofupdateddocumentswillbereportedintheupdatedattributeoftheHTTPAPIresult.
replace:whenauniquekeyconstrainterroroccurs,trytofullyreplacetheexistingdocumentwiththedataspecifiedintheimport.Thismaystillfailifthedocumentwouldviolatesecondaryuniqueindexes.ThenumberofreplaceddocumentswillbereportedintheupdatedattributeoftheHTTPAPIresult.
ignore:whenauniquekeyconstrainterroroccurs,ignorethiserror.Therewillbenoinsert,updateorreplacefortheparticulardocument.IgnoreddocumentswillbereportedseparatelyintheignoredattributeoftheHTTPAPIresult.
Thedefaultvalueiserror.
Afewexamplesforusingarangoimpwiththe--on-duplicateoptioncanbefoundhere:http://jsteemann.github.io/blog/2015/04/14/updating-documents-with-arangoimp/
WhatsNewin2.6
597
Miscellaneouschanges
SomeLinux-basedArangoDBpackagesarenowusingtcmallocformemoryallocator.
UpgradedICUlibrarytoversion54.Thisincreasesperformanceinmanyplaces.
Allowtosplitanedgeindexintobucketswhichareresizedindividually.Thedefaultvalueis1,resemblingthepre-2.6behavior.Usingmultiplebucketswillleadtotheindexentriesbeingdistributedtotheindividualbuckets,witheachbucketbeingresponsibleonlyforafractionofthetotalindexentries.Usingmultiplebucketsmayleadtomorefrequentbutmuchfasterindexbucketresizes,andisrecommendedforbiggeredgecollections.
Defaultconfigurationvalueforoption--server.backlog-sizewaschangedfrom10to64.
Defaultconfigurationvalueforoption--database.ignore-datafile-errorswaschangedfromtruetofalse
Fulltextindexcannowindextextvaluescontainedindirectsub-objectsoftheindexedattribute.
PreviousversionsofArangoDBonlyindexedtheattributevalueifitwasastring.Sub-attributesoftheindexattributewereignoredwhenfulltextindexing.
Now,iftheindexattributevalueisanobject,theobject'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.Iftheindexattributevalueisanarray,thearray'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.
Forexample,withafulltextindexpresentonthetranslationsattribute,thefollowingtextvalueswillnowbeindexed:
varc=db._create("example");
c.ensureFulltextIndex("translations");
c.insert({translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}});
c.insert({translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"});
c.insert({translations:["ArangoDB","document","database","Foxx"]});
c.fulltext("translations","лиса").toArray();//returnsonlyfirstdocument
c.fulltext("translations","Fox").toArray();//returnsfirstandseconddocuments
c.fulltext("translations","prefix:Fox").toArray();//returnsallthreedocuments
Addedconfigurationoption--server.foxx-queues-poll-interval
ThisstartupoptioncontrolsthefrequencywithwhichtheFoxxqueuesmanagerischeckingthequeue(orqueues)forjobstobeexecuted.
Thedefaultvalueis1second.Loweringthisvaluewillresultinthequeuemanagerwakingupandcheckingthequeuesmorefrequently,whichmayincreaseCPUusageoftheserver.WhennotusingFoxxqueues,thisvaluecanberaisedtosavesomeCPUtime.
Addedconfigurationoption--server.foxx-queues
ThisstartupoptioncontrolswhethertheFoxxqueuemanagerwillcheckqueueandjobentriesinthe_systemdatabaseonly.RestrictingtheFoxxqueuemanagertothe_systemdatabasewillleadtothequeuemanagerhavingtocheckonlythequeuescollectionofasingledatabase,whereasmakingitcheckthequeuesofalldatabasesmightresultinmoreworktobedoneandmoreCPUtimetobeusedbythequeuemanager.
WhatsNewin2.6
598
IncompatiblechangesinArangoDB2.6ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.6,andadjustanyclientprogramsifnecessary.
Requirements
ArangoDB'sbuilt-inwebinterfacenowusescookiesforsessionmanagement.Sessioninformationidsarestoredincookies,soclientsusingthewebinterfacemustacceptcookiesinordertologinanduseit.
Foxxchanges
FoxxQueues
FoxxQueuejobtypedefinitionswerepreviouslybasedonfunctionsandhadtoberegisteredbeforeuse.Duetochangesin2.5thisresultedinproblemswhenrestartingtheserverordefiningjobtypesincorrectly.
Function-basedjobtypeshavebeendeprecatedin2.6andwillberemovedentirelyin2.7.
Inordertoconvertexistingfunction-basedjobtypestothenewscript-basedjobtypes,createcustomscriptsinyourFoxxappandreferencethembytheirnameandthemountpointoftheapptheyaredefinedin.OfficialjobtypesfromtheFoxxappstorecanbeupgradedbyupgradingfromthe1.xversiontothe2.xversionofthesameapp.
Inordertoupgradequeuedjobstothenewjobtypes,youneedtoupdatethetypepropertyoftheaffectedjobsinthedatabase's_jobssystemcollection.Inordertoseethecollectioninthewebinterfaceyouneedtoenablethecollectiontype"System"inthecollectionlistoptions.
Example:
Before:"type":"mailer.postmark"
After:"type":{"name":"mailer","mount":"/my-postmark-mailer"}
FoxxSessions
Theoptionsjwtandtypeofthecontrollermethodcontroller.activateSessionshavebeendeprecatedin2.6andwillberemovedentirelyin2.7.
IfyouwanttousepureJWTsessions,youcanusethesessions-jwtFoxxappfromtheFoxxappstore.
IfyouwanttouseyourownJWT-basedsessions,youcanusetheJWTfunctionsinthecryptomoduledirectly.
Insteadofusingthetypeoptionyoucanjustusethecookieandheaderoptionsontheirown,whichbothnowacceptthevaluetruetoenablethemwiththeirdefaultconfigurations.
TheoptionsessionStorageApphasbeenrenamedtosessionStorageandnowalsoacceptssessionstoragesdirectly.TheoldoptionsessionStorageAppwillberemovedentirelyin2.7.
Libraries
ThebundledversionofthejoilibraryusedinFoxxwasupgradedtoversion6.0.8.ThismayaffectFoxxapplicationsthatdependonthelibrary.
AQLchanges
AQLLENGTHfunction
Incompatiblechangesin2.6
599
ThereturnvalueoftheAQLLENGTHfunctionwaschangedifLENGTHisappliedonnullorabooleanvalue:
LENGTH(null)nowreturns0.InpreviousversionsofArangoDB,thisreturned4.
LENGTH(false)nowreturns0.InpreviousversionsofArangoDB,thereturnvaluewas5.
LENGTH(true)nowreturns1.InpreviousversionsofArangoDB,thereturnvaluewas4.
AQLgraphfunctions
In2.6thegraphfunctionsdidundergoaperformancelifting.Duringthisprocesswehadtoadopttheresultformatandtheoptionsforsomeofthem.ManygraphfunctionsnowhaveanoptionincludeDatawhichallowstotriggeriftheresultofthisfunctionshouldcontainfullyextracteddocumentsincludeData:trueoronlythe_idvaluesincludeData:false.Inmostusecasesthe_idissufficienttocontinueandtheextractionofdataisanunnecessaryoperation.TheAQLfunctionssupportingthisadditionaloptionare:
SHORTEST_PATHNEIGHBORSGRAPH_SHORTEST_PATHGRAPH_NEIGHBORSGRAPH_EDGES
FurthermoretheresultSHORTEST_PATHhaschanged.Theoldformatreturnedalistofallverticesonthepath.Optionallyitcouldincludeeachsub-pathforthesevertices.Allofthedocumentswerefullyextracted.Example:
[
{
vertex:{
_id:"vertex/1",
_key:"1",
_rev:"1234"
name:"Alice"
},
path:{
vertices:[
{
_id:"vertex/1",
_key:"1",
_rev:"1234"
name:"Alice"
}
],
edges:[]
}
},
{
vertex:{
_id:"vertex/2",
_key:"2",
_rev:"5678"
name:"Bob"
},
path:{
vertices:[
{
_id:"vertex/1",
_key:"1",
_rev:"1234"
name:"Alice"
},{
_id:"vertex/2",
_key:"2",
_rev:"5678"
name:"Bob"
}
],
edges:[
{
_id:"edge/1",
_key:"1",
_rev:"9876",
type:"loves"
Incompatiblechangesin2.6
600
}
]
}
}
]
Thenewversionismorecompact.EachSHORTEST_PATHwillonlyreturnonedocumenthavingtheattributesvertices,edges,distance.Thedistanceiscomputedtakingintoaccountthegivenweight.OptionallythedocumentscanbeextractedwithincludeData:trueExample:
{
vertices:[
"vertex/1",
"vertex/2"
],
edges:[
"edge/1"
],
distance:1
}
ThenextfunctionthatreturnsadifferentformatisNEIGHBORS.Since2.5itreturnedanobjectwithedgeandvertexforeachconnectededge.Example:
[
{
vertex:{
_id:"vertex/2",
_key:"2",
_rev:"5678"
name:"Bob"
},
edge:{
_id:"edge/1",
_key:"1",
_rev:"9876",
type:"loves"
}
}
]
With2.6itwillonlyreturnthevertexdirectly,againusingincludeData:true.Bydefaultitwillreturnadistinctsetofneighbors,usingtheoptiondistinct:falsewillincludethesamevertexforeachedgepointingtoit.
Example:
[
"vertex/2"
]
FunctionandAPIchanges
Graphmeasurementsfunctions
AllgraphmeasurementsfunctionsinJavaScriptmodulegeneral-graphthatcalculatedasinglefigurepreviouslyreturnedanarraycontainingjustthefigure.Nowthesefunctionswillreturnthefiguredirectlyandnotputitinsideanarray.
Theaffectedfunctionsare:
graph._absoluteEccentricity
graph._eccentricity
graph._absoluteCloseness
graph._closeness
graph._absoluteBetweenness
Incompatiblechangesin2.6
601
graph._betweenness
graph._radius
graph._diameter
Clientprogramscallingthesefunctionsshouldbeadjustedsotheyprocessthescalarvaluereturnedbythefunctioninsteadofthepreviousarrayvalue.
CursorAPI
AbatchSizevalue0isnowdisallowedwhencallingthecursorAPIviaHTTPPOST/_api/cursor.
TheHTTPRESTAPIPOST/_api/cursordoesnotacceptabatchSizeparametervalueof0anylonger.Abatchsizeof0nevermademuchsense,butpreviousversionsofArangoDBdidnotcheckforthisvalue.NowcreatingacursorusingabatchSizevalue0willresultinanHTTP400errorresponse.
DocumentURLsreturned
TheRESTAPImethodGET/_api/document?collection=...(thatmethodwillreturnpartialURLstoalldocumentsinthecollection)willnowproperlyprefixdocumentaddressURLswiththecurrentdatabasename.
PreviousversionsofArangoDBreturnedtheURLsstartingwith/_api/butwithoutthecurrentdatabasename,e.g./_api/document/mycollection/mykey.Startingwith2.6,theresponseURLswillincludethedatabasenameaswell,e.g./_db/_system/_api/document/mycollection/mykey.
Fulltextindexing
Fulltextindexeswillnowalsoindextextvaluescontainedindirectsub-objectsoftheindexedattribute.
PreviousversionsofArangoDBonlyindexedtheattributevalueifitwasastring.Sub-attributesoftheindexattributewereignoredwhenfulltextindexing.
Now,iftheindexattributevalueisanobject,theobject'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.Iftheindexattributevalueisanarray,thearray'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.
Deprecatedserverfunctionality
Simplequeries
Thefollowingsimplequeryfunctionsarenowdeprecated:
collection.nearcollection.withincollection.geocollection.fulltextcollection.rangecollection.closedRange
ThisalsoleadtothefollowingRESTAPImethodsbeingdeprecatedfromnowon:
PUT/_api/simple/nearPUT/_api/simple/withinPUT/_api/simple/fulltextPUT/_api/simple/range
ItisrecommendedtoreplacecallstothesefunctionsorAPIswithequivalentAQLqueries,whicharemoreflexiblebecausetheycanbecombinedwithotheroperations:
FORdocINNEAR(@@collection,@latitude,@longitude,@limit)
RETURNdoc
FORdocINWITHIN(@@collection,@latitude,@longitude,@radius,@distanceAttributeName)
RETURNdoc
Incompatiblechangesin2.6
602
FORdocINFULLTEXT(@@collection,@attributeName,@queryString,@limit)
RETURNdoc
FORdocIN@@collection
FILTERdoc.value>=@left&&doc.value<@right
LIMIT@skip,@limit
RETURNdoc`
TheabovesimplequeryfunctionsandRESTAPImethodsmayberemovedinfutureversionsofArangoDB.
UsingnegativevaluesforSimpleQuery.skip()isalsodeprecated.ThisfunctionalitywillberemovedinfutureversionsofArangoDB.
AQLfunctions
TheAQLSKIPLISTfunctionhasbeendeprecatedbecauseitisobsolete.
ThefunctionwasintroducedinolderversionsofArangoDBwithalesspowerfulqueryoptimizertoretrievedatafromaskiplistindexusingaLIMITclause.
Since2.3thesamegoalcanbeachievedbyusingregularAQLconstructs,e.g.
FORdocIN@@collection
FILTERdoc.value>=@value
SORTdoc.value
LIMIT1
RETURNdoc
Startupoptionchanges
Optionsadded
Thefollowingconfigurationoptionshavebeenaddedin2.6:
--server.session-timeout:allowscontrollingthetimeoutofusersessionsinthewebinterface.Thevalueisspecifiedinseconds.
--server.foxx-queues:controlswhethertheFoxxqueuemanagerwillcheckqueueandjobentries.DisablingthisoptioncanreduceserverloadbutwillpreventjobsaddedtoFoxxqueuesfrombeingprocessedatall.
Thedefaultvalueistrue,enablingtheFoxxqueuesfeature.
--server.foxx-queues-poll-interval:allowsadjustingthefrequencywithwhichtheFoxxqueuesmanagerischeckingthequeue(orqueues)forjobstobeexecuted.
Thedefaultvalueis1second.Loweringthisvaluewillresultinthequeuemanagerwakingupandcheckingthequeuesmorefrequently,whichmayincreaseCPUusageoftheserver.
Note:thisoptiononlyhasaneffectwhen--server.foxx-queuesisnotsettofalse.
Optionsremoved
Thefollowingconfigurationoptionshavebeenremovedin2.6.:
--log.severity:thedocsfor--log.severitymentionedlotsofseverities(e.g.exception,technical,functional,development)butonlyafewseverities(e.g.all,human)wereactuallyused,withhumanbeingthedefaultandallenablingtheadditionalloggingofincomingrequests.
Theoptionpretendedtocontrolalotofthingswhichitactuallydidn't.Additionally,theoption--log.requests-filewasaroundforalongtimealready,alsocontrollingrequestlogging.
Becausethe--log.severityoptioneffectivelydidnotcontrolthatmuch,itwasremoved.Asideeffectofremovingtheoptionisthat2.5installationsstartedwithoption--log.severityallwillnotlogrequestsaftertheupgradeto2.6.Thiscanbeadjustedbysettingthe--log.requests-fileoptioninstead.
Incompatiblechangesin2.6
603
Defaultvalueschanged
Thedefaultvaluesforthefollowingoptionshavechangedin2.6:
--database.ignore-datafile-errors:thedefaultvalueforthisoptionwaschangedfromtruetofalse.
Ifthenewdefaultvalueoffalseisused,thenarangodwillrefuseloadingcollectionsthatcontaindatafileswithCRCmismatchesorothererrors.Acollectionwithdatafileerrorswillthenbecomeunavailable.Thispreventsfollowuperrorsfromhappening.
Theonlywaytoaccesssuchcollectionistousethedatafiledebugger(arango-dfdb)andtrytorepairortruncatethedatafilewithit.
--server.request-timeout:thedefaultvaluewasincreasedfrom300to1200secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).
--server.connect-timeout:thedefaultvaluewasincreasedfrom3to5secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).
Incompatiblechangesin2.6
604
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.5.ArangoDB2.5alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.
V8versionupgrade
Thebuilt-inversionofV8hasbeenupgradedfrom3.29.54to3.31.74.1.ThisallowsactivatingadditionalES6(alsodubbedHarmonyorES.next)featuresinArangoDB,bothintheArangoShellandtheArangoDBserver.Theycanbeusedforscriptingandinserver-sideactionssuchasFoxxroutes,traversalsetc.
ThefollowingadditionalES6featuresbecomeavailableinArangoDB2.5bydefault:
iteratorsandgeneratorstemplatestringsenhancedobjectliteralsenhancednumericliteralsblockscopingwithletandconstantvariablesusingconst(note:constantvariablesrequireusingstrictmode,too)additionalstringmethods(suchasstartsWith,repeatetc.)
Indeximprovements
Sparsehashandskiplistindexes
Hashandskiplistindexescanoptionallybemadesparse.Sparseindexesexcludedocumentsinwhichatleastoneoftheindexattributesiseithernotsetorhasavalueofnull.
Assuchdocumentsareexcludedfromsparseindexes,theymaycontainfewerdocumentsthantheirnon-sparsecounterparts.Thisenablesfasterindexingandcanleadtoreducedmemoryusageincasetheindexedattributedoesoccuronlyinsome,butnotalldocumentsofthecollection.Sparseindexeswillalsoreducethenumberofcollisionsinnon-uniquehashindexesincasenon-existingoroptionalattributesareindexed.
Inordertocreateasparseindex,anobjectwiththeattributesparsecanbeaddedtotheindexcreationcommands:
db.collection.ensureHashIndex(attributeName,{sparse:true});
db.collection.ensureHashIndex(attributeName1,attributeName2,{sparse:true});
db.collection.ensureUniqueConstraint(attributeName,{sparse:true});
db.collection.ensureUniqueConstraint(attributeName1,attributeName2,{sparse:true});
db.collection.ensureSkiplist(attributeName,{sparse:true});
db.collection.ensureSkiplist(attributeName1,attributeName2,{sparse:true});
db.collection.ensureUniqueSkiplist(attributeName,{sparse:true});
db.collection.ensureUniqueSkiplist(attributeName1,attributeName2,{sparse:true});
Notethatinplaceoftheabovespecializedindexcreationcommands,itisrecommendedtousethemoregeneralindexcreationcommandensureIndex:
db.collection.ensureIndex({type:"hash",sparse:true,unique:true,fields:[attributeName]});
db.collection.ensureIndex({type:"skiplist",sparse:false,unique:false,fields:["a","b"]});
Whennotexplicitlyset,thesparseattributedefaultstofalsefornewhashorskiplistindexes.
Thiscausesachangeinbehaviorwhencreatingauniquehashindexwithoutspecifyingthesparseflag:in2.4,uniquehashindexeswereimplicitlysparse,alwaysexcludingnullvalues.Therewasnooptiontocontrolthisbehavior,andsparsitywasneithersupportedfornon-uniquehashindexesnorskiplistsin2.4.Thisimplicitsparsityofuniquehashindexeswasconsideredaninconsistency,andthereforethebehaviorwascleanedupin2.5.Asof2.5,indexeswillonlybecreatedsparseifsparsityisexplicitlyrequested.Existinguniquehashindexesfrom2.4orbeforewillautomaticallybemigratedsotheyarestillsparseaftertheupgradeto2.5.
WhatsNewin2.5
605
Geoindexesareimplicitlysparse,meaningdocumentswithouttheindexedlocationattributeorcontaininginvalidlocationcoordinatevalueswillbeexcludedfromtheindexautomatically.Thisisalsoachangewhencomparedtopre-2.5behavior,whendocumentswithmissingorinvalidcoordinatevaluesmayhavecausederrorsoninsertionwhenthegeoindex'uniqueflagwassetanditsignoreNullflagwasnot.Thiswasconfusingandhasbeenrectifiedin2.5.ThemethodensureGeoConstraint()nowdoesthesameasensureGeoIndex().Furthermore,theattributesconstraint,unique,ignoreNullandsparseflagsarenowcompletelyignoredwhencreatinggeoindexes.
Thesameistrueforfulltextindexes.Thereisnoneedtospecifynon-uniquenessorsparsityforgeoorfulltextindexes.
Assparseindexesmayexcludesomedocuments,theycannotbeusedforeverytypeofquery.Sparsehashindexescannotbeusedtofinddocumentsforwhichatleastoneoftheindexedattributeshasavalueofnull.Forexample,thefollowingAQLquerycannotuseasparseindex,evenifonewascreatedonattributeattr:
FORdocIncollection
FILTERdoc.attr==null
RETURNdoc
Ifthelookupvalueisnon-constant,asparseindexmayormaynotbeused,dependingontheothertypesofconditionsinthequery.Iftheoptimizercansafelydeterminethatthelookupvaluecannotbenull,asparseindexmaybeused.Whenuncertain,theoptimizerwillnotmakeuseofasparseindexinaqueryinordertoproducecorrectresults.
Forexample,thefollowingqueriescannotuseasparseindexonattrbecausetheoptimizerwillnotknowbeforehandwhetherthecomparisonvaluesfordoc.attrwillincludenull:
FORdocIncollection
FILTERdoc.attr==SOME_FUNCTION(...)
RETURNdoc
FORotherINotherCollection
FORdocIncollection
FILTERdoc.attr==other.attr
RETURNdoc
Sparseskiplistindexescanbeusedforsortingiftheoptimizercansafelydetectthattheindexrangedoesnotincludenullforanyoftheindexattributes.
Selectivityestimates
Indexesoftypeprimary,edgeandhashnowprovideselectivityestimates.ThesewillbeusedbytheAQLqueryoptimizerwhendecidingaboutindexusage.Usingselectivityestimatescanleadtofasterqueryexecutionwhenmoreselectiveindexesareused.
TheselectivityestimatesarealsoreturnedbytheGET/_api/indexRESTAPImethodinasub-attributeselectivityEstimateforeachindexthatsupportsit.Thisattributewillbeomittedforindexesthatdonotprovideselectivityestimates.Ifprovided,theselectivityestimatewillbeanumericvaluebetween0and1.
Selectivityestimateswillalsobereportedintheresultofcollection.getIndexes()forallindexesthatsupportthis.Ifnoselectivityestimatecanbedeterminedforanindex,theattributeselectivityEstimatewillbeomittedhere,too.
Thewebinterfacealsoshowsselectivityestimatesforeachindexthatsupportsthis.
Currentlythefollowingindextypescanprovideselectivityestimates:
primaryindexedgeindexhashindex(uniqueandnon-unique)
Noselectivityestimateswillbeprovidedforindexeswhenrunninginclustermode.
AQLOptimizerimprovements
Sortremoval
WhatsNewin2.5
606
TheAQLoptimizerrule"use-index-for-sort"willnowremovesortsalsoincaseanon-sortedindex(e.g.ahashindex)isusedforonlyequalitylookupsandallsortattributesarecoveredbytheequalitylookupconditions.
Forexample,inthefollowingquerytheextrasortondoc.valuewillbeoptimizedawayprovidedthereisanindexondoc.value):
FORdocINcollection
FILTERdoc.value==1
SORTdoc.value
RETURNdoc
TheAQLoptimizerrule"use-index-for-sort"nowalsoremovessortincasethesortcriteriaexcludestheleft-mostindexattributes,buttheleft-mostindexattributesareusedbytheindexforequality-onlylookups.
Forexample,inthefollowingquerywithaskiplistindexonvalue1,value2,thesortcanbeoptimizedaway:
FORdocINcollection
FILTERdoc.value1==1
SORTdoc.value2
RETURNdoc
Constantattributepropagation
ThenewAQLoptimizerrulepropagate-constant-attributeswilllookforattributesthatareequality-comparedtoaconstantvalue,andwillpropagatethecomparisonvalueintootherequalitylookups.ThisrulewillonlylookinsideFILTERconditions,andinsertconstantvaluesfoundinFILTERs,too.
Forexample,therulewillinsert42insteadofi.valueinthesecondFILTERofthefollowingquery:
FORiINc1
FORjINc2
FILTERi.value==42
FILTERj.value==i.value
RETURN1
Interleavedprocessing
TheoptimizerwillnowinspectAQLdata-modificationqueriesanddetectifthequery'sdata-modificationpartcanruninlockstepwiththedataretrievalpartofthequery,orifthedataretrievalpartmustbeexecutedandcompletedfirstbeforethedata-modificationcanstart.
Executingbothdataretrievalanddata-modificationinlockstepallowsusingmuchsmallerbuffersforintermediateresults,reducingthememoryusageofqueries.Notallqueriesareeligibleforthisoptimization,andtheoptimizerwillonlyapplytheoptimizationwhenitcansafelydetectthatthedata-modificationpartofthequerywillnotmodifydatatobefoundbytheretrievalpart.
Queryexecutionstatistics
ThefilteredattributewasaddedtoAQLqueryexecutionstatistics.ThevalueofthisattributeindicateshowmanydocumentswerefilteredbyFilterNodesintheAQLquery.NotethatIndexRangeNodescanalsofilterdocumentsbyselectingonlytherequiredrangesfromtheindex.ThefilteredvaluewillnotincludetheworkdonebyIndexRangeNodes,butonlytheworkperformedbyFilterNodes.
Languageimprovements
DynamicattributenamesinAQLobjectliterals
ThischangeallowsusingarbitraryexpressionstoconstructattributenamesinobjectliteralsspecifiedinAQLqueries.Todisambiguateexpressionsandotherunquotedattributenames,dynamicattributenamesneedtobeenclosedinbrackets([and]).
Example:
WhatsNewin2.5
607
FORiIN1..100
RETURN{[CONCAT('value-of-',i)]:i}
AQLfunctions
ThefollowingAQLfunctionswereaddedin2.5:
MD5(value):generatesanMD5hashofvalueSHA1(value):generatesanSHA1hashofvalueRANDOM_TOKEN(length):generatesarandomstringvalueofthespecifiedlength
SimplifyFoxxusage
ThankstoouruserfeedbackwelearnedthatFoxxisapowerful,yetrathercomplicatedconcept.With2.5wemadeitlesscomplicatedwhilekeepingallitsstrength.Thatincludesarewriteofthedocumentationaswellassomecodechangesasfollows:
MovedFoxxapplicationstoadifferentfolder.
Until2.4foxxappswerestoredinthefollowingfolderstructure:<app-path>/databases/<dbname>/<appname>:<appversion>.Thiscausedsometroubleasappswherecachedbasedonnameandversionandupdatesdidnotapply.Alsothepathonfilesystemandtheapp'saccessURLhadnorelationtooneanother.NowthepathonfilesystemisidenticaltotheURL(excepttheappendedAPP):<app-path>/_db/<dbname>/<mointpoint>/APP
RewriteofFoxxrouting
TheroutingofFoxxhasbeenexposedtomajorinternalchangesweadjustedbecauseofuserfeedback.Thisallowsustosetthedevelopmentmodepermountpointwithouthavingtochangepathsandholdappsatseparatelocations.
FoxxDevelopmentmode
Thedevelopmentmodeuseduntil2.4isgone.Ithasbeenreplacedbyamuchmorematureversion.Thisincludesthedeprecationofthejavascript.dev-app-pathparameter,whichisuselesssince2.5.Insteadofhavingtwoseparateappdirectoriesforproductionanddevelopment,appsnowresideinoneplace,whichisusedforproductionaswellasfordevelopment.Appscanstillbeputintodevelopmentmode,changingtheirbehaviorcomparedtoproductionmode.Developmentmodeappsarestillrereadfromdiskateveryrequest,andstilltheyshipmoredebugoutput.
Thischangehasalsomadethestartupoptions--javascript.frontend-development-modeand--javascript.dev-app-pathobsolete.Theformeroptionwillnothaveanyeffectwhenset,andthelatteroptionisonlyreadandusedduringtheupgradeto2.5anddoesnothaveanyeffectslater.
Foxxinstallprocess
InstallingFoxxappshasbeenatwostepprocess:importthemintoArangoDBandmountthemataspecificmountpoint.Theseoperationshavebeenjoinedtogether.Youcaninstallanappatonemountpoint,that'sit.Nofetch,mount,unmount,purgecycleanymore.Thecommandshavebeensimplifiedtojust:
install:getyourFoxxappupandrunninguninstall:shutitdownanderaseitfromdisk
Foxxerroroutput
Until2.4theerrorsproducedbyFoxxwerenotoptimal.Often,theerrormessagewasjustunabletoparsemanifestandcontainedonlyaninternalstacktrace.In2.5wemademajorimprovementsthere,includingamuchmorefinegrainederroroutputthathelpsyoudebugyourFoxxapps.Theerrormessageprintedisnowmuchclosertoitssourceandshouldhelpyoutrackitdown.
AlsoweaddedthedefaulthandlersforunhandlederrorsinFoxxapps:
YouwillgetaniceinternalerrorpagewheneveryourFoxxappiscalledbutwasnotinstalledduetoanyerror
WhatsNewin2.5
608
Youwillgetapropererrormessagewhenhavinganuncaughterrorappearsinanyapproute
InproductionmodethemessagesabovewillNOTcontainanyinformationaboutyourFoxxinternalsandaresafetobeexposedtothirdpartyusers.Indevelopmentmodethemessagesabovewillcontainthestacktrace(ifavailable),makingiteasierforyourin-housedevstotrackdownerrorsintheapplication.
Foxxconsole
WeaddedaconsoleobjecttoFoxxapps.AllFoxxappsnowhaveaconsoleobjectimplementingthefamiliarConsoleAPIintheirglobalscope,whichcanbeusedtologdiagnosticmessagestothedatabase.Thisconsolealsoallowstoreadtheerroroutputofonespecificfoxx.
Foxxrequests
Weaddedorg/arangodb/requestmodule,whichprovidesasimpleAPIformakingHTTPrequeststoexternalservices.ThisisenablesFoxxtobedirectlypartofamicroservicearchitecture.
WhatsNewin2.5
609
IncompatiblechangesinArangoDB2.5ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.5,andadjustanyclientprogramsifnecessary.
Changedbehavior
V8
TheV8versionshippedwithArangoDBwasupgradedfrom3.29.59to3.31.74.1.ThisleadstoadditionalECMAScript6(ES6or"harmony")featuresbeingenabledbydefaultinArangoDB'sscriptingenvironment.
Apartfromthat,achangeintheinterpretationofcommand-lineoptionsbyV8mayaffectusers.ArangoDBpassesthevalueofthecommand-lineoption--javascript.v8-optionstoV8andleavesinterpretationofthecontentstoV8.Forexample,theArangoDBoption--javascript.v8-options="--harmony"couldbeusedtotellV8toenableitsharmonyfeatures.
InArangoDB2.4,thefollowingharmonyoptionsweremadeavailablebyV8:
--harmony_scoping(enableharmonyblockscoping)--harmony_modules(enableharmonymodules(impliesblockscoping))--harmony_proxies(enableharmonyproxies)--harmony_generators(enableharmonygenerators)--harmony_numeric_literals(enableharmonynumericliterals(0o77,0b11))--harmony_strings(enableharmonystring)--harmony_arrays(enableharmonyarrays)--harmony_arrow_functions(enableharmonyarrowfunctions)--harmony_classes(enableharmonyclasses)--harmony_object_literals(enableharmonyobjectliteralextensions)--harmony(enableallharmonyfeatures(exceptproxies))
Therewastheoption--harmony,whichturnedonalmostallharmonyfeatures.
InArangoDB2.5,V8providesthefollowingharmony-relatedoptions:
--harmony(enableallcompletedharmonyfeatures)--harmony_shipping(enableallshippedharmonyfeatures)--harmony_modules(enable"harmonymodules(impliesblockscoping)"(inprogress))--harmony_arrays(enable"harmonyarraymethods"(inprogress))--harmony_array_includes(enable"harmonyArray.prototype.includes"(inprogress))--harmony_regexps(enable"harmonyregularexpressionextensions"(inprogress))--harmony_arrow_functions(enable"harmonyarrowfunctions"(inprogress))--harmony_proxies(enable"harmonyproxies"(inprogress))--harmony_sloppy(enable"harmonyfeaturesinsloppymode"(inprogress))--harmony_unicode(enable"harmonyunicodeescapes"(inprogress))--harmony_tostring(enable"harmonytoString")--harmony_numeric_literals(enable"harmonynumericliterals")--harmony_strings(enable"harmonystringmethods")--harmony_scoping(enable"harmonyblockscoping")--harmony_classes(enable"harmonyclasses(impliesblockscoping&objectliteralextension)")--harmony_object_literals(enable"harmonyobjectliteralextensions")--harmony_templates(enable"harmonytemplateliterals")
Notethatthereareextraoptionsforbettercontrollingthededicatedfeatures,andespeciallythatthemeaningofthe--harmonyoptionhaschangedfromenablingallharmonyfeaturestoallcompletedharmonyfeatures!
Usersshouldadjustthevalueof--javascript.v8-optionsaccordingly.
PleasenotethatincompleteharmonyfeaturesaresubjecttochangeinfutureV8releases.
Incompatiblechangesin2.5
610
Sparseindexes
Hashindexesandskiplistindexescannowbecreatedinasparsevariant.Whennotexplicitlyset,thesparseattributedefaultstofalsefornewindexes.
Thiscausesachangeinbehaviorwhencreatingauniquehashindexwithoutspecifyingthesparseflag.Theuniquehashindexwillbecreatedinanon-sparsevariantinArangoDB2.5.
In2.4andbefore,uniquehashindexeswereimplicitlysparse,alwaysexcludingnullvaluesfromtheindex.Therewasnooptiontocontrolthisbehavior,andsparsitywasneithersupportedfornon-uniquehashindexesnorskiplistsin2.4.Thisimplicitsparsityofjustuniquehashindexeswasconsideredaninconsistency,andthereforethebehaviorwascleanedupin2.5.
Asof2.5,hashandskiplistindexeswillonlybecreatedsparseifsparsityisexplicitlyrequested.Thismayrequireachangeinindex-creatingclientcode,butonlyiftheclientcodecreatesuniquehashindexesandiftheyarestillintendedtobesparse.Inthiscase,theclientcodeshouldexplicitlysetthesparseflagtotruewhencreatingauniquehashindex.
Existinguniquehashindexesfrom2.4orbeforewillautomaticallybemigratedsotheyarestillsparseaftertheupgradeto2.5.Fortheseindexes,thesparseattributewillbepopulatedautomaticallywithavalueoftrue.
Geoindexesareimplicitlysparse,meaningdocumentswithouttheindexedlocationattributeorcontaininginvalidlocationcoordinatevalueswillbeexcludedfromtheindexautomatically.Thisisalsoachangewhencomparedtopre-2.5behavior,whendocumentswithmissingorinvalidcoordinatevaluesmayhavecausederrorsoninsertionwhenthegeoindex'uniqueflagwassetanditsignoreNullflagwasnot.
Thiswasconfusingandhasbeenrectifiedin2.5.ThemethodensureGeoConstraint()nowdoesthesameasensureGeoIndex().Furthermore,theattributesconstraint,unique,ignoreNullandsparseflagsarenowcompletelyignoredwhencreatinggeoindexes.ClientindexcreationcodethereforedoesnotneedtosettheignoreNullorconstraintattributeswhencreatingageoindex.
Thesameistrueforfulltextindexes.Thereisnoneedtospecifynon-uniquenessorsparsityforgeoorfulltextindexes.Theywillalwaysbenon-uniqueandsparse.
MovedFoxxapplicationstoadifferentfolder.
Until2.4foxxappswerestoredinthefollowingfolderstructure:<app-path>/databases/<dbname>/<appname>:<appversion>.Thiscausedsometroubleasappswherecachedbasedonnameandversionandupdatesdidnotapply.Alsothepathonfilesystemandtheapp'saccessURLhadnorelationtooneanother.NowthepathonfilesystemisidenticaltotheURL(excepttheappendedAPP):<app-path>/_db/<dbname>/<mointpoint>/APP
FoxxDevelopmentmode
Thedevelopmentmodeuseduntil2.4isgone.Ithasbeenreplacedbyamuchmorematureversion.Thisincludesthedeprecationofthejavascript.dev-app-pathparameter,whichisuselesssince2.5.Insteadofhavingtwoseparateappdirectoriesforproductionanddevelopment,appsnowresideinoneplace,whichisusedforproductionaswellasfordevelopment.Appscanstillbeputintodevelopmentmode,changingtheirbehaviorcomparedtoproductionmode.Developmentmodeappsarestillrereadfromdiskateveryrequest,andstilltheyshipmoredebugoutput.
Thischangehasalsomadethestartupoptions--javascript.frontend-development-modeand--javascript.dev-app-pathobsolete.Theformeroptionwillnothaveanyeffectwhenset,andthelatteroptionisonlyreadandusedduringtheupgradeto2.5anddoesnothaveanyeffectslater.
Foxxinstallprocess
InstallingFoxxappshasbeenatwostepprocess:importthemintoArangoDBandmountthemataspecificmountpoint.Theseoperationshavebeenjoinedtogether.Youcaninstallanappatonemountpoint,that'sit.Nofetch,mount,unmount,purgecycleanymore.Thecommandshavebeensimplifiedtojust:
install:getyourFoxxappupandrunninguninstall:shutitdownanderaseitfromdiskDeprecatedfeatures
Foxx:methodModel#toJSONSchema(id)isdeprecated,itwillraiseawarningifyouuseit.PleaseuseFoxx.toJSONSchema(id,model)
Incompatiblechangesin2.5
611
instead.
Removedfeatures
Startupswitch--javascript.frontend-development-mode:Itsmajorpurposewasinternaldevelopmentanyway.Nowthewebfrontendcanbesettodevelopmentmodesimilartoanyotherfoxxapp.Startupswitch--javascript.dev-app-path:WasusedforthedevelopmentmodeofFoxx.Thisisintegratedwiththenormalapp-pathnowandcanbetriggeredonapplevel.Thesecondapp-pathissuperfluous.Foxx:controller.collection:PleaseuseappContext.collectioninstead.Foxx:FoxxRepository.modelPrototype:PleaseuseFoxxRepository.modelinstead.Foxx:Model.extend({},{attributes:{}}):PleaseuseModel.extend({schema:{}})instead.Foxx:requestContext.bodyParam(paramName,description,Model):PleaseuserequestContext.bodyParam(paramName,options)instead.Foxx:requestContext.queryParam({type:string}):PleaseuserequestContext.queryParam({type:joi})instead.Foxx:requestContext.pathParam({type:string}):PleaseuserequestContext.pathParam({type:joi})instead.Graph:Themodulesorg/arangodb/graphandorg/arangodb/graph-blueprint:Pleaseusemoduleorg/arangodb/general-graphinstead.NOTE:Thisdoesnotmeanwedonotsupportblueprintsanymore.Generalgraphcoverseverythingthegraph--blueprintdid,plusmanymorefeatures.General-Graph:Inthemoduleorg/arangodb/general-graphthefunctions_undirectedRelationand_directedRelationarenolongeravailable.Bothfunctionshavebeenunifiedto_relation.
Incompatiblechangesin2.5
612
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.4.ArangoDB2.4alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.
V8versionupgrade
Thebuilt-inversionofV8hasbeenupgradedfrom3.16.14to3.29.59.ThisactivatesseveralES6(alsodubbedHarmonyorES.next)featuresinArangoDB,bothintheArangoShellandtheArangoDBserver.Theycanbeusedforscriptingandinserver-sideactionssuchasFoxxroutes,traversalsetc.
ThefollowingES6featuresareavailableinArangoDB2.4bydefault:
iteratorstheofoperatorsymbolspredefinedcollectionstypes(Map,Setetc.)typedarrays
ManyotherES6featuresaredisabledbydefault,butcanbemadeavailablebystartingarangodorarangoshwiththeappropriateoptions:
arrowfunctionsproxiesgeneratorsString,Array,andNumberenhancementsconstantsenhancedobjectandnumericliterals
ToactivatealltheseES6featuresinarangodorarangosh,startitwiththefollowingoptions:
arangosh--javascript.v8-options="--harmony--harmony_generators"
MoredetailsontheavailableES6featurescanbefoundinthisblog.
FoxxGeneratorArangoDB2.4isshippedwithFoxxGenerator,aframeworkforbuildingstandardizedHypermediaAPIseasily.ThegeneratedAPIscanbeconsumedwithclienttoolsthatunderstandSiren.
HypermediaisthesimpleideathatourHTTPAPIsshouldhavelinksbetweentheirendpointsinthesamewaythatourwebsiteshavelinksbetweenthem.FoxxGeneratorisbasedontheideathatyoucanrepresentanAPIasastatechart:Everyendpointisastateandthelinksarethetransitionsbetweenthem.Usingyourdescriptionofstatesandtransitions,itcanthencreateanAPIforyou.
TheFoxxGeneratorcancreateAPIsbasedonasemanticdescriptionofentitiesandtransitions.AblogseriesontheusecasesandhowtousetheFoxxgeneratorishere:
part1part2part3
AcookbookrecipeforgettingstartedwithFoxxGeneratorishere.
AQLimprovements
Optimizerimprovements
WhatsNewin2.4
613
TheAQLoptimizerhasbeenenhancedtouseofindexesinqueriesinseveraladditionalcases.FilterscontainingtheINoperatorcannowmakeuseofindexes,andmultipleOR-orAND-combinedfilterconditionscannowalsouseindexesifthefilterconditionsrefertothesameindexedattribute.
Hereareafewexamplesofqueriesthatcannowuseindexesbutcouldn'tbefore:
FORdocINcollection
FILTERdoc.indexedAttribute==1||doc.indexedAttribute>99
RETURNdoc
FORdocINcollection
FILTERdoc.indexedAttributeIN[3,42]||doc.indexedAttribute>99
RETURNdoc
FORdocINcollection
FILTER(doc.indexedAttribute>2&&doc.indexedAttribute<10)||
(doc.indexedAttribute>23&&doc.indexedAttribute<42)
RETURNdoc
Additionally,theoptimizerruleremove-filter-covered-by-indexhasbeenadded.ThisruleremovesFilterNodesandCalculationNodesfromanexecutionplanifthefilterconditionisalreadycoveredbyapreviousIndexRangeNode.Removingthefilter'sCalculationNodeandtheFilterNodeitselfwillspeedupqueryexecutionbecausethequeryrequireslesscomputation.
Furthermore,thenewoptimizerruleremove-sort-randwillremoveaSORTRAND()statementandmovetherandomiterationintotheappropriateEnumerateCollectionNode.Thisisusuallymoreefficientthanindividuallyenumeratingandsorting.
Data-modificationqueriesreturningdocuments
INSERT,REMOVE,UPDATEorREPLACEqueriesnowcanoptionallyreturnthedocumentsinserted,removed,updated,orreplaced.Thisishelpfulfortrackingtheauto-generatedattributes(e.g._key,_rev)createdbyanINSERTandinalotofothersituations.
Inordertoreturndocumentsfromadata-modificationquery,thestatementmustimmediatelybeimmediatelyfollowedbyaLETstatementthatassignseitherthepseudo-valueNEWorOLDtoavariable.ThisLETstatementmustbefollowedbyaRETURNstatementthatreturnsthevariableintroducedbyLET:
FORiIN1..100
INSERT{value:i}INtestLETinserted=NEWRETURNinserted
FORuINusers
FILTERu.status=='deleted'
REMOVEuINusersLETremoved=OLDRETURNremoved
FORuINusers
FILTERu.status=='notactive'
UPDATEuWITH{status:'inactive'}INusersLETupdated=NEWRETURNupdated
NEWreferstotheinsertedormodifieddocumentrevision,andOLDreferstothedocumentrevisionbeforeupdateorremoval.INSERTstatementscanonlyrefertotheNEWpseudo-value,andREMOVEoperationsonlytoOLD.UPDATEandREPLACEcanrefertoeither.
Inallcasesthefulldocumentswillbereturnedwithalltheirattributes,includingthepotentiallyauto-generatedattributessuchas_id,_key,or_revandtheattributesnotspecifiedintheupdateexpressionofapartialupdate.
Languageimprovements
COUNTclause
AnoptionalCOUNTclausewasaddedtotheCOLLECTstatement.TheCOUNTclauseallowsformoreefficientcountingofvalues.
InpreviousversionsofArangoDBonehadtowritethefollowingtocountdocuments:
RETURNLENGTH(
FORdocINcollection
FILTER...somecondition...
RETURNdoc
)
WhatsNewin2.4
614
WiththeCOUNTclause,thequerycanbemodifiedto
FORdocINcollection
FILTER...somecondition...
COLLECTWITHCOUNTINTOlength
RETURNlength
ThelatterquerywillbemuchmoreefficientbecauseitwillnotproduceanyintermediateresultswithneedtobeshippedfromasubqueryintotheLENGTHfunction.
TheCOUNTclausecanalsobeusedtocountthenumberofitemsineachgroup:
FORdocINcollection
FILTER...somecondition...
COLLECTgroup=doc.groupWITHCOUNTINTOlength
return{group:group,length:length}
COLLECTmodifications
InArangoDB2.4,COLLECToperationscanbemademoreefficientifonlyasmallfragmentofthegroupvaluesisneededlater.Forthesecases,COLLECTprovidesanoptionalconversionexpressionfortheINTOclause.Thisexpressioncontrolsthevaluethatisinsertedintothearrayofgroupvalues.Itcanbeusedforprojections.
ThefollowingqueryonlycopiesthedateRegisteredattributeofeachdocumentintothegroups,potentiallysavingalotofmemoryandcomputationtimecomparedtocopyingdoccompletely:
FORdocINcollection
FILTER...somecondition...
COLLECTgroup=doc.groupINTOdates=doc.dateRegistered
return{group:group,maxDate:MAX(dates)}
Comparethistothefollowingvariantofthequery,whichwastheonlywaytoachievethesameresultinpreviousversionsofArangoDB:
FORdocINcollection
FILTER...somecondition...
COLLECTgroup=doc.groupINTOdates
return{group:group,maxDate:MAX(dates[*].doc.dateRegistered)}
Theabovequerywillneedtocopythefulldocattributeintothelengthsvariable,whereasthenewvariantwillonlycopythedateRegisteredattributeofeachdoc.
Subquerysyntax
InpreviousversionsofArangoDB,subqueriesrequiredextraparenthesesaroundthem,andthiscausedconfusionwhensubquerieswereusedasfunctionparameters.Forexample,thefollowingquerydidnotwork:
LETvalues=LENGTH(
FORdocINcollectionRETURNdoc
)
buthadtobewrittenasfollows:
LETvalues=LENGTH((
FORdocINcollectionRETURNdoc
))
Thiswasunintuitiveandisfixedinversion2.4sothatbothvariantsofthequeryareacceptedandproducethesameresult.
Webinterface
WhatsNewin2.4
615
TheApplicationstabforFoxxapplicationsinthewebinterfacehasgotacompleteredesign.
ItwillnowonlyshowapplicationsthatarecurrentlyrunninginArangoDB.Foraselectedapplication,anewdetailedviewhasbeencreated.Thisviewprovidesabetteroverviewoftheapp,e.g.:
authorlicenseversioncontributorsdownloadlinksAPIdocumentation
InstallinganewFoxxapplicationontheserverismadeeasyusingthenewAddapplicationbutton.TheAddapplicationdialogprovidesallthefeaturesalreadyavailableinthefoxx-managerconsoleapplicationplussomemore:
installaFoxxapplicationfromGithubinstallaFoxxapplicationfromazipfileinstallaFoxxapplicationfromArangoDB'sapplicationstorecreateanewFoxxapplicationfromscratch:thisfeatureusesageneratortocreateaFoxxapplicationwithpre-definedCRUDmethodsforagivenlistofcollections.ThegeneratedFoxxappcaneitherbedownloadedasazipfileorbeinstalledontheserver.StartingwithanewFoxxapphasneverbeeneasier.
Miscellaneousimprovements
Defaultendpointis127.0.0.1
ThedefaultendpointfortheArangoDBserverhasbeenchangedfrom0.0.0.0to127.0.0.1.ThiswillmakenewArangoDBinstallationsunaccessiblefromclientsotherthanlocalhostunlesstheconfigurationischanged.Thisisasecurityprecautionmeasurethathasbeenrequestedasafeaturealotoftimes.
Ifyouarethedevelopmentoption--enable-relative,theendpointwillstillbe0.0.0.0.
Systemcollectionsinreplication
Bydefault,systemcollectionsarenowincludedinreplicationandallreplicationAPIreturnvalues.Thiswillleadtouseraccountsandcredentialsdatabeingreplicatedfrommastertoslaveservers.Thismayoverwriteslave-specificdatabaseusers.
Ifthisisundesired,the_userscollectioncanbeexcludedfromreplicationeasilybysettingtheincludeSystemattributetofalseinthefollowingcommands:
replication.sync({includeSystem:false});replication.applier.properties({includeSystem:false});
Thiswillexcludeallsystemcollections(including_aqlfunctions,_graphsetc.)fromtheinitialsynchronizationandthecontinuousreplication.
Ifthisisalsoundesired,itisalsopossibletospecifyalistofcollectionstoexcludefromtheinitialsynchronizationandthecontinuousreplicationusingtherestrictCollectionsattribute,e.g.:
require("org/arangodb/replication").applier.properties({
includeSystem:true,
restrictType:"exclude",
restrictCollections:["_users","_graphs","foo"]
});
WhatsNewin2.4
616
IncompatiblechangesinArangoDB2.4ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.4,andadjustanyclientprogramsifnecessary.
Changedbehavior
V8upgrade
ThebundledV8versionhasbeenupgradedfrom3.16.14to3.29.59.
Thenewversionprovidesbettererrorchecking,whichcanleadtosubtlechangesintheexecutionofJavaScriptcode.
Thefollowingcode,thoughnonsense,runswithouterrorin2.3and2.4whenstrictmodeisnotenabled:
(function(){
a=true;
a.foo=1;
})();
Whenenablingstrictmode,thefunctionwillthrowanerrorin2.4butnotin2.3:
(function(){
"usestrict";
a=true;
a.foo=1;
})();
TypeError:Cannotassigntoreadonlyproperty'foo'oftrue
Thoughthisisachangeinbehavioritcanbeconsideredanimprovement.Thenewversionactuallyuncoversanerrorthatwentundetectedintheoldversion.
Errormessageshavealsochangedslightlyinthenewversion.ApplicationsthatrelyontheexacterrormessagesoftheJavaScriptenginemayneedtobeadjustedsotheylookfortheupdatederrormessages.
Defaultendpoint
Thedefaultendpointforarangodisnow127.0.0.1.
ThischangewillmodifytheIPaddressArangoDBlistensonto127.0.0.1bydefault.ThiswillmakenewArangoDBinstallationsunaccessiblefromclientsotherthanlocalhostunlesstheconfigurationischanged.Thisisasecurityfeature.
TomakeArangoDBaccessiblefromanyclient,changetheserver'sconfiguration(--server.endpoint)toeithertcp://0.0.0.0:8529ortheserver'spubliclyvisibleIPaddress.
Replication
SystemcollectionsarenowincludedinthereplicationandallreplicationAPIreturnvaluesbydefault.
Thiswillleadtouseraccountsandcredentialsdatabeingreplicatedfrommastertoslaveservers.Thismayoverwriteslave-specificdatabaseusers.
Thismaybeconsideredafeatureorananti-feature,soitisconfigurable.
Ifreplicationofsystemcollectionsisundesired,theycanbeexcludedfromreplicationbysettingtheincludeSystemattributetofalseinthefollowingcommands:
initialsynchronization:replication.sync({includeSystem:false})continuousreplication:replication.applier.properties({includeSystem:false})
Incompatiblechangesin2.4
617
Thiswillexcludeallsystemcollections(including_aqlfunctions,_graphsetc.)fromtheinitialsynchronizationandthecontinuousreplication.
Ifthisisalsoundesired,itisalsopossibletospecifyalistofcollectionstoexcludefromtheinitialsynchronizationandthecontinuousreplicationusingtherestrictCollectionsattribute,e.g.:
require("org/arangodb/replication").applier.properties({
includeSystem:true,
restrictType:"exclude",
restrictCollections:["_users","_graphs","foo"]
});
Theaboveexamplewillingeneralincludesystemcollections,butwillexcludethespecifiedthreecollectionsfromcontinuousreplication.
TheHTTPRESTAPImethodsforfetchingthereplicationinventoryandfordumpingcollectionsalsosupporttheincludeSystemcontrolflagviaaURLparameterofthesamename.
Buildprocesschanges
Severaloptionsfortheconfigurecommandhavebeenremovedin2.4.Theoptions
--enable-all-in-one-v8
--enable-all-in-one-icu
--enable-all-in-one-libev
--with-libev=DIR
--with-libev-lib=DIR
--with-v8=DIR
--with-v8-lib=DIR
--with-icu-config=FILE
arenotavailableanymorebecausethebuildprocesswillalwaysusethebundledversionsofthelibraries.
WhenbuildingArangoDBfromsourceinadirectorythatalreadycontainedapre-2.4version,itwillbenecessarytorunamakesupercleancommandonceandafullrebuildafterwards:
gitpull
makesuperclean
makesetup
./configure<optionsgohere>
make
MiscellaneouschangesAsaconsequenceofglobalrenaminginthecodebase,theoptionmergeArrayshasbeenrenamedtomergeObjects.ThisoptioncontrolswhetherJSONobjectswillbemergedonanupdateoperationoroverwritten.Thedefaulthasbeen,andstillis,tomerge.Notspecifyingtheparameterwillleadtoamerge,asithasbeenthebehaviorinArangoDBeversince.
ThisaffectstheHTTPRESTAPImethodPATCH/_api/document/collection/key.ItsoptionalURLparametermergeArraysfortheoptionhasbeenrenamedtomergeObjects.
TheAQLUPDATEstatementisalsoaffected,asitsoptionmergeArrayshasalsobeenrenamedtomergeObjects.The2.3query
UPDATEdocINcollectionWITH{...}INcollectionOPTIONS{mergeArrays:false}
shouldthusberewrittentothefollowingin2.4:
UPDATEdocINcollectionWITH{...}INcollectionOPTIONS{mergeObjects:false}
Deprecatedfeatures
Incompatiblechangesin2.4
618
ForFoxxControllerobjects,themethodcollection()isdeprecatedandwillberemovedinfutureversionofArangoDB.Usingthismethodwillissueawarning.PleaseuseapplicationContext.collection()instead.
ForFoxxRepositoryobjects,thepropertymodelPrototypeisnowdeprecated.Usingitwillissueawarning.PleaseuseFoxxRepository.modelinstead.
InFoxxController/RequestContext,callingmethodbodyParam()withthreeargumentsisdeprecated.Pleaseuse.bodyParam(paramName,options)instead.
InFoxxController/RequestContextcallingmethodqueryParam({type:string})isdeprecated.PleaseuserequestContext.queryParam({type:joi})instead.
InFoxxController/RequestContextcallingmethodpathParam({type:string})isdeprecated.PleaseuserequestContext.pathParam({type:joi})instead.
ForFoxxModel,callingModel.extend({},{attributes:{}})isdeprecated.PleaseuseModel.extend({schema:{}})instead.
Inmoduleorg/arangodb/general-graph,thefunctions_undirectedRelation()and_directedRelation()aredeprecatedandwillberemovedinafutureversionofArangoDB.Bothfunctionshavebeenunifiedto_relation().
Themodulesorg/arangodb/graphandorg/arangodb/graph-blueprintaredeprecated.Pleaseusemoduleorg/arangodb/general-graphinstead.
TheHTTPRESTAPI_api/graphandallitsmethodsaredeprecated.PleaseusethegeneralgraphAPI_api/gharialinstead.
Removedfeatures
Thefollowingreplication-relatedJavaScriptmethodsbecameobsoleteinArangoDB2.2andhavebeenremovedinArangoDB2.4:
require("org/arangodb/replication").logger.start()
require("org/arangodb/replication").logger.stop()
require("org/arangodb/replication").logger.properties()
TheRESTAPImethodsforthesefunctionshavealsobeenremovedinArangoDB2.4:
HTTPPUT/_api/replication/logger-startHTTPPUT/_api/replication/logger-stopHTTPGET/_api/replication/logger-configHTTPPUT/_api/replication/logger-config
Clientapplicationsthatcalloneofthesemethodsshouldbeadjustedbyremovingthecallstothesemethods.Thisshouldn'tbeproblematicasthesemethodshavebeenno-opssinceArangoDB2.2anyway.
Incompatiblechangesin2.4
619
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.3.ArangoDB2.3alsocontainsseveralbugfixesthatarenotlistedhere.
AQLimprovements
Frameworkimprovements
AQLqueriesarenowsentthroughaqueryoptimizerframeworkbeforeexecution.Thequeryoptimizerframeworkwillfirstconverttheinternalrepresentationofthequery,theabstractsyntaxtree,intoaninitialexecutionplan.
Theexecutionplanisthensendthroughoptimizerrulesthatmaydirectlymodifytheplaninplaceorcreateanewvariantoftheplan.Newplansmightagainbeoptimized,allowingtheoptimizertocarryoutseveraloptimizations.
Aftercreatingplans,theoptimizerwillestimatethecostsforeachplanandpicktheplanwiththelowestcost(termedtheoptimalplan)fortheactualqueryexecution.
Withtheexplain()methodofArangoStatementuserscancheckwhichexecutionplantheoptimizerpickorretrievealistofotherplansthatoptimizerdidnotchoose.Theplanwillrevealmanydetailsaboutwhichindexesareusedetc.explain()willalsoreturntheofoptimizerrulesappliedsouserscanvalidatewhetherornotaqueryallowsusingaspecificoptimization.
ExecutionofAQLquerieshasbeenrewritteninC++,allowingmanyqueriestoavoidtheconversionofdocumentsbetweenArangoDB'sinternallow-leveldatastructureandtheV8objectrepresentationformat.
Theframeworkforoptimizerrulesisnowalsogenerallycluster-aware,allowingspecificoptimizationsforqueriesthatruninacluster.Additionally,theoptimizerwasdesignedtobeextensibleinordertoaddmoreoptimizationsinthefuture.
Languageimprovements
Alternativeoperatorsyntax
ArangoDB2.3allowstousethefollowingalternativeformsforthelogicaloperators:
AND:logicalandOR:logicalorNOT:negation
Thisnewsyntaxisjustanalternativetotheoldsyntax,allowingeasiermigrationfromSQL.Theoldsyntaxisstillfullysupportedandwillbe:
&&:logicaland||:logicalor!:negation
NOTINoperator
AQLnowhasadedicatedNOTINoperator.
Previously,aNOTINwasonlyachievablebywritinganegatedINcondition:
FORiIN...FILTER!(iIN[23,42])...
InArangoDB2.3,thesameresultcannowalternativelybeachievedbywritingthemoreintuitivevariant:
FORiIN...FILTERiNOTIN[23,42]...
Improvementsofbuilt-infunctions
WhatsNewin2.3
620
ThefollowingAQLstringfunctionshavebeenadded:
LTRIM(value,characters):left-trimsastringvalueRTRIM(value,characters):right-trimsastringvalueFIND_FIRST(value,search,start,end):findsthefirstoccurrenceofasearchstringFIND_LAST(value,search,start,end):findsthelastoccurrenceofasearchstringSPLIT(value,separator,limit):splitsastringintoanarray,usingaseparatorSUBSTITUTE(value,search,replace,limit):replacescharactersorstringsinsideanother
ThefollowingotherAQLfunctionshavebeenadded:
VALUES(document):returnsthevaluesofanobjectasanarray(thisisthecounterparttothealreadyexistingATTRIBUTESfunction)ZIP(attributes,values):returnsanobjectconstructedfromattributesandvaluespassedinseparateparametersPERCENTILE(values,n,method):returnsthenthspercentileofthevaluesprovided,usingrankorinterpolationmethod
ThealreadyexistingfunctionsCONCATandCONCAT_SEPARATORnowsupportarrayarguments,e.g.:
/*"foobarbaz"*/
CONCAT(['foo','bar','baz'])
/*"foo,bar,baz"*/
CONCAT_SEPARATOR(",",['foo','bar','baz'])
AQLqueriesthrowlessexceptions
InpreviousversionsofArangoDB,AQLqueriesabortedwithanexceptioninmanysituationsandthrewaruntimeexception.Forexample,exceptionswerethrownwhentryingtofindavalueusingtheINoperatorinanon-arrayelement,whentryingtousenon-booleanvalueswiththelogicaloperands&&or||or!,whenusingnon-numericvaluesinarithmeticoperations,whenpassingwrongparametersintofunctionsetc.
ThefactthatmanyAQLoperatorscouldthrowexceptionsledtoalotofquestionsfromusers,andalotofmore-verbose-than-necessaryqueries.Forexample,thefollowingqueryfailedwhenthereweredocumentsthatdidnothaveatopicsattributeatall:
FORdocINmycollection
FILTERdoc.topicsIN["something","whatever"]
RETURNdoc
Thisforceduserstorewritethequeryasfollows:
FORdocINmycollection
FILTERIS_LIST(doc.topics)&&doc.topicsIN["something","whatever"]
RETURNdoc
InArangoDB2.3thishasbeenchangedtomakeAQLeasiertouse.Thechangeprovidesanextrabenefit,andthatisthatnon-throwingoperatorsallowthequeryoptimizertoperformmuchmoretransformationsinthequerywithoutchangingitsoverallresult.
Hereisasummaryofchanges:
whenanon-arrayvalueisusedontheright-handsideoftheINoperator,theresultwillbefalseinArangoDB2.3,andnoexceptionwillbethrown.thebooleanoperators&&and||donotthrowinArangoDB2.3ifanyoftheoperandsisnotabooleanvalue.Instead,theywillperformanimplicitcastofthevaluestobooleans.Theirresultwillbeasfollows:
lhs&&rhswillreturnlhsifitisfalseorwouldbefalsewhenconvertedintoaboolean.Iflhsistrueorwouldbetruewhenconvertedtoaboolean,rhswillbereturned.lhs||rhswillreturnlhsifitistrueorwouldbetruewhenconvertedintoaboolean.Iflhsisfalseorwouldbefalsewhenconvertedtoaboolean,rhswillbereturned.!valuewillreturnthenegatedvalueofvalueconvertedintoaboolean
thearithmeticoperators(+,-,*,/,%)canbeappliedtoanyvalueandwillnotthrowexceptionswhenappliedtonon-numericvalues.Instead,anyvalueusedintheseoperatorswillbecastedtoanumericvalueimplicitly.Ifnonumericresultcanbeproducedbyanarithmeticoperator,itwillreturnnullinArangoDB2.3.Thisisalsotruefordivisionbyzero.passingargumentsofinvalidtypesintoAQLfunctionsdoesnotthrowaruntimeexceptioninmostcases,butmayproduceruntime
WhatsNewin2.3
621
warnings.Built-inAQLfunctionsthatreceiveinvalidargumentswillthenreturnnull.
Performanceimprovements
Non-uniquehashindexes
Theperformanceofinsertionintonon-uniquehashindexeshasbeenimprovedsignificantly.Thisfixesperformanceproblemsincaseattributeswereindexesthatcontainedonlyveryfewdistinctvalues,orwhenmostofthedocumentsdidnotevencontaintheindexedattribute.Thisalsofixesproblemswhenloadingcollectionswithsuchindexes.
Theinsertiontimenowscaleslinearlywiththenumberofdocumentsregardlessofthecardinalityoftheindexedattribute.
Reverseiterationoverskiplistindexes
AQLqueriescannowuseasortedskiplistindexforreverseiteration.ThisallowsseveralqueriestorunfasterthaninpreviousversionsofArangoDB.
Forexample,thefollowingAQLquerycannowusetheindexondoc.value:
FORdocINmycollection
FILTERdoc.value>23
SORTdoc.valuesDESC
RETURNdoc
PreviousversionsofArangoDBdidnotusetheindexbecauseofthedescending(DESC)sort.
Additionally,thenewAQLoptimizercanuseanindexforsortingnoweveniftheAQLquerydoesnotcontainaFILTERstatement.ThisoptimizationwasnotavailableinpreviousversionsofArangoDB.
AddedbasicsupportforhandlingbinarydatainFoxx
BufferobjectscannowbeusedwhensettingtheresponsebodyofanyFoxxaction.ThisallowsFoxxactionstoreturnbinarydata.
RequestswithbinarypayloadcanbeprocessedinFoxxapplicationsbyusingthenewmethodres.rawBodyBuffer().ThiswillreturntheunparsedrequestbodyasaBufferobject.
Thereisnowalsothemethodreq.requestParts()availableinFoxxtoretrievetheindividualcomponentsofamultipartHTTPrequest.Thatcanbeusedforexampletoprocessfileuploads.
Additionally,theres.send()methodhasbeenaddedasaconveniencemethodforreturningstrings,JSONobjectsorBuffersfromaFoxxaction.Itprovidessomeauto-detectionbasedonitsparametervalue:
res.send("<p>someHTML</p>");//returnsanHTMLstring
res.send({success:true});//returnsaJSONobject
res.send(newBuffer("somebinarydata"));//returnsbinarydata
Theconveniencemethodres.sendFile()cannowbeusedtoreturnthecontentsofafilefromaFoxxaction.Theyfilemaycontainbinarydata:
res.sendFile(applicationContext.foxxFilename("image.png"));
Thefilesystemmethodsfs.write()andfs.readBuffer()canbeusedtoworkwithbinarydata,too:
fs.write()willperformanauto-detectionofitssecondparameter'svaluesoitworkswithBufferobjects:
fs.write(filename,"somedata");//savesastringvalueinfile
fs.write(filename,newBuffer("somebinarydata"));//saves(binary)contentsofabuffer
fs.readBuffer()hasbeenaddedasamethodtoreadthecontentsofanarbitraryfileintoaBufferobject.
WhatsNewin2.3
622
Webinterface
Batchdocumentremovalandmovefunctionalityhasbeenaddedtothewebinterface,makingiteasiertoworkwithmultipledocumentsatonce.Additionally,basicJSONimportandexporttoolshavebeenadded.
Command-lineoptionsadded
Thecommand-lineoption--javascript.v8-contextswasaddedtoarangodtoprovidebettercontroloverthenumberofV8contextscreatedinarangod.
Previously,thenumberofV8contextsarangodcreatedatstartupwasequaltothenumberofserverthreads(asspecifiedbyoption--server.threads).
InsomesituationsitmaybemoresensibletocreatedifferentamountsofthreadsandV8contexts.ThisisbecauseeachV8contextscreatedwillconsumememoryandrequiresCPUresourcesforperiodicgarbagecollection.Contrary,serverthreadsdonothavesuchhighmemoryorCPUfootprint.
Iftheoption--javascript.v8-contextsisnotspecified,thenumberofV8contextscreatedatstartupwillremainequaltothenumberofserverthreads.ThusnochangeinconfigurationisrequiredtokeepthesamebehaviorasinpreviousArangoDBversions.
Thecommand-lineoption--log.use-local-timewasaddedtoprintdatesandtimesinArangoDB'slogintheserver-localtimezoneinsteadofUTC.Ifitisnotset,thetimezonewilldefaulttoUTC.
Theoption--backslash-escapehasbeenaddedtoarangoimp.SpecifyingthisoptionwillusethebackslashastheescapecharacterforliteralquoteswhenparsingCSVfiles.Theescapecharacterforliteralquotesisstillthedoublequotecharacter.
Miscellaneousimprovements
ArangoDB'sbuilt-inHTTPservernowsupportsHTTPpipelining.
TheArangoShelltutorialfromthearangodb.comwebsiteisnowintegratedintotheArangoDBshell.
PowerfulFoxxEnhancementsWiththenewjobqueuefeatureyoucanrunasyncjobstocommunicatewithexternalservices,FoxxqueriesmakewritingcomplexAQLqueriesmucheasierandFoxxsessionswillhandletheauthenticationandsessionhassleforyou.
FoxxQueries
WritinglongAQLqueriesinJavaScriptcanquicklybecomeunwieldy.Asof2.3ArangoDBbundlestheArangoDBQueryBuildermodulethatprovidesaJavaScriptAPIforwritingcomplexAQLquerieswithoutstringconcatenation.Allbuilt-infunctionsthatacceptAQLstringsnowsupportquerybuilderinstancesdirectly.AdditionallyFoxxprovidesamethodFoxx.createQueryforcreatingparametrizedqueriesthatcanreturnFoxxmodelsorapplyarbitrarytransformationstothequeryresults.
FoxxSessions
ThesessionfunctionalityinFoxxhasbeencompletelyrewritten.TheoldactivateAuthenticationAPIisstillsupportedbutmaybedeprecatedinthefuture.ThenewactivateSessionsAPIsupportscookiesorconfigurableheaders,providesoptionalJSONWebTokenandcryptographicsigningsupportandusesthenewsessionsFoxxapp.
ArangoDB2.3providesFoxxappsforusermanagementandsaltedhash-basedauthenticationwhichcanbereplacedwithorsupplementedbyalternativeimplementations.Foranexampleappusingboththebuilt-inauthenticationandOAuth2seetheFoxxSessionsExampleapp.
FoxxQueues
FoxxnowprovidesasyncworkersviatheFoxxQueuesAPI.Jobsenqueuedinajobqueuewillbeexecutedasynchronouslyoutsideoftherequest/responsecycleofFoxxcontrollersandcanbeusedtocommunicatewithexternalservicesorperformtasksthattakealongtimetocompleteormayrequiremultipleattempts.
WhatsNewin2.3
623
Jobscanbescheduledinadvanceorsettobeexecutedimmediately,thenumberofretryattempts,theretrydelayaswellassuccessandfailurehandlerscanbedefinedforeachjobindividually.Jobtypesthatintegratevariousexternalservicesfortransactionale-mails,loggingandusertrackingcanbefoundintheFoxxappregistry.
Misc
TherequestandresponseobjectsinFoxxcontrollersnowprovidemethodsforreadingandwritingrawcookiesandsignedcookies.
MountedFoxxappswillnowbeloadedwhenarangodstartsratherthanatthefirstdatabaserequest.Thismayresultinslightlyslowerstartuptimes(butafasterresponseforthefirstrequest).
WhatsNewin2.3
624
IncompatiblechangesinArangoDB2.3ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB2.3,andadjustanyclientprogramsifnecessary.
Configurationfilechanges
Threadsandcontexts
Thenumberofserverthreadsspecifiedisnowtheminimumofthreadsstarted.Therearesituationinwhichthreadsarewaitingforresultsofdistributeddatabaseservers.Inthiscasethenumberofthreadsisdynamicallyincreased.
WithArangoDB2.3,thenumberofserverthreadscanbeconfiguredindependentlyofthenumberofV8contexts.Theconfigurationoption--javascript.v8-contextswasaddedtoarangodtoprovidebettercontroloverthenumberofV8contextscreatedinarangod.
Previously,thenumberofV8contextsarangodcreatedatstartupwasequaltothenumberofserverthreads(asspecifiedbyoption--server.threads).
InsomesituationsitmaybemoresensibletocreatedifferentamountsofthreadsandV8contexts.ThisisbecauseeachV8contextscreatedwillconsumememoryandrequiresCPUresourcesforperiodicgarbagecollection.Contrary,serverthreadsdonothavesuchhighmemoryorCPUfootprint.
Iftheoption--javascript.v8-contextsisnotspecified,thenumberofV8contextscreatedatstartupwillremainequaltothenumberofserverthreads.ThusnochangeinconfigurationisrequiredtokeepthesamebehaviorasinpreviousArangoDBversions.
Ifyouareusingthedefaultconfigfilesormergethemwithyourlocalconfigfiles,pleasereviewifthedefaultnumberofserverthreadsisokayinyourenvironment.AdditionallyyoushouldverifythatthenumberofV8contextscreated(asspecifiedinoption--javascript.v8-contexts)isokay.
Syslog
Thecommand-lineoption--log.syslogwasusedinpreviousversionsofArangoDBtoturnloggingtosyslogonoroff:whensettingtoanon-emptystring,syslogloggingwasturnedon,otherwiseturnedoff.Whensyslogloggingwasturnedon,loggingwasdonewiththeapplicationnamespecifiedin--log.application,whichdefaultedtotriagens.Therewasalsoacommand-lineoption--log.hostnamewhichcouldbesetbutdidnothaveanyeffect.
Thisbehaviorturnedouttobeunintuitiveandwaschangedin2.3asfollows:
thecommand-lineoption--log.syslogisdeprecatedanddoesnothaveanyeffectwhenstartingArangoDB.toturnonsyslogloggingin2.3,theoption--log.facilityhastobesettoanon-emptystring.ThevalueforfacilityisOS-dependent(possiblevaluescanbefoundin/usr/include/syslog.horthelike-usershouldbeavailableonmanysystems).thedefaultvaluefor--log.applicationhasbeenchangedfromtriagenstoarangod.thecommand-lineoption--log.hostnameisdeprecatedanddoesnothaveanyeffectwhenstartingArangoDB.Instead,thehostnamewillbesetbysyslogautomatically.whenloggingtosyslog,ArangoDBnowomitsthedatetimeprefixandtheprocessid,becausethey'llbeaddedbysyslogautomatically.
AQL
AQLqueriesthrowlessexceptions
ArangoDB2.3containsacompletelyrewrittenAQLqueryoptimizerandexecutionengine.ThismeansthatAQLquerieswillbeexecutedwithadifferentenginethaninArangoDB2.2andearlier.PartsofAQLqueriesmightbeexecutedindifferentorderthanbeforebecausetheAQLoptimizerhasmorefreedomtomovethingsaroundinaquery.
Incompatiblechangesin2.3
625
InpreviousversionsofArangoDB,AQLqueriesabortedwithanexceptioninmanysituationsandthrewaruntimeexception.ExceptionswerethrownwhentryingtofindavalueusingtheINoperatorinanon-arrayelement,whentryingtousenon-booleanvalueswiththelogicaloperands&&or||or!,whenusingnon-numericvaluesinarithmeticoperations,whenpassingwrongparametersintofunctionsetc.
InArangoDB2.3thishasbeenchangedinmanycasestomakeAQLmoreuser-friendlyandtoallowtheoptimizationtoperformmuchmorequeryoptimizations.
Hereisasummaryofchanges:
whenanon-arrayvalueisusedontheright-handsideoftheINoperator,theresultwillbefalseinArangoDB2.3,andnoexceptionwillbethrown.thebooleanoperators&&and||donotthrowinArangoDB2.3ifanyoftheoperandsisnotabooleanvalue.Instead,theywillperformanimplicitcastofthevaluestobooleans.Theirresultwillbeasfollows:
lhs&&rhswillreturnlhsifitisfalseorwouldbefalsewhenconvertedintoaboolean.Iflhsistrueorwouldbetruewhenconvertedtoaboolean,rhswillbereturned.lhs||rhswillreturnlhsifitistrueorwouldbetruewhenconvertedintoaboolean.Iflhsisfalseorwouldbefalsewhenconvertedtoaboolean,rhswillbereturned.!valuewillreturnthenegatedvalueofvalueconvertedintoaboolean
thearithmeticoperators(+,-,*,/,%)canbeappliedtoanyvalueandwillnotthrowexceptionswhenappliedtonon-numericvalues.Instead,anyvalueusedintheseoperatorswillbecastedtoanumericvalueimplicitly.Ifnonumericresultcanbeproducedbyanarithmeticoperator,itwillreturnnullinArangoDB2.3.Thisisalsotruefordivisionbyzero.passingargumentsofinvalidtypesintoAQLfunctionsdoesnotthrowaruntimeexceptioninmostcases,butmayproduceruntimewarnings.Built-inAQLfunctionsthatreceiveinvalidargumentswillthenreturnnull.
NestedFORloopexecutionorder
Thequeryoptimizerin2.3maypermutetheorderofnestedFORloopsinAQLqueries,providedthatexchangingtheloopswillnotalteraqueryresult.However,achangeintheorderofreturnedvaluesisallowedbecausenosortorderisguaranteedbyAQL(andwasnever)unlessanexplicitSORTstatementisusedinaquery.
ChangedreturnvaluesofArangoQueryCursor.getExtra()
ThereturnvalueofArangoQueryCursor.getExtra()hasbeenchangedinArangoDB2.3.Itnowcontainsastatsattributewithstatisticsaboutthequerypreviouslyexecuted.Italsocontainsawarningsattributewithwarningsthathappenedduringqueryexecution.Thereturnvaluestructurehasbeenunifiedin2.3forbothread-onlyanddata-modificationqueries.
Thereturnvaluelookslikethisforaread-onlyquery:
arangosh>stmt=db._createStatement("FORiINmycollectionRETURNi");stmt.execute().getExtra()
{
"stats":{
"writesExecuted":0,
"writesIgnored":0,
"scannedFull":2600,
"scannedIndex":0
},
"warnings":[]
}
Fordata-modificationqueries,ArangoDB2.3returnsaresultwiththesamestructure:
arangosh>stmt=db._createStatement("FORiINxxREMOVEiINxx");stmt.execute().getExtra()
{
"stats":{
"writesExecuted":2600,
"writesIgnored":0,
"scannedFull":2600,
"scannedIndex":0
},
"warnings":[]
}
Incompatiblechangesin2.3
626
InArangoDB2.2,thereturnvalueofArangoQueryCursor.getExtra()wasemptyforread-onlyqueriesandcontainedanattributeoperationswithtwosub-attributesfordata-modificationqueries:
arangosh>stmt=db._createStatement("FORiINmycollectionRETURNi");stmt.execute().getExtra()
{
}
arangosh>stmt=db._createStatement("FORiINmycollectionREMOVEiINmycollection");stmt.execute().getExtra()
{
"operations":{
"executed":2600,
"ignored":0
}
}
ChangedreturnvaluesinHTTPmethodPOST/_api/cursor
ThepreviouslymentionedchangealsoleadstothestatisticsbeingreturnedintheHTTPRESTAPImethodPOST/_api/cursor.Previously,thereturnvaluecontainedanoptionalextraattributethatwasfilledonlyfordata-modificationqueriesandinsomeothercasesasfollows:
{
"result":[],
"hasMore":false,
"extra":{
"operations":{
"executed":2600,
"ignored":0
}
}
}
WiththechangedresultstructureinArangoDB2.3,theextraattributeintheresultwilllooklikethis:
{
"result":[],
"hasMore":false,
"extra":{
"stats":{
"writesExecuted":2600,
"writesIgnored":0,
"scannedFull":0,
"scannedIndex":0
},
"warnings":[]
}
}
IfthequeryoptionfullCountisrequested,thefullCountresultvaluewillalsobereturnedinsidethestatsattributeoftheextraattribute,andnotdirectlyasanattributeinsidetheextraattributeasin2.2.NotethatafullCountwillonlybepresentinextra.statsifitwasrequestedasanoptionforthequery.
TheresultinArangoDB2.3willalsocontainawarningsattributewiththearrayofwarningsthathappenedduringqueryexecution.
ChangedreturnvaluesinArangoStatement.explain()
ThereturnvalueofArangoStatement.explain()haschangedsignificantlyinArangoDB2.3.Thenewreturnvaluestructureisnotcompatiblewiththestructurereturnedby2.2.
InArangoDB2.3,thefullexecutionplanforanAQLqueryisreturnedalongsideallappliedoptimizerrules,optimizationwarningsetc.Itisalsopossibletohavetheoptimizerreturnallexecutionplans.Thisrequiredanewdatastructure.
ClientprogramsthatuseArangoStatement.explain()ortheHTTPRESTAPImethodPOST/_api/explainmayneedtobeadjustedtousethenewreturnformat.
Incompatiblechangesin2.3
627
ThereturnvalueofArangoStatement.parse()hasbeenextendedinArangoDB2.3.Inadditiontotheexistingattributes,ArangoDB2.3willalsoreturnanastattributecontainingtheabstractsyntaxtreeofthestatement.Thisextraattributecansafelybeignoredbyclientprograms.
Variablesnotupdatableinqueries
PreviousversionsofArangoDBallowedthemodificationofvariablesinsideAQLqueries,e.g.
LETcounter=0
FORiIN1..10
LETcounter=counter+1
RETURNcounter
Whilethisisadmittedlyaconvenientfeature,thenewqueryoptimizerdesigndidnotallowtokeepit.Additionally,updatingvariablesinsideaquerywouldpreventalotofoptimizationstoqueriesthatwewouldliketheoptimizertomake.Additionally,updatingvariablesinqueriesthatrunondifferentnodesinaclusterwouldlikecausenon-deterministicbehaviorbecausequeriesarenotexecutedlinearly.
ChangedreturnvalueofTO_BOOL
TheAQLfunctionTO_BOOLnowalwaysreturnstrueifitsargumentisanarrayoranobject.InpreviousversionsofArangoDB,thefunctionreturnedfalseforemptyarraysorforobjectswithoutattributes.
ChangedreturnvalueofTO_NUMBER
TheAQLfunctionTO_NUMBERnowreturnsnullifitsargumentisanobjectoranarraywithmorethanonemember.InpreviousversionofArangoDB,thereturnvalueinthesecaseswas0.TO_NUMBERwillreturn0foremptyarray,andthenumericequivalentofthearraymember'svalueforarrayswithasinglemember.
NewAQLkeywords
ThefollowingkeywordshavebeenaddedtoAQLinArangoDB2.3:
NOTANDOR
UnquotedusageofthesekeywordsforattributenamesinAQLquerieswilllikelyfailinArangoDB2.3.Ifanysuchattributenameneedstobeusedinaquery,itshouldbeenclosedinbacktickstoindicatetheusageofaliteralattributename.
Removedfeatures
Bitarrayindexes
Bitarrayindexeswereonlyhalf-waydocumentedandintegratedinpreviousversionsofArangoDBsotheirbenefitwaslimited.ThesupportforbitarrayindexeshasthusbeenremovedinArangoDB2.3.Itisnotpossibletocreateindexesoftype"bitarray"withArangoDB2.3.
WhenacollectionisopenedthatcontainsabitarrayindexdefinitioncreatedwithapreviousversionofArangoDB,ArangoDBwillignoreitandlogthefollowingwarning:
indextype'bitarray'isnotsupportedinthisversionofArangoDBandisignored
FutureversionsofArangoDBmayautomaticallyremovesuchindexdefinitionssothewarningswilleventuallydisappear.
Otherremovedfeatures
TheHTTPRESTAPImethodatPOST/_admin/modules/flushhasbeenremoved.
Incompatiblechangesin2.3
628
Knownissues
InArangoDB2.3.0,AQLqueriescontainingfilterconditionswithanINexpressionwillnotyetuseanindex:
FORdocINcollectionFILTERdoc.indexedAttributeIN[...]RETURNdoc
FORdocINcollection
FILTERdoc.indexedAttributeIN[...]
RETURNdoc
We’recurrentlyworkingongettingtheINoptimizationsdone,andwillshipthemina2.3maintenancereleasesoon(e.g.2.3.1or2.3.2).
Incompatiblechangesin2.3
629
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.2.ArangoDB2.2alsocontainsseveralbugfixesthatarenotlistedhere.
AQLimprovements
DatamodificationAQLqueries
Uptoincludingversion2.1,AQLsupporteddataretrievaloperationsonly.StartingwithArangoDBversion2.2,AQLalsosupportsthefollowingdatamodificationoperations:
INSERT:insertnewdocumentsintoacollectionUPDATE:partiallyupdateexistingdocumentsinacollectionREPLACE:completelyreplaceexistingdocumentsinacollectionREMOVE:removeexistingdocumentsfromacollection
Data-modificationoperationsarenormallycombinedwithotherAQLstatementssuchasFORloopsandFILTERconditionstodeterminethesetofdocumentstooperateon.Forexample,thefollowingquerywillfindalldocumentsincollectionusersthatmatchaspecificconditionandsettheirstatusvariabletoinactive:
FORuINusers
FILTERu.status=='notactive'
UPDATEuWITH{status:'inactive'}INusers
Thefollowingquerycopiesalldocumentsfromcollectionusersintocollectionbackup:
FORuINusers
INSERTuINbackup
Andthisqueryremovesdocumentsfromcollectionbackup:
FORdocINbackup
FILTERdoc.lastModified<DATE_NOW()-3600
REMOVEdocINbackup
Formoreinformationondata-modificationqueries,pleaserefertoDatamodificationqueries.
Updatablevariables
Previously,thevalueofavariableassignedinanAQLquerywiththeLETkeywordwasnotupdatableinanAQLquery.Thispreventedstatementslikethefollowingfrombeingexecutable:
LETsum=0
FORvINvalues
SORTv.year
LETsum=sum+v.value
RETURN{year:v.year,value:v.value,sum:sum}
OtherAQLimprovements
addedAQLTRANSLATEfunction
Thisfunctioncanbeusedtoperformlookupsfromstaticobjects,e.g.
LETcountryNames={US:"UnitedStates",UK:"UnitedKingdom",FR:"France"}
RETURNTRANSLATE("FR",countryNames)
WhatsNewin2.2
630
LETlookup={foo:"foo-replacement",bar:"bar-replacement",baz:"baz-replacement"}
RETURNTRANSLATE("foobar",lookup,"notcontained!")
Write-aheadlogAllwriteoperationsinanArangoDBserverwillnowbeautomaticallyloggedintheserver'swrite-aheadlog.Thewrite-aheadlogisasetofappend-onlylogfiles,anditisusedincaseofacrashrecoveryandforreplication.
Datafromthewrite-aheadlogwilleventuallybemovedintothejournalsordatafilesofcollections,allowingtheservertoremoveolderwrite-aheadlogfiles.
Cross-collectiontransactionsinArangoDBshouldbenefitconsiderablybythischange,aslesswritesthaninpreviousversionsarerequiredtoensurethedataofmultiplecollectionsareatomicallyanddurablycommitted.Alldata-modifyingoperationsinsidetransactions(insert,update,remove)willwritetheiroperationsintothewrite-aheadlogdirectlynow.Inpreviousversions,suchoperationswerebuffereduntilthecommitorrollbackoccurred.TransactionswithmultipleoperationsshouldthereforerequirelessphysicalmemorythaninpreviousversionsofArangoDB.
Thedatainthewrite-aheadlogcanalsobeusedinthereplicationcontext.InpreviousversionsofArangoDB,replicatingfromamasterrequiredturningonaspecialreplicationloggeronthemaster.Thereplicationloggercausedanextrawriteoperationintothe_replicationsystemcollectionforeachactualwriteoperation.Thisextrawriteisnowsuperfluous.Instead,slavescanreaddirectlyfromthemaster'swrite-aheadlogtogetinformedaboutmostrecentdatachanges.Thisremovestheneedtostoredata-modificationoperationsinthe_replicationcollectionaltogether.
Fortheconfigurationofthewrite-aheadlog,pleaserefertoWrite-aheadlogoptions.
Theintroductionofthewrite-aheadlogalsoremovestheneedtoconfigureandstartthereplicationloggeronamaster.ThoughthereplicationloggerobjectisstillavailableinArangoDB2.2toensureAPIcompatibility,starting,stopping,orconfiguringitwillhavenoeffect.
PerformanceimprovementsRemovedsortingofattributenameswhenincollectionshaper
InpreviousversionsofArangoDB,addingadocumentwithpreviouslynot-usedattributenamescausedafullsortofallattributenamesusedinthecollection.Thesortingwasdonetoensurefastcomparisonsofattributenamesinsomerareedgecases,butitconsiderablysloweddowninsertsintocollectionswithmanydifferentorevenuniqueattributenames.
Specializedprimaryindeximplementationtoallowfasterhashtablerebuildingandreducelookupsindatafilesfortheactualvalueof_key.Thisalsoreducestheamountofrandommemoryaccessesforprimaryindexinserts.
Reclamationofindexmemorywhendeletinglastdocumentincollection
Deletingdocumentsfromacollectiondidnotleadtoindexsizesbeingreduced.Instead,theindexmemorywaskeptallocatedandre-usedlaterwhenacollectionwasrefilledwithnewdocuments.Now,indexmemoryofprimaryindexesandhashindexesisreclaimedinstantlywhenthelastdocumentinacollectionisremoved.
Preventbufferingoflongprintresultsinarangosh'sandarangod'sprintcommand
Thischangewillemitbufferedintermediateprintresultsanddiscardtheoutputbuffertoquicklydeliverprintresultstotheuser,andtopreventconstructingverylargebuffersforlargeresults.
Miscellaneousimprovements
Addedinsertmethodasanaliasforsave.Documentscannowbeinsertedintoacollectionusingeithermethod:
db.test.save({foo:"bar"});
db.test.insert({foo:"bar"});
Cleanupofoptionsfordata-modificationoperations
WhatsNewin2.2
631
Manyofthedata-modificationoperationshadsignatureswithmanyoptionalboolparameters,e.g.:
db.test.update("foo",{bar:"baz"},true,true,true)
db.test.replace("foo",{bar:"baz"},true,true)
db.test.remove("foo",true,true)
db.test.save({bar:"baz"},true)
Suchlongparameterlistswereunintuitiveandhardtousewhenonlyoneoftheoptionalparametersshouldhavebeenset.
TomaketheAPIsmoreusable,theoperationsnowunderstandthefollowingalternativesignature:
collection.update(key,update-document,options)
collection.replace(key,replacement-document,options)
collection.remove(key,options)
collection.save(document,options)
Examples:
db.test.update("foo",{bar:"baz"},{overwrite:true,keepNull:true,waitForSync:true})
db.test.replace("foo",{bar:"baz"},{overwrite:true,waitForSync:true})
db.test.remove("foo",{overwrite:true,waitForSync:true})
db.test.save({bar:"baz"},{waitForSync:true})
Added--overwriteoptiontoarangoimp
Thisallowsremovingalldocumentsinacollectionbeforeimportingintoitusingarangoimp.
Honorstartupoption--server.disable-statisticswhendecidingwhetherornottostartperiodicstatisticscollectionjobs
Previously,thestatisticscollectionjobswerestartedeveniftheserverwasstartedwiththe--server.disable-statisticsflagbeingsettotrue.Nowiftheoptionissettotrue,nostatisticswillbecollectedontheserver.
DisallowstoringofJavaScriptobjectsthatcontainJavaScriptnativeobjectsoftypeDate,Function,RegExporExternal,e.g.
db.test.save({foo:/bar/});
db.test.save({foo:newDate()});
Thiswillnowprint
Error:<data>cannotbeconvertedintoJSONshape:couldnotshapedocument
Previously,objectsofthesetypesweresilentlyconvertedintoanemptyobject(i.e.{})andnowarningwasissued.
Tostoresuchobjectsinacollection,explicitlyconvertthemintostringslikethis:
db.test.save({foo:String(/bar/)});
db.test.save({foo:String(newDate())});
Removedfeatures
MRubyintegrationforarangod
ArangoDBhadanexperimentalMRubyintegrationinsomeofthepublishbuilds.Thiswasn'tcontinuouslydeveloped,andsoithasbeenremovedinArangoDB2.2.
Thischangehasledtothefollowingstartupoptionsbeingsuperfluous:
--ruby.gc-interval
--ruby.action-directory
--ruby.modules-path
--ruby.startup-directory
WhatsNewin2.2
632
SpecifyingthesestartupoptionswilldonothinginArangoDB2.2,sousingtheseoptionsshouldbeavoidedfromnowonastheymightberemovedinafutureversionofArangoDB.
Removedstartupoptions
ThefollowingstartupoptionshavebeenremovedinArangoDB2.2.Specifyingthemintheserver'sconfigurationfilewillnotproduceanerrortomakemigrationeasier.Still,usageoftheseoptionsshouldbeavoidedastheywillnothaveanyeffectandmightfullyberemovedinafutureversionofArangoDB:
--database.remove-on-drop
--database.force-sync-properties
--random.no-seed
--ruby.gc-interval
--ruby.action-directory
--ruby.modules-path
--ruby.startup-directory
--server.disable-replication-logger
MultiCollectionGraphs
ArangoDBisamultimodeldatabasewithnativegraphsupport.Inversion2.2thefeaturesforgraphshavebeenimprovedbyintegrationofanewgraphmodule.Allgraphscreatedwiththeoldmoduleareautomaticallymigratedintothenewmodulebutcanstillbeusedbytheoldmodule.
Newgraphmodule
Uptoincludingversion2.1,ArangoDBofferedamoduleforgraphsandgraphoperations.Thismoduleallowedyoutouseexactlyoneedgecollectiontogetherwithonevertexcollectioninagraph.WithArangoDBversion2.2thisgraphmoduleisdeprecatedandanewgraphmoduleisoffered.Thisnewmoduleallowstocombineanarbitrarynumberofvertexcollectionsandedgecollectionsinthesamegraph.Foreachedgecollectionalistofcollectionscontainingsourceverticesandalistofcollectionscontainingtargetverticescanbedefined.IfanedgeisstoredArangoDBchecksifthisedgeisvalidinthiscollection.Furthermoreifavertexisremovedfromoneofthecollectionsallconnectededgeswillberemovedaswell,givingtheguaranteeofnolooseendsinthegraphs.Thelayoutofthegraphcanbemodifiedatruntimebyaddingorremovingcollectionsandchangingthedefinitionsforedgecollections.Alloperationsonthegraphlevelaretransactionalbydefault.
GraphsinAQL
MulticollectiongraphshavebeenaddedtoAQLaswell.Basicfunctionality(gettingvertices,edges,neighbors)canbeexecutedusingtheentiregraph.Alsomoreadvancedfeatureslikeshortestpathcalculations,characteristicfactorsofthegraphortraversalshavebeenintegratedintoAQL.Forthesefunctionsallgraphscreatedwiththegraphmodulecanbeused.
WhatsNewin2.2
633
FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.1.ArangoDB2.1alsocontainsseveralbugfixesthatarenotlistedhere.
NewEdgesIndex
Theedgesindex(usedtostoreconnectionsbetweennodesinagraph)internallyusesanewdatastructure.Thisdatastructureimprovestheperformancewhenpopulatingtheedgeindex(i.e.whenloadinganedgecollection).Forlargegraphsloadingcanbe20timesfasterthanwithArangoDB2.0.
Additionally,thenewindexfixesperformanceproblemsthatoccurredwhenmanyduplicate_fromor_tovalueswerecontainedintheindex.Furthermore,thenewindexsupportsfasterremovalofedges.
Finally,whenloadinganexistingcollectionandbuildingtheedgesindexforthecollection,lessmemoryre-allocationswillbeperformed.
Overall,thisshouldconsiderablyspeeduploadingedgecollections.
Thenewindextypereplacestheoldedgesindextypeautomatically,withoutanychangesbeingrequiredbytheenduser.
TheAPIofthenewindexiscompatiblewiththeAPIoftheoldindex.Stillitispossiblethatthenewindexreturnsedgesinadifferentorderthantheoldindex.Thisisstillconsideredtobecompatiblebecausetheoldindexhadneverguaranteedanyresultordereither.
AQLImprovements
AQLoffersfunctionalitytoworkwithdates.DatesarenodatatypesoftheirowninAQL(neithertheyareinJSON,whichisoftenusedasaformattoshipdataintoandoutofArangoDB).Instead,datesinAQLareinternallyrepresentedbyeithernumbers(timestamps)orstrings.ThedatefunctionsinAQLprovidemechanismstoconvertfromanumerictimestamptoastringrepresentationandviceversa.
TherearetwodatefunctionsinAQLtocreatedatesforfurtheruse:
DATE_TIMESTAMP(date)CreatesaUTCtimestampvaluefromdate
DATE_TIMESTAMP(year,month,day,hour,minute,second,millisecond):Sameasbefore,butallowsspecifyingtheindividualdatecomponentsseparately.Allparametersafterdayareoptional.
DATE_ISO8601(date):ReturnsanISO8601datetimestringfromdate.ThedatetimestringwillalwaysuseUTCtime,indicatedbytheZatitsend.
DATE_ISO8601(year,month,day,hour,minute,second,millisecond):sameasbefore,butallowsspecifyingtheindividualdatecomponentsseparately.Allparametersafterdayareoptional.
Thesetwoabovedatefunctionsacceptthefollowinginputvalues:
numerictimestamps,indicatingthenumberofmillisecondselapsedsincetheUNIXepoch(i.e.January1st197000:00:00UTC).Anexampletimestampvalueis1399472349522,whichtranslatesto2014-05-07T14:19:09.522Z.
datetimestringsinformatsYYYY-MM-DDTHH:MM:SS.MMM,YYYY-MM-DDHH:MM:SS.MMM,orYYYY-MM-DD.Millisecondsarealwaysoptional.
Atimezonedifferencemayoptionallybeaddedattheendofthestring,withthehoursandminutesthatneedtobeaddedorsubtractedtothedatetimevalue.Forexample,2014-05-07T14:19:09+01:00canbeusedtospecifyaonehouroffset,and2014-05-07T14:19:09+07:30canbespecifiedforsevenandhalfhoursoffset.Negativeoffsetsarealsopossible.Alternativelytoanoffset,aZcanbeusedtoindicateUTC/Zulutime.
Anexamplevalueis2014-05-07T14:19:09.522ZmeaningMay7th2014,14:19:09and522milliseconds,UTC/Zulutime.Anotherexamplevaluewithouttimecomponentis2014-05-07Z.
Pleasenotethatifnotimezoneoffsetisspecifiedinadatestring,ArangoDBwillassumeUTCtimeautomatically.Thisisdonetoensureportabilityofqueriesacrossserverswithdifferenttimezonesettings,andbecausetimestampswillalwaysbeUTC-based.
WhatsNewin2.1
634
individualdatecomponentsasseparatefunctionarguments,inthefollowingorder:
yearmonthdayhourminutesecondmillisecond
Allcomponentsfollowingdayareoptionalandcanbeomitted.Notethatnotimezoneoffsetscanbespecifiedwhenusingseparatedatecomponents,andUTC/Zulutimewillbeused.
ThefollowingcallstoDATE_TIMESTAMPareequivalentandwillallreturn1399472349522:
DATE_TIMESTAMP("2014-05-07T14:19:09.522")
DATE_TIMESTAMP("2014-05-07T14:19:09.522Z")
DATE_TIMESTAMP("2014-05-0714:19:09.522")
DATE_TIMESTAMP("2014-05-0714:19:09.522Z")
DATE_TIMESTAMP(2014,5,7,14,19,9,522)
DATE_TIMESTAMP(1399472349522)
ThesameistrueforcallstoDATE_ISO8601thatalsoacceptsvariableinputformats:
DATE_ISO8601("2014-05-07T14:19:09.522Z")
DATE_ISO8601("2014-05-0714:19:09.522Z")
DATE_ISO8601(2014,5,7,14,19,9,522)
DATE_ISO8601(1399472349522)
Theabovefunctionsareallequivalentandwillreturn"2014-05-07T14:19:09.522Z".
ThefollowingdatefunctionscanbeusedwithdatescreatedbyDATE_TIMESTAMPandDATE_ISO8601:
DATE_DAYOFWEEK(date):Returnstheweekdaynumberofdate.Thereturnvalueshavethefollowingmeanings:
0:Sunday1:Monday2:Tuesday3:Wednesday4:Thursday5:Friday6:Saturday
DATE_YEAR(date):Returnstheyearpartofdateasanumber.
DATE_MONTH(date):Returnsthemonthpartofdateasanumber.
DATE_DAY(date):Returnsthedaypartofdateasanumber.
DATE_HOUR(date):Returnsthehourpartofdateasanumber.
DATE_MINUTE(date):Returnstheminutepartofdateasanumber.
DATE_SECOND(date):Returnsthesecondspartofdateasanumber.
DATE_MILLISECOND(date):Returnsthemillisecondspartofdateasanumber.
Thefollowingotherdatefunctionsarealsoavailable:
DATE_NOW():Returnsthecurrenttimeasatimestamp.
Notethatthisfunctionisevaluatedoneveryinvocationandmayreturndifferentvalueswheninvokedmultipletimesinthesamequery.
ThefollowingotherAQLfunctionshavebeenaddedinArangoDB2.1:
FLATTEN:thisfunctioncanturnanarrayofsub-arraysintoasingleflatarray.Allarrayelementsintheoriginalarraywillbeexpandedrecursivelyuptoaconfigurabledepth.Theexpandedvalueswillbeaddedtothesingleresultarray.
WhatsNewin2.1
635
Example:
FLATTEN([1,2,[3,4],5,[6,7],[8,[9,10]])
willexpandthesub-arraysonthefirstlevelandproduce:
[1,2,3,4,5,6,7,8,[9,10]]
Tofullyflattenthearray,themaximumdepthcanbespecified(e.g.withavalueof2):
FLATTEN([1,2,[3,4],5,[6,7],[8,[9,10]],2)
Thiswillfullyexpandthesub-arraysandproduce:
[1,2,3,4,5,6,7,8,9,10]
CURRENT_DATABASE:thisfunctionwillreturnthenameofthedatabasethecurrentqueryisexecutedin.
CURRENT_USER:thisfunctionreturnsthenameofthecurrentuserthatisexecutingthequery.Ifauthorizationisturnedofforthequeryisexecutedoutsideofarequestcontext,nouserispresentandthefunctionwillreturnnull.
ClusterDumpandRestore
Thedumpandrestoretools,arangodumpandarangorestore,cannowbeusedtodumpandrestorecollectionsinacluster.Additionally,acollectiondumpfromastandaloneArangoDBservercanbeimportedintoacluster,andviceversa.
WebInterfaceImprovements
Thewebinterfaceinversion2.1hasamorecompactdashboard.Itprovideschartswithtime-seriesforincomingrequests,HTTPtransfervolumeandsomeserverresourceusagefigures.
Additionallyitprovidestrendindicators(e.g.15minaverages)anddistributioncharts(akahistogram)forsomefigures.
FoxxImprovements
ToeasilyaccessafileinsidethedirectoryofaFoxxapplicationfromwithinFoxx,Foxx'sapplicationContextnowprovidesthefoxxFilename()function.Itcanbeusedtoassemblethefullfilenameofafileinsidetheapplication'sdirectory.TheapplicationContextcanbeaccessedasglobalvariablefromanymodulewithinaFoxxapplication.
ThefilenamecanbeusedinsideFoxxactionsorsetup/teardownscripts,e.g.topopulateaFoxxapplication'scollectionwithdata.
TherequirefunctionnowalsopreferslocalmoduleswhenusedfrominsideaFoxxapplication.ThisallowsputtingmodulesinsidetheFoxxapplicationdirectoryandrequiringthemeasily.Italsoallowsusingapplication-specificversionsoflibrariesthatarebundledwithArangoDB(suchasunderscore.js).
WindowsInstaller
TheWindowsinstallershippedwithArangoDBnowsupportsinstallationofArangoDBforthecurrentuserorallusers,withtherequiredprivileges.ItalsosupportstheinstallationofArangoDBasaservice.
Fixesfor32bitsystems
SeveralissueshavebeenfixedthatoccurredonlywhenusingArangoDBona32bitsoperatingsystem,specifically:
acrashinathirdpartycomponentusedtomanageclusterdata
WhatsNewin2.1
636
athirdpartylibrarythatfailedtoinitializeon32bitWindows,makingarangodandarangoshcrashimmediately.
overflowsofvaluesusedfornanosecond-precisiontimeouts:theseoverflowshaveledtoinvalidvaluesbeingpassedtosocketoperations,makingthemfailandre-trytoooften
Updateddrivers
SeveraldriversforArangoDBhavebeencheckedforcompatibilitywith2.1.Thecurrentlistofdriverswithcompatibilitynotescanbefoundonlinehere.
C++11usage
WehavemovedseveralfilesfromCtoC++,allowingmorecodereuseandreducingtheneedforshippingdatabetweenthetwo.WehavealsodecidedtorequireC++11supportforArangoDB,whichallowsustousesomeofthesimplifications,featuresandguaranteesthatthisstandardhasinstock.
ThatalsomeansacompilerwithC++11supportisrequiredtobuildArangoDBfromsource.ForinstanceGNUCCofatleastversion4.8.
MiscellaneousImprovementsCancelableasynchronousjobs:severalpotentiallylong-runningjobscannowbecanceledviaanexplicitcanceloperation.Thisallowsstoppinglong-runningqueries,traversalsorscriptswithoutshuttingdownthecompleteArangoDBprocess.Jobcancelationisprovidedforasynchronouslyexecutedjobsasisdescribedin@refHttpJobCancel.
Server-sideperiodictaskmanagement:anArangoDBservernowprovidesfunctionalitytoregisterandunregisterperiodictasks.Tasksareuser-definedJavaScriptactionsthatcanberunperiodicallyandautomatically,independentofanyHTTPrequests.
Thefollowingtaskmanagementfunctionsareprovided:
require("org/arangodb/tasks").register():registersaperiodictaskrequire("org/arangodb/tasks").unregister():unregistersandremovesaperiodictaskrequire("org/arangodb/tasks").get():retrievesaspecifictasksorallexistingtasks
Anexampletask(tobeexecutedevery15seconds)canberegisteredlikethis:
vartasks=require("org/arangodb/tasks");
tasks.register({
name:"thisisanexampletaskwithparameters",
period:15,
command:function(params){
vargreeting=params.greeting;
vardata=JSON.stringify(params.data);
require('console').log('%sfromparametertask:%s',greeting,data);
},
params:{greeting:"hi",data:"howareyou?"}
});
Pleaserefertothesection@refTasksformoredetails.
Thefiguresmethodofacollectionnowreturnsdataaboutthecollection'sindexmemoryconsumption.Thereturnedvalueindexes.sizewillcontainthetotalamountofmemoryacquiredbyallindexesofthecollection.Thisfigurecanbeusedtoassessthememoryimpactofindexes.
CapitalizedHTTPresponseheaders:fromversion2.1,ArangoDBwillreturncapitalizedHTTPheadersbydefault,e.g.Content-Lengthinsteadofcontent-length.ThoughtheHTTPspecificationstatesthatheadersfieldnamearecase-insensitive,severalolderclienttoolsrelyonaspecificcaseinHTTPresponseheaders.ThischangesmakeArangoDBabitmorecompatiblewiththose.
Simplifiedusageofdb._createStatement():toeasilyrunanAQLquery,themethoddb._createStatementnowallowspassingtheAQLqueryasastring.Previouslyitrequiredtheusertopassanobjectwithaqueryattribute(whichthencontainedthequerystring).
WhatsNewin2.1
637
ArangoDBnowsupportsbothversions:
db._createStatement(queryString);
db._createStatement({query:queryString});
WhatsNewin2.1
638
AppendixReferences:Briefoverviewsoverinterfacesandobjects
db:thedbobjectcollection:thecollectionobject
JavaScriptModules:Listofbuilt-inandsupportedJSmodulesDeprecated:FeaturesthatareconsideredobsoleteandmaygetremovedeventuallyErrorcodesandmeanings:ListofallpossibleerrorsthatcanbeencounteredGlossary:Disambiguationpage
Appendix
639
The"db"ObjectThedbobjectisavailableinarangoshbydefault,andcanalsobeimportedandusedinFoxxservices.
db.namereturnsacollectionobjectforthecollectionname.
Thefollowingmethodsexistsonthe_dbobject:
Database
db._createDatabase(name,options,users)db._databases()db._dropDatabase(name,options,users)db._useDatabase(name)
Indexes
db._index(index)db._dropIndex(index)
Properties
db._id()db._isSystem()db._name()db._path()db._version()
Collection
db._collection(name)db._create(name)db._drop(name)db._truncate(name)
AQL
db._createStatement(query)db._query(query)db._explain(query)
Document
db._document(object)db._exists(object)db._remove(selector)db._replace(selector,data)db._update(selector,data)
db
641
The"collection"ObjectThefollowingmethodsexistonthecollectionobject(returnedbydb.name):
Collection
collection.checksum()collection.count()collection.drop()collection.figures()collection.load()collection.properties()collection.revision()collection.rotate()collection.toArray()collection.truncate()collection.type()collection.unload()
Indexes
collection.dropIndex(index)collection.ensureIndex(description)collection.getIndexes(name)collection.index(index)
Document
collection.all()collection.any()collection.closedRange(attribute,left,right)collection.document(object)collection.documents(keys)collection.edges(vertex-id)collection.exists(object)collection.firstExample(example)collection.inEdges(vertex-id)collection.insert(data)collection.edges(vertices)collection.iterate(iterator,options)collection.outEdges(vertex-id)collection.queryByExample(example)collection.range(attribute,left,right)collection.remove(selector)collection.removeByKeys(keys)collection.rename()collection.replace(selector,data)collection.replaceByExample(example,data)collection.save(data)collection.update(selector,data)collection.updateByExample(example,data)
collection
642
JavaScriptModulesArangoDBusesaNode.jscompatiblemodulesystem.Youcanusethefunctionrequireinordertoloadamoduleorlibrary.Itreturnstheexportedvariablesandfunctionsofthemodule.
Theglobalvariablesglobal,process,console,Buffer,__filenameand__dirnameareavailablethroughoutArangoDBandFoxx.
Nodecompatibilitymodules
ArangoDBsupportsanumberofmodulesforcompatibilitywithNode.js,including:
assertimplementsbasicassertionandtestingfunctions.
bufferimplementsabinarydatatypeforJavaScript.
consoleisawellknownloggingfacilitytoalltheJavaScriptdevelopers.ArangoDBimplementsmostoftheConsoleAPI,withtheexceptionsofprofileandcount.
eventsimplementsaneventemitter.
fsprovidesafilesystemAPIforthemanipulationofpaths,directories,files,links,andtheconstructionoffilestreams.ArangoDBimplementsmostFilesystem/Afunctions.
moduleprovidesdirectaccesstothemodulesystem.
pathimplementsfunctionsdealingwithfilenamesandpaths.
punycodeimplementsconversionfunctionsforpunycodeencoding.
querystringprovidesutilitiesfordealingwithquerystrings.
streamprovidesastreaminginterface.
string_decoderimplementslogicfordecodingbuffersintostrings.
urlprovidesutilitiesforURLresolutionandparsing.
utilprovidesgeneralutilityfunctionslikeformatandinspect.
AdditionallyArangoDBprovidespartialimplementationsforthefollowingmodules:
net:onlyisIP,isIPv4andisIPv6.
process:onlyenvandcwd;stubsforargv,stdout.isTTY,stdout.write,nextTick.
timers:stubsforsetImmediate,setTimeout,setInterval,clearImmediate,clearTimeout,clearIntervalandref.
tty:onlyisatty(alwaysreturnsfalse).
vm:onlyrunInThisContext.
ThefollowingNode.jsmodulesarenotavailableatall:child_process,cluster,constants,crypto(butsee@arangodb/cryptobelow),dgram,dns,domain,http(butsee@arangodb/requestbelow),https,os,sys,tls,v8,zlib.
ArangoDBSpecificModulesTherearealargenumberofArangoDB-specificmodulesusingthe@arangodbnamespace,mostlyforinternalusebyArangoDBitself.Thefollowinghoweverarenoteworthy:
@arangodbprovidesdirectaccesstothedatabaseanditscollections.
@arangodb/cryptoprovidesvariouscryptographyfunctionsincludinghashingalgorithms.
JavaScriptModules
643
@arangodb/requestprovidesthefunctionalityformakingsynchronousHTTP/HTTPSrequests.
@arangodb/foxxisthenamespaceprovidingthevariousbuildingblocksoftheFoxxmicroserviceframework.
BundledNPMModules
ThefollowingNPMmodulesarepreinstalled:
aqbistheArangoDBQueryBuilderandcanbeusedtoconstructAQLquerieswithachainingJavaScriptAPI.
chaiisafull-featuredassertionlibraryforwritingJavaScripttests.
dedentisasimpleutilityfunctionforformattingmulti-linestrings.
error-stack-parserparsesstacktracesintoamoreusefulformat.
graphql-syncisanArangoDB-compatibleGraphQLserver/schemaimplementation.
highlight.jsisanHTMLsyntaxhighlighter.
i(inflect)isautilitylibraryforinflecting(e.g.pluralizing)words.
iconv-liteisautilitylibraryforconvertingbetweencharacterencodings
joiisavalidationlibrarythatissupportedthroughouttheFoxxframework.
js-yamlisaJavaScriptimplementationoftheYAMLdataformat(apartialsupersetofJSON).
lodashisautilitybeltforJavaScriptprovidingvarioususefulhelperfunctions.
minimatchisaglobmatcherformatchingwildcardsinfilepaths.
qsprovidesutilitiesfordealingwithquerystringsusingadifferentformatthanthequerystringmodule.
semverisautilitylibraryforhandlingsemverversionnumbers.
sinonisamockinglibraryforwritingteststubs,mocksandspies.
timezoneisalibraryforconvertingdatetimevaluesbetweenformatsandtimezones.
JavaScriptModules
644
ArangoDBModuleconstarangodb=require('@arangodb')
Note:ThismoduleshouldnotbeconfusedwiththearangojsJavaScriptdriverwhichcanbeusedtoaccessArangoDBfromoutsidethedatabase.AlthoughtheAPIssharesimilaritiesandthefunctionalityoverlaps,thetwoarenotcompatiblewitheachotherandcannotbeusedinterchangeably.
The dbobjectarangodb.db
Thedbobjectrepresentsthecurrentdatabaseandletsyouaccesscollectionsandrunqueries.Formoreinformationseethedbobjectreference.
Examples
constdb=require('@arangodb').db;
constthirteen=db._query('RETURN5+8').next();
The aqltemplatestringhandlerarangodb.aql
TheaqlfunctionisaJavaScripttemplatestringhandler.ItcanbeusedtowritecomplexAQLqueriesasmulti-linestringswithouthavingtoworryaboutbindVarsandthedistinctionbetweencollectionsandregularparameters.
TouseitjustprefixaJavaScripttemplatestring(theoneswithbackticksinsteadofquotes)withitsimportname(e.g.aql)andpassinvariableslikeyouwouldwitharegulartemplatestring.ThestringwillautomaticallybeconvertedintoanobjectwithqueryandbindVarsattributeswhichyoucanpassdirectlytodb._querytoexecute.Ifyoupassinacollectionitwillbeautomaticallyrecognizedasacollectionreferenceandhandledaccordingly.
Youcanalsousetheaql.literalhelpertomarkstringscontainingAQLsnippetsthatshouldbeinlineddirectlyintothequeryratherthanbetreatedasbindvariables.
aql.literalallowsyoutopassaarbitrarystringsintoyourAQLandthuswillopenyoutoAQLinjectionattacksifyouarepassinginuntrusteduserinputunsanitized.
TofindoutmoreaboutAQLseetheAQLdocumentation.
Examples
constaql=require('@arangodb').aql;
constfilterValue=23;
constmydata=db._collection('mydata');
constresult=db._query(aql`
FORdIN${mydata}
FILTERd.num>${filterValue}
RETURNd
`).toArray();
constfilterGreen=aql.literal('FILTERd.color=="green"');
constresult2=db._query(aql`
FORdIN${mydata}
${filterGreen}
RETURNd
`).toArray()
@arangodb
645
The queryhelperarangodb.query
Inmostcasesyouwilllikelyusetheaqltemplatehandlertocreateaqueryyoudirectlypasstodb._query.TomakethiseveneasierArangoDBprovidesthequerytemplatehandler,whichbehavesexactlylikeaqlbutalsodirectlyexecutesthequeryandreturnstheresultcursorinsteadofthequeryobject:
constquery=require('@arangodb').query;
constfilterValue=23;
constmydata=db._collection('mydata');
constresult=query`
FORdIN${mydata}
FILTERd.num>${filterValue}
RETURNd
`.toArray();
The errorsobjectarangodb.errors
ThisobjectprovidesusefulobjectsforeacherrorcodeArangoDBmightuseinArangoErrorerrors.ThisishelpfulwhentryingtocatchspecificerrorsraisedbyArangoDB,e.g.whentryingtoaccessadocumentthatdoesnotexist.EachobjecthasacodepropertycorrespondingtotheerrorNumfoundonArangoErrorerrors.
Foracompletelistoftheerrornamesandcodesyoumayencounterseetheappendixonerrorcodes.
Examples
consterrors=require('@arangodb').errors;
try{
someCollection.document('does-not-exist');
}catch(e){
if(e.isArangoError&&e.errorNum===errors.ERROR_ARANGO_DOCUMENT_NOT_FOUND.code){
thrownewError('Documentdoesnotexist');
}
thrownewError('Somethingwentwrong');
}
The timefunctionarangodb.time
Thisfunctionprovidesthecurrenttimeinsecondsasafloatingpointvaluewithmicrosecondprecisison.
ThisfunctioncanbeusedinsteadofDate.now()whenadditionalprecisionisneeded.
Examples
consttime=require('@arangodb').time;
conststart=time();
db._query(someVerySlowQuery);
console.log(`Elapsedtime:${time()-start}secs`);
@arangodb
646
ConsoleModuleglobal.console===require('console')
Note:Youdon'tneedtoloadthismoduledirectly.TheconsoleobjectisgloballydefinedthroughoutArangoDBandprovidesaccesstoallfunctionsinthismodule.
console.assert
console.assert(expression,format,argument1,...)
Teststhatanexpressionistrue.Ifnot,logsamessageandthrowsanexception.
Examples
console.assert(value==="abc","expected:value===abc,actual:",value);
console.debug
console.debug(format,argument1,...)
Formatstheargumentsaccordingtoformatandlogstheresultasdebugmessage.Notethatdebugmessageswillonlybeloggediftheserverisstartedwithloglevelsdebugortrace.
Stringsubstitutionpatterns,whichcanbeusedinformat.
%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink
Examples
console.debug("%s","thisisatest");
console.dir
console.dir(object)
Logsalistingofallpropertiesoftheobject.
Exampleusage:
console.dir(myObject);
console.errorconsole.error(format,argument1,...)
Formatstheargumentsaccordingto@FA{format}andlogstheresultaserrormessage.
Stringsubstitutionpatterns,whichcanbeusedinformat.
%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink
Exampleusage:
console.error("error'%s':%s",type,message);
console
647
console.getline
console.getline()
Readsinalinefromtheconsoleandreturnsitasstring.
console.groupconsole.group(format,argument1,...)
Formatstheargumentsaccordingtoformatandlogstheresultaslogmessage.Opensanestedblocktoindentallfuturemessagessent.CallgroupEndtoclosetheblock.Representationofblockisuptotheplatform,itcanbeaninteractiveblockorjustasetofindentedsubmessages.
Exampleusage:
console.group("userattributes");
console.log("name",user.name);
console.log("id",user.id);
console.groupEnd();
console.groupCollapsedconsole.groupCollapsed(format,argument1,...)
Sameasconsole.group.
console.groupEnd
console.groupEnd()
Closesthemostrecentlyopenedblockcreatedbyacalltogroup.
console.infoconsole.info(format,argument1,...)
Formatstheargumentsaccordingtoformatandlogstheresultasinfomessage.
Stringsubstitutionpatterns,whichcanbeusedinformat.
%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink
Exampleusage:
console.info("The%sjumpedover%dfences",animal,count);
console.log
console.log(format,argument1,...)
Formatstheargumentsaccordingtoformatandlogstheresultaslogmessage.Thisisanaliasforconsole.info.
console.timeconsole.time(name)
Createsanewtimerunderthegivenname.CalltimeEndwiththesamenametostopthetimerandlogthetimeelapsed.
Exampleusage:
console.time("mytimer");
...
console
648
console.timeEnd("mytimer");//thiswillprinttheelapsedtime
console.timeEndconsole.timeEnd(name)
Stopsatimercreatedbyacalltotimeandlogsthetimeelapsed.
console.timeEnd
console.trace()
LogsastacktraceofJavaScriptexecutionatthepointwhereitiscalled.
console.warnconsole.warn(format,argument1,...)
Formatstheargumentsaccordingtoformatandlogstheresultaswarnmessage.
Stringsubstitutionpatterns,whichcanbeusedinformat.
%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink
console
649
CryptoModuleconstcrypto=require('@arangodb/crypto')
Thecryptomoduleprovidesimplementationsofvarioushashingalgorithmsaswellascryptographyrelatedfunctions.
Nonces
Thesefunctionsdealwithcryptographicnonces.
createNonce
crypto.createNonce():string
Createsacryptographicnonce.
Returnsthecreatednonce.
checkAndMarkNonce
crypto.checkAndMarkNonce(nonce):void
Checksandmarksanonce.
Arguments
nonce:string
Thenoncetocheckandmark.
Returnsnothing.
Randomvalues
Thefollowingfunctionsdealwithgeneratingrandomvalues.
rand
crypto.rand():number
Generatesarandomintegerthatmaybepositive,negativeorevenzero.
Returnsthegeneratednumber.
genRandomAlphaNumbers
crypto.genRandomAlphaNumbers(length):string
Generatesastringofrandomalpabeticalcharactersanddigits.
Arguments
length:number
Thelengthofthestringtogenerate.
Returnsthegeneratedstring.
genRandomNumbers
crypto.genRandomNumbers(length):string
Generatesastringofrandomdigits.
crypto
650
Arguments
length:number
Thelengthofthestringtogenerate.
Returnsthegeneratedstring.
genRandomSalt
crypto.genRandomSalt(length):string
Generatesastringofrandom(printable)ASCIIcharacters.
Arguments
length:number
Thelengthofthestringtogenerate.
Returnsthegeneratedstring.
JSONWebTokens(JWT)
ThesemethodsimplementtheJSONWebTokenstandard.
jwtEncode
crypto.jwtEncode(key,message,algorithm):string
GeneratesaJSONWebTokenforthegivenmessage.
Arguments
key:string|null
Thesecretcryptographickeytobeusedtosignthemessageusingthegivenalgorithm.Notethatthisfunctionwillraiseanerrorifthekeyisomittedbutthealgorithmexpectsakey,andalsoifthealgorithmdoesnotexpectakeybutakeyisprovided(e.g.whenusing"none").
message:string
MessagetobeencodedasJWT.Notethatthemessagewillonlybebase64-encodedandsigned,notencrypted.Donotstoresensitiveinformationintokensunlesstheywillonlybehandledbytrustedparties.
algorithm:string
Nameofthealgorithmtouseforsigningthemessage,e.g."HS512".
ReturnstheJSONWebToken.
jwtDecode
crypto.jwtDecode(key,token,noVerify):string|null
Arguments
key:string|null
Thesecretcryptographickeythatwasusedtosignthemessageusingthealgorithmindicatedbythetoken.Notethatthisfunctionwillraiseanerrorifthekeyisomittedbutthealgorithmexpectsakey.
Ifthealgorithmdoesnotexpectakeybutakeyisprovided,thetokenwillfailtoverify.
token:string
Thetokentodecode.
crypto
651
Notethatthefunctionwillraiseanerrorifthetokenismalformed(e.g.doesnothaveexactlythreesegments).
noVerify:boolean(Default:false)
Whetherverificationshouldbeskipped.Ifthisissettotruethesignatureofthetokenwillnotbeverified.Otherwisethefunctionwillraiseanerrorifthesignaturecannotbeverifiedusingthegivenkey.
ReturnsthedecodedJSONmessageornullifnotokenisprovided.
jwtAlgorithms
AhelperobjectcontainingthesupportedJWTalgorithms.EachattributenamecorrespondstoaJWTalgandthevalueisanobjectwithsignandverifymethods.
jwtCanonicalAlgorithmName
crypto.jwtCanonicalAlgorithmName(name):string
AhelperfunctionthattranslatesaJWTalgvaluefoundinaJWTheaderintothecanonicalnameofthealgorithminjwtAlgorithms.Raisesanerrorifnoalgorithmwithamatchingnameisfound.
Arguments
name:string
Algorithmnametolookup.
Returnsthecanonicalnameforthealgorithm.
Hashingalgorithms
md5crypto.md5(message):string
HashesthegivenmessageusingtheMD5algorithm.
Arguments
message:string
Themessagetohash.
Returnsthecryptographichash.
sha1
crypto.sha1(message):string
HashesthegivenmessageusingtheSHA-1algorithm.
Arguments
message:string
Themessagetohash.
Returnsthecryptographichash.
sha224
crypto.sha224(message):string
HashesthegivenmessageusingtheSHA-224algorithm.
Arguments
crypto
652
message:string
Themessagetohash.
Returnsthecryptographichash.
sha256crypto.sha256(message):string
HashesthegivenmessageusingtheSHA-256algorithm.
Arguments
message:string
Themessagetohash.
Returnsthecryptographichash.
sha384crypto.sha384(message):string
HashesthegivenmessageusingtheSHA-384algorithm.
Arguments
message:string
Themessagetohash.
Returnsthecryptographichash.
sha512
crypto.sha512(message):string
HashesthegivenmessageusingtheSHA-512algorithm.
Arguments
message:string
Themessagetohash.
Returnsthecryptographichash.
Miscellaneous
constantEqualscrypto.constantEquals(str1,str2):boolean
Comparestwostrings.Thisfunctioniteratesovertheentirelengthofbothstringsandcanhelpmakingcertaintimingattacksharder.
Arguments
str1:string
Thefirststringtocompare.
str2:string
Thesecondstringtocompare.
Returnstrueifthestringsareequal,falseotherwise.
pbkdf2
crypto
653
crypto.pbkdf2(salt,password,iterations,keyLength):string
GeneratesaPBKDF2-HMAC-SHA1hashofthegivenpassword.
Arguments
salt:string
Thecryptographicsalttohashthepasswordwith.
password:string
Themessageorpasswordtohash.
iterations:number
Thenumberofiterations.Thisshouldbeaveryhighnumber.OWASPrecommended64000iterationsin2012andrecommendsdoublingthatnumbereverytwoyears.
WhenusingPBKDF2forpasswordhashesitisalsorecommendedtoaddarandomvalue(typicallybetween0and32000)tothatnumberthatisdifferentforeachuser.
keyLength:number
Thekeylength.
Returnsthecryptographichash.
hmac
crypto.hmac(key,message,algorithm):string
GeneratesanHMAChashofthegivenmessage.
Arguments
key:string
Thecryptographickeytousetohashthemessage.
message:string
Themessagetohash.
algorithm:string
Thenameofthealgorithmtouse.
Returnsthecryptographichash.
crypto
654
FilesystemModuleconstfs=require('fs')
TheimplementationtriestofollowtheCommonJSFilesystem/A/0specificationwherepossible.
WorkingDirectory
Thedirectoryfunctionsbelowshouldn'tusethecurrentworkingdirectoryoftheserverlike.or./test.Youwillnotbeabletotellwhethertheenvironmenttheserverisrunninginwillpermitdirectorylisting,readingorwritingoffiles.
YoushouldeitherbaseyourdirectorieswithgetTempPath(),orasafoxxserviceusethemodule.context.basePath.
SingleFileDirectoryManipulation
exists
checksifafileofanytypeordirectoryexistsfs.exists(path)
Returnstrueifafile(ofanytype)oradirectoryexistsatagivenpath.Ifthefileisabrokensymboliclink,returnsfalse.
isFile
testsifpathisafilefs.isFile(path)
Returnstrueifthepathpointstoafile.
isDirectory
testsifpathisadirectoryfs.isDirectory(path)
Returnstrueifthepathpointstoadirectory.
size
getsthesizeofafilefs.size(path)
Returnsthesizeofthefilespecifiedbypath.
mtime
getsthelastmodificationtimeofafilefs.mtime(filename)
Returnsthelastmodificationdateofthespecifiedfile.ThedateisreturnedasaUnixtimestamp(numberofsecondselapsedsinceJanuary11970).
pathSeparatorfs.pathSeparator
Ifyouwanttocombinetwopathsyoucanusefs.pathSeparatorinsteadof/or\.
join
fs.join(path,filename)
Thefunctionreturnsthecombinationofthepathandfilename,e.g.fs.join(Hello/World,foo.bar)wouldreturnHello/World/foo.bar.
getTempFile
fs
655
returnsthenamefora(new)temporaryfilefs.getTempFile(directory,createFile)
Returnsthenameforanewtemporaryfileindirectorydirectory.IfcreateFileistrue,anemptyfilewillbecreatedsonootherprocesscancreateafileofthesamename.
Note:Thedirectorydirectorymustexist.
getTempPath
returnsthetemporarydirectoryfs.getTempPath()
Returnstheabsolutepathofthetemporarydirectory
makeAbsolute
makesagivenpathabsolutefs.makeAbsolute(path)
Returnsthegivenstringifitisanabsolutepath,otherwiseanabsolutepathtothesamelocationisreturned.
chmod
setsfilepermissionsofspecifiedfiles(nonwindowsonly)fs.exists(path)
Returnstrueonsuccess.
list
returnsthedirectorylistingfs.list(path)
Thefunctionsreturnsthenamesofallthefilesinadirectory,inlexicallysortedorder.Throwsanexceptionifthedirectorycannotbetraversed(orpathisnotadirectory).
Note:thismeansthatlist("x")ofadirectorycontaining"a"and"b"wouldreturn["a","b"],not["x/a","x/b"].
listTree
returnsthedirectorytreefs.listTree(path)
Thefunctionreturnsanarraythatstartswiththegivenpath,andallofthepathsrelativetothegivenpath,discoveredbyadepthfirsttraversalofeverydirectoryinanyvisiteddirectory,reportingbutnottraversingsymboliclinkstodirectories.Thefirstpathisalways"" ,thepathrelativetoitself.
makeDirectory
createsadirectoryfs.makeDirectory(path)
Createsthedirectoryspecifiedbypath.
makeDirectoryRecursive
createsadirectoryfs.makeDirectoryRecursive(path)
Createsthedirectoryhierarchyspecifiedbypath.
remove
removesafilefs.remove(filename)
Removesthefilefilenameatthegivenpath.Throwsanexceptionifthepathcorrespondstoanythingthatisnotafileorasymboliclink.If"path"referstoasymboliclink,removesthesymboliclink.
removeDirectory
fs
656
removesanemptydirectoryfs.removeDirectory(path)
Removesadirectoryifitisempty.Throwsanexceptionifthepathisnotanemptydirectory.
removeDirectoryRecursive
removesadirectoryfs.removeDirectoryRecursive(path)
Removesadirectorywithallsubelements.Throwsanexceptionifthepathisnotadirectory.
FileIO
read
readsinafilefs.read(filename)
Readsinafileandreturnsthecontentasstring.PleasenotethatthefilecontentmustbeencodedinUTF-8.
read64
readsinafileasbase64fs.read64(filename)
Readsinafileandreturnsthecontentasstring.ThefilecontentisBase64encoded.
readBuffer
readsinafilefs.readBuffer(filename)
ReadsinafileandreturnsitscontentinaBufferobject.
readFileSyncfs.readFileSync(filename,encoding)
Readsthecontentsofthefilespecifiedinfilename.Ifencodingisspecified,thefilecontentswillbereturnedasastring.Supportedencodingsare:
utf8orutf-8ascii
base64
ucs2orucs-2utf16leorutf16behex
Ifnoencodingisspecified,thefilecontentswillbereturnedinaBufferobject.
write
fs.write(filename,content)
Writesthecontentintoafile.ContentcanbeastringoraBufferobject.Ifthefilealreadyexists,itistruncated.
writeFileSync
fs.writeFileSync(filename,content)
Thisisanaliasforfs.write(filename,content).
appendfs.append(filename,content)
Writesthecontentintoafile.ContentcanbeastringoraBufferobject.Ifthefilealreadyexists,thecontentisappendedattheend.
fs
657
RecursiveManipulation
copyRecursive
copiesadirectorystructurefs.copyRecursive(source,destination)
Copiessourcetodestination.Exceptionswillbethrownon:
Failuretocopythefilespecifyingadirectoryfordestinationwhensourceisafilespecifyingadirectoryassourceanddestination
CopyFile
copiesafileintoatargetfilefs.copyFile(source,destination)
Copiessourcetodestination.IfDestinationisadirectory,afileofthesamenamewillbecreatedinthatdirectory,elsethecopywillgetthespecifiedfilename.
move
renamesafilefs.move(source,destination)
Movessourcetodestination.Failuretomovethefile,orspecifyingadirectoryfordestinationwhensourceisafilewillthrowanexception.Likewise,specifyingadirectoryassourceanddestinationwillfail.
ZIP
unzipFile
unzipsafilefs.unzipFile(filename,outpath,skipPaths,overwrite,password)
Unzipsthezipfilespecifiedbyfilenameintothepathspecifiedbyoutpath.Overwritesanyexistingtargetfilesifoverwriteissettotrue.
Returnstrueifthefilewasunzippedsuccessfully.
zipFile
zipsafilefs.zipFile(filename,chdir,files,password)
Storesthefilesspecifiedbyfilesinthezipfilefilename.Ifthefilefilenamealreadyexists,anerroristhrown.Thelistofinputfilesfilesmustbegivenasalistofabsolutefilenames.Ifchdirisnotempty,thechdirprefixwillbestrippedfromthefilenameinthezipfile,sowhenitisunzippedfilenameswillberelative.Specifyingapasswordisoptional.
Returnstrueifthefilewaszippedsuccessfully.
fs
658
RequestModuleconstrequest=require('@arangodb/request')
TherequestmoduleprovidesthefunctionalityformakingHTTPrequests.
MakingHTTPrequests
HTTPmethodhelpers
InadditiontotherequestfunctionconvenienceshorthandsareavailableforeachHTTPmethodintheformof,i.e.:
request.head(url,options)
request.get(url,options)
request.post(url,options)
request.put(url,options)
request.delete(url,options)
request.patch(url,options)
Theseareequivalenttousingtherequestfunctiondirectly,i.e.:
request[method](url,options)
//isequivalentto
request({method,url,...options});
Forexample:
constrequest=require('@arangodb/request');
request.get('http://localhost',{headers:{'x-session-id':'keyboardcat'}});
//isequivalentto
request({
method:'get',
url:'http://localhost',
headers:{'x-session-id':'keyboardcat'}
});
Therequestfunction
TherequestfunctioncanbeusedtomakeHTTPrequests.
request(options)
PerformsanHTTPrequestandreturnsaResponseobject.
Parameter
Therequestfunctiontakesthefollowingoptions:
urloruri:thefully-qualifiedURLoraparsedURLfromurl.parse.qs(optional):objectcontainingquerystringvaluestobeappendedtotheURL.useQuerystring:iftrue,usequerystringmoduletohandlequerystrings,otherwiseuseqsmodule.Default:false.method(optional):HTTPmethod(case-insensitive).Default:"GET".headers(optional):HTTPheaders(case-insensitive).Default:{}.body(optional):requestbody.MustbeastringorBuffer,oraJSONserializablevalueifjsonistrue.json:iftrue,bodywillbeserializedtoaJSONstringandtheContent-Typeheaderwillbesetto"application/json".AdditionallytheresponsebodywillalsobeparsedasJSON(unlessencodingissettonull).Default:false.form(optional):whensettoastringorobjectandnobodyhasbeenset,bodywillbesettoaquerystringrepresentationofthatvalueandtheContent-Typeheaderwillbesetto"application/x-www-form-urlencoded".AlsoseeuseQuerystring.auth(optional):anobjectwiththepropertiesusernameandpasswordforHTTPBasicauthenticationorthepropertybearerfor
request
659
HTTPBearertokenauthentication.
sslProtocol(optional):whichtlsversionshouldbeusedtoconnecttotheurl.Thedefaultis4whichisTLS1.0.Seesslprotocolformoreopitions.followRedirect:whetherHTTP3xxredirectsshouldbefollowed.Default:true.maxRedirects:themaximumnumberofredirectstofollow.Default:10.encoding:encodingtobeusedfortheresponsebody.Ifsettonull,theresponsebodywillbereturnedasaBuffer.Default:"utf-8".timeout:numberofmillisecondstowaitforaresponsebeforeabortingtherequest.returnBodyOnError:whethertheresponsebodyshouldbereturnedevenwhentheserverresponseindicatesanerror.Default:true.
ThefunctionreturnsaResponseobjectwiththefollowingproperties:
rawBody:therawresponsebodyasaBuffer.body:theparsedresponsebody.Ifencodingwasnotsettonull,thisisastring.Ifadditionallyjsonwassettotrueandtheresponsebodyiswell-formedJSON,thisistheparsedJSONdata.headers:anobjectcontainingtheresponseheaders.OtherwisethisisidenticaltorawBody.statusCodeandstatus:theHTTPstatuscodeoftheresponse,e.g.404.message:theHTTPstatusmessageoftheresponse,e.g.NotFound.
Forms
Therequestmodulesupportsapplication/x-www-form-urlencoded(URLencoded)formuploads:
constrequest=require('@arangodb/request');
varres=request.post('http://service.example/upload',{form:{key:'value'}});
//or
varres=request.post({url:'http://service.example/upload',form:{key:'value'}});
//or
varres=request({
method:'post',
url:'http://service.example/upload',
form:{key:'value'}
});
Formdatawillbeencodedusingtheqsmodulebydefault.
Ifyouwanttousethequerystringmoduleinstead,simplyusetheuseQuerystringoption.
JSON
IfyouwanttosubmitJSON-serializablevaluesasrequestbodies,justsetthejsonoption:
constrequest=require('@arangodb/request');
varres=request.post('http://service.example/notify',{body:{key:'value'},json:true});
//or
varres=request.post({url:'http://service.example/notify',body:{key:'value'},json:true});
//or
varres=request({
method:'post',
url:'http://service.example/notify',
body:{key:'value'},
json:true
});
HTTPauthentication
TherequestmodulesupportsbothHTTPBasicauthentication.Justpassthecredentialsviatheauthoption:
constrequest=require('@arangodb/request');
varres=request.get(
request
660
'http://service.example/secret',
{auth:{username:'jcd',password:'bionicman'}}
);
//or
varres=request.get({
url:'http://service.example/secret',
auth:{username:'jcd',password:'bionicman'}
});
//or
varres=request({
method:'get',
url:'http://service.example/secret',
auth:{username:'jcd',password:'bionicman'}
});
AlternativelyyoucansupplythecredentialsviatheURL:
constrequest=require('@arangodb/request');
varusername='jcd';
varpassword='bionicman';
varres=request.get(
'http://'+
encodeURIComponent(username)+
':'+
encodeURIComponent(password)+
'@service.example/secret'
);
YoucanalsouseBearertokenauthentication:
constrequest=require('@arangodb/request');
varres=request.get(
'http://service.example/secret',
{auth:{bearer:'keyboardcat'}}
);
//or
varres=request.get({
url:'http://service.example/secret',
auth:{bearer:'keyboardcat'}
});
//or
varres=request({
method:'get',
url:'http://service.example/secret',
auth:{bearer:'keyboardcat'}
});
request
661
Module"actions"constactions=require('@arangodb/actions')
Theactionmoduleprovidestheinfrastructurefordefininglow-levelHTTPactions.
IfyouwanttodefineHTTPendpointsinArangoDByoushouldprobablyusetheFoxxmicroserviceframeworkinstead.
Basics
Errormessage
actions.getErrorMessage(code)
Returnstheerrormessageforanerrorcode.
StandardHTTPResultGeneratorsactions.defineHttp(options)
Definesanewaction.Theoptionsareasfollows:
options.url
TheURL,whichcanbeusedtoaccesstheaction.Thispathmightcontainslashes.Notethatthisactionwillalsobecalled,ifaurlisgivensuchthatoptions.urlisaprefixofthegivenurlandnolongerdefinitionmatches.
options.prefix
Iffalse,thenonlyusetheactionforexactmatches.Thedefaultistrue.
options.callback(request,response)
Therequestargumentcontainsadescriptionoftherequest.Arequestparameterfooisaccessibleasrequest.parametrs.foo.Arequestheaderbarisaccessibleasrequest.headers.bar.Assumethattheactionisdefinedfortheurl/foo/barandtherequesturlis/foo/bar/hugo/egon.Thenthesuffixparts["hugo","egon"]areavailibleinrequest.suffix.
Thecallbackmustdefinefilltheresponse.
response.responseCode:theresponsecoderesponse.contentType:thecontenttypeoftheresponseresponse.body:thebodyoftheresponse
YoucanusethefunctionsResultOkandResultErrortoeasilygeneratearesponse.
Resultok
actions.resultOk(req,res,code,result,headers)
Thefunctiondefinesaresponse.codeisthestatuscodetoreturn.resultistheresultobject,whichwillbereturnedasJSONobjectinthebody.headersisanarrayofheaderstoreturned.Thefunctionaddstheattributeerrorwithvaluefalseandcodewithvaluecodetotheresult.
Resultbad
actions.resultBad(req,res,error-code,msg,headers)
Thefunctiongeneratesanerrorresponse.
Resultnotfoundactions.resultNotFound(req,res,code,msg,headers)
actions
662
Thefunctiongeneratesanerrorresponse.
Resultunsupported
actions.resultUnsupported(req,res,headers)
Thefunctiongeneratesanerrorresponse.
Resulterror
actions.resultError(req,res,code,errorNum,errorMessage,headers,keyvals)*
Thefunctiongeneratesanerrorresponse.TheresponsebodyisanarraywithanattributeerrorMessagecontainingtheerrormessageerrorMessage,errorcontainingtrue,codecontainingcode,errorNumcontainingerrorNum,anderrorMessagecontainingtheerrormessageerrorMessage.keyvalsaremixedintotheresult.
ResultnotImplementedactions.resultNotImplemented(req,res,msg,headers)
Thefunctiongeneratesanerrorresponse.
Resultpermanentredirect
actions.resultPermanentRedirect(req,res,options,headers)
Thefunctiongeneratesaredirectresponse.
Resulttemporaryredirectactions.resultTemporaryRedirect(req,res,options,headers)
Thefunctiongeneratesaredirectresponse.
ArangoDBResultGenerators
Collectionnotfoundactions.collectionNotFound(req,res,collection,headers)
Thefunctiongeneratesanerrorresponse.
Indexnotfound
actions.indexNotFound(req,res,collection,index,headers)
Thefunctiongeneratesanerrorresponse.
Resultexceptionactions.resultException(req,res,err,headers,verbose)
Thefunctiongeneratesanerrorresponse.If@FA{verbose}issettotrueornotspecified(thedefault),thentheerrorstacktracewillbeincludedintheerrormessageifavailable.If@FA{verbose}isastringitwillbeprependedbeforetheerrormessageandthestacktracewillalsobeincluded.
actions
663
QueriesModuleconstqueries=require('@arangodb/aql/queries')
ThequerymoduleprovidestheinfrastructureforworkingwithcurrentlyrunningAQLqueriesviaarangosh.
Properties
queries.properties()Returnstheserverscurrentquerytrackingconfiguration;wechangetheslowquerythresholdtogetbetterresults:
arangosh>varqueries=require("@arangodb/aql/queries");
arangosh>queries.properties();
arangosh>queries.properties({slowQueryThreshold:1});
showexecutionresults
Currentlyrunningqueries
Wecreateataskthatspawnsqueries,sowehaveniceoutput.Sincethistaskusesresources,youmaywanttoincreaseperiod(andnotforgettoremoveit...afterwards):
arangosh>vartheQuery='FORsleepLoooongIN1..5LETsleepLoooonger=SLEEP(1000)RETURN
sleepLoooong';
arangosh>vartasks=require("@arangodb/tasks");
arangosh>tasks.register({
........>id:"mytask-1",
........>name:"thisisasampletasktospawnaslowaqlquery",
........>command:"require('@arangodb').db._query('"+theQuery+"');"
........>});
arangosh>queries.current();
showexecutionresultsThefunctionreturnsthecurrentlyrunningAQLqueriesasanarray.
SlowqueriesThefunctionreturnsthelastAQLqueriesthatexceededtheslowquerythresholdasanarray:
arangosh>queries.slow();
[]
ClearslowqueriesClearthelistofslowAQLqueries:
arangosh>queries.clearSlow();
{
"code":200
}
arangosh>queries.slow();
[]
queries
664
KillKillarunningAQLquery:
arangosh>varrunningQueries=queries.current().filter(function(query){
........>returnquery.query===theQuery;
........>});
arangosh>queries.kill(runningQueries[0].id);
{
"code":200
}
queries
665
Write-aheadlogconstwal=require('internal').wal
Thismoduleprovidesfunctionalityforadministeringthewrite-aheadlogs.Mostofthesefunctionsonlyreturnsensiblevalueswheninvokedwiththemmfilesenginebeingactive.
Configuration
retrievestheconfigurationofthewrite-aheadloginternal.wal.properties()
Retrievestheconfigurationofthewrite-aheadlog.TheresultisaJSONarraywiththefollowingattributes:
allowOversizeEntries:whetherornotoperationsthatarebiggerthanasinglelogfilecanbeexecutedandstoredlogfileSize:thesizeofeachwrite-aheadlogfilehistoricLogfiles:themaximumnumberofhistoriclogfilestokeepreserveLogfiles:themaximumnumberofreservelogfilesthatArangoDBallocatesinthebackgroundsyncInterval:theintervalforautomaticsynchronizationofnot-yetsynchronizedwrite-aheadlogdata(inmilliseconds)throttleWait:themaximumwaittimethatoperationswillwaitbeforetheygetabortedifcaseofwrite-throttling(inmilliseconds)throttleWhenPending:thenumberofunprocessedgarbage-collectionoperationsthat,whenreached,willactivatewrite-throttling.Avalueof0meansthatwrite-throttlingwillnotbetriggered.
Examples
arangosh>require("internal").wal.properties();
showexecutionresultsconfiguresthewrite-aheadloginternal.wal.properties(properties)
Configuresthebehaviorofthewrite-aheadlog.propertiesmustbeaJSONJSONobjectwiththefollowingattributes:
allowOversizeEntries:whetherornotoperationsthatarebiggerthanasinglelogfilecanbeexecutedandstoredlogfileSize:thesizeofeachwrite-aheadlogfilehistoricLogfiles:themaximumnumberofhistoriclogfilestokeepreserveLogfiles:themaximumnumberofreservelogfilesthatArangoDBallocatesinthebackgroundthrottleWait:themaximumwaittimethatoperationswillwaitbeforetheygetabortedifcaseofwrite-throttling(inmilliseconds)throttleWhenPending:thenumberofunprocessedgarbage-collectionoperationsthat,whenreached,willactivatewrite-throttling.Avalueof0meansthatwrite-throttlingwillnotbetriggered.
Specifyinganyoftheaboveattributesisoptional.Notspecifiedattributeswillbeignoredandtheconfigurationforthemwillnotbemodified.
Examples
arangosh>require("internal").wal.properties({
........>allowOverSizeEntries:true,
........>logfileSize:32*1024*1024});
showexecutionresults
Flushing
flushesthecurrentlyopenWALlogfileinternal.wal.flush(waitForSync,waitForCollector)
Flushesthewrite-aheadlog.Byflushingthecurrentlyactivewrite-aheadlogfile,thedatainitcanbetransferredtocollectionjournalsanddatafiles.Thisisusefultoensurethatalldataforacollectionispresentinthecollectionjournalsanddatafiles,forexample,whendumpingthedataofacollection.
Write-aheadlog
666
ThewaitForSyncoptiondetermineswhetherornottheoperationshouldblockuntilthenot-yetsynchronizeddatainthewrite-aheadlogwassynchronizedtodisk.
ThewaitForCollectoroperationcanbeusedtospecifythattheoperationshouldblockuntilthedataintheflushedloghasbeencollectedbythewrite-aheadloggarbagecollector.Notethatsettingthisoptiontotruemightblockforalongtimeiftherearelong-runningtransactionsandthewrite-aheadloggarbagecollectorcannotfinishgarbagecollection.
Examples
arangosh>require("internal").wal.flush();
Write-aheadlog
667
TaskManagementconsttasks=require('@arangodb/tasks')
Note:IfyouaretryingtoscheduletasksinFoxxyoushouldconsiderusingtheFoxxqueuesmoduleinstead,whichprovidesamorehigh-levelAPIthatalsopersiststasksacrossreboots.
IntroductiontoTaskManagementinArangoDB
ArangoDBcanexecuteuser-definedJavaScriptfunctionsasone-shotorperiodictasks.Thisfunctionalitycanbeusedtoimplementtimedorrecurringjobsinthedatabase.
TasksinArangoDBconsistofaJavaScriptsnippetorfunctionthatisexecutedwhenthetaskisscheduled.Ataskcanbeaone-shottask(meaningitisrunonceandnotrepeated)oraperiodictask(meaningthatitisre-scheduledaftereachexecution).Taskscanhaveoptionalparameters,whicharedefinedattasksetuptime.Theparametersspecifiedattasksetuptimewillbepassedasargumentstothetaskwheneveritgetsexecuted.PeriodicTaskshaveanexecutionfrequencythatneedstobespecifiedwhenthetaskissetup.One-shottaskshaveaconfigurabledelayafterwhichthey'llgetexecuted.
Taskswillbeexecutedontheservertheyhavebeensetupon.Taskswillnotbeshippedaroundinacluster.Ataskwillbeexecutedinthecontextofthedatabaseitwascreatedin.However,whendroppingadatabase,anytasksthatwerecreatedinthecontextofthisdatabasewillremainactive.Itisthereforesensibletofirstunregisterallactivetasksforadatabasebeforedroppingthedatabase.
TasksregisteredinArangoDBwillbeexecuteduntiltheservergetsshutdownorrestarted.Afterarestartoftheserver,anyuser-definedone-shotorperiodictaskswillbelost.
CommandsforWorkingwithTasksArangoDBprovidesthefollowingcommandsforworkingwithtasks.Allcommandscanbeaccessedviathetasksmodule,whichcanbeloadedlikethis:
require("@arangodb/tasks")
PleasenotethatthetasksmoduleisavailableinsidetheArangoDBserveronly.ItcannotbeusedfromtheArangoShellorArangoDB'swebinterface.
Registeratask
Toregisteratask,theJavaScriptsnippetorfunctionneedstobespecifiedinadditiontotheexecutionfrequency.Optionally,ataskcanhaveanidandaname.Ifnoidisspecified,itwillbeauto-assignedforanewtask.Thetaskidisalsothemeanstoaccessorunregisteratasklater.Tasknamesareinformationalonly.Theycanbeusedtomakeataskdistinguishablefromothertasksalsorunningontheserver.
Thefollowingserver-sidecommandsregisteratask.ThecommandtobeexecutedisaJavaScriptstringsnippetwhichprintsamessagetotheserver'slogfile:
consttasks=require("@arangodb/tasks");
tasks.register({
id:"mytask-1",
name:"thisisasnippettask",
period:15,
command:"require('console').log('hellofromsnippettask');"
});
Theabovehasregisterataskwithidmytask-1,whichwillbeexecutedevery15secondsontheserver.Thetaskwillwritealogmessagewheneveritisinvoked.
TaskscanalsobesetupusingaJavaScriptcallbackfunctionlikethis:
TaskManagement
668
consttasks=require("@arangodb/tasks");
tasks.register({
id:"mytask-2",
name:"thisisafunctiontask",
period:15,
command:function(){
require('console').log('hellofromfunctiontask');
}
});
Itisimportanttonotethatthecallbackfunctionislateboundandwillbeexecutedinadifferentcontextthaninthecreationcontext.Thecallbackfunctionmustthereforenotaccessanyvariablesdefinedoutsideofitsownscope.Thecallbackfunctioncanstilldefineanduseitsownvariables.
Topassparameterstoatask,theparamsattributecanbesetwhenregisteringatask.NotethattheparametersarelimitedtodatatypesusableinJSON(meaningnocallbackfunctionscanbepassedasparametersintoatask):
consttasks=require("@arangodb/tasks");
tasks.register({
id:"mytask-3",
name:"thisisaparametertask",
period:15,
command:function(params){
vargreeting=params.greeting;
vardata=JSON.stringify(params.data);
require('console').log('%sfromparametertask:%s',greeting,data);
},
params:{greeting:"hi",data:"howareyou?"}
});
Registeringaone-shottaskworksthesameway,exceptthattheperiodattributemustbeomitted.Ifperiodisomitted,thenthetaskwillbeexecutedjustonce.Thetaskinvocationdelaycanoptionallybespecifiedwiththeoffsetattribute:
consttasks=require("@arangodb/tasks");
tasks.register({
id:"mytask-once",
name:"thisisaone-shottask",
offset:10,
command:function(params){
require('console').log('youwillseemejustonce!');
}
});
Note:Whenspecifyinganoffsetvalueof0,ArangoDBwillinternallyaddaverysmallvaluetotheoffsetsowillbeslightlygreaterthanzero.
Unregisteratask
Afterataskhasbeenregistered,itcanbeunregisteredusingitsid:
consttasks=require("@arangodb/tasks");
tasks.unregister("mytask-1");
Notethatunregisteringanon-existingtaskwillthrowanexception.
ListalltasksTogetanoverviewofwhichtasksareregistered,thereisthegetmethod.Ifthegetmethodiscalledwithoutanyarguments,itwillreturnanarrayofalltasks:
TaskManagement
669
consttasks=require("@arangodb/tasks");
tasks.get();
Ifgetiscalledwithataskidargument,itwillreturninformationaboutthisparticulartask:
consttasks=require("@arangodb/tasks");
tasks.get("mytask-3");
Thecreatedattributeofataskrevealswhenataskwascreated.ItisreturnedasaUnixtimestamp.
TaskManagement
670
DeprecatedFeatureslistedinthissectionshouldnolongerbeused,becausetheyareconsideredobsoleteandmaygetremovedinafuturerelease.Theyarecurrentlykeptforbackwardcompatibility.Thereareusuallybetteralternativestoreplacetheoldfeatureswith:
SimpleQueries:Ideomaticinterfaceinarangoshtoperformtrivialqueries.TheyaresupersededbyAQLqueries,whichcanalsoberuninarangosh.AQLisalanguageonitsownandwaymorepowerfulthanSimpleQueriescouldeverbe.Infact,the(stillsupported)SimpleQueriesaretranslatedinternallytoAQL,thentheAQLqueryisoptimizedandrunagainstthedatabaseinrecentversions,becauseofbetterperformanceandreducedmaintenancecomplexity.
Actions:SnippetsofJavaScriptcodeontheserver-sideforminimalcustomendpoints.SincetheFoxxrevampin3.0,itbecamereallyeasytowriteFoxxMicroservices,whichallowyoutodefinecustomendpointsevenwithcomplexbusinesslogic.
Deprecated
671
SimpleQueries
ItisrecommendedtouseAQLinstead.
Simplequeriescanbeusedifthequeryconditionisstraightforward,i.e.,adocumentreference,alldocuments,aquery-by-example,orasimplegeoquery.Inasimplequeryyoucanspecifyexactlyonecollectionandonequerycriteria.InthefollowingsectionswedescribetheJavaScriptshellinterfaceforsimplequeries,whichyoucanusewithintheArangoDBshellandwithinactionsandtransactions.ForotherlanguagesseethecorrespondinglanguageAPIdocumentation.
YoucanfindalistofqueriesatCollectionMethods.
SimpleQueries
672
SequentialAccessandCursorsIfaqueryreturnsacursor,thenyoucanusehasNextandnexttoiterateovertheresultsetortoArraytoconvertittoanarray.
Ifthenumberofqueryresultsisexpectedtobebig,itispossibletolimittheamountofdocumentstransferredbetweentheserverandtheclienttoaspecificvalue.ThisvalueiscalledbatchSize.ThebatchSizecanoptionallybesetbeforeorwhenasimplequeryisexecuted.Iftheserverhasmoredocumentsthanshouldbereturnedinasinglebatch,theserverwillsetthehasMoreattributeintheresult.Itwillalsoreturntheidoftheserver-sidecursorintheidattributeintheresult.ThisidcanbeusedwiththecursorAPItofetchanyoutstandingresultsfromtheserveranddisposetheserver-sidecursorafterwards.
TheinitialbatchSizevaluecanbesetusingthesetBatchSizemethodthatisavailableforeachtypeofsimplequery,orwhenthesimplequeryisexecutedusingitsexecutemethod.IfnobatchSizevalueisspecified,theserverwillpickareasonabledefaultvalue.
HasNext
checksifthecursorisexhaustedcursor.hasNext()
ThehasNextoperatorreturnstrue,thenthecursorstillhasdocuments.Inthiscasethenextdocumentcanbeaccessedusingthenextoperator,whichwilladvancethecursor.
Examples
arangosh>vara=db.five.all();
arangosh>while(a.hasNext())print(a.next());
showexecutionresults
Next
returnsthenextresultdocumentcursor.next()
IfthehasNextoperatorreturnstrue,thentheunderlyingcursorofthesimplequerystillhasdocuments.Inthiscasethenextdocumentcanbeaccessedusingthenextoperator,whichwilladvancetheunderlyingcursor.Ifyouusenextonanexhaustedcursor,thenundefinedisreturned.
Examples
arangosh>db.five.all().next();
showexecutionresults
SetBatchsize
setsthebatchsizeforanyfollowingrequestscursor.setBatchSize(number)
Setsthebatchsizeforqueries.Thebatchsizedetermineshowmanyresultsareatmosttransferredfromtheservertotheclientinonechunk.
GetBatchsize
returnsthebatchsizecursor.getBatchSize()
Returnsthebatchsizeforqueries.Ifthereturnedvalueisundefined,theserverwilldetermineasensiblebatchsizeforanyfollowingrequests.
ExecuteQuery
executesaqueryquery.execute(batchSize)
SimpleQueries
673
Executesasimplequery.IftheoptionalbatchSizevalueisspecified,theserverwillreturnatmostbatchSizevaluesinoneroundtrip.ThebatchSizecannotbeadjustedafterthequeryisfirstexecuted.
Note:Thereisnoneedtoexplicitlycalltheexecutemethodifanothermeansoffetchingthequeryresultsischosen.Thefollowingtwoapproachesleadtothesameresult:
arangosh>result=db.users.all().toArray();
arangosh>q=db.users.all();q.execute();result=[];while(q.hasNext()){
result.push(q.next());}
showexecutionresultsThefollowingtwoalternativesbothuseabatchSizeandreturnthesameresult:
arangosh>q=db.users.all();q.setBatchSize(20);q.execute();while(q.hasNext()){
print(q.next());}
arangosh>q=db.users.all();q.execute(20);while(q.hasNext()){print(q.next());}
showexecutionresults
Dispose
disposestheresultcursor.dispose()
Ifyouarenolongerinterestedinanyfurtherresults,youshouldcalldisposeinordertofreeanyresourcesassociatedwiththecursor.Aftercallingdisposeyoucannolongeraccessthecursor.
Count
countsthenumberofdocumentscursor.count()
Thecountoperatorcountsthenumberofdocumentintheresultsetandreturnsthatnumber.Thecountoperatorignoresanylimitsandreturnsthetotalnumberofdocumentsfound.
Note:Notallsimplequeriessupportcounting.Inthiscasenullisreturned.
cursor.count(true)
Iftheresultsetwaslimitedbythelimitoperatorordocumentswereskipedusingtheskipoperator,thecountoperatorwithargumenttruewillusethenumberofelementsinthefinalresultset-afterapplyinglimitandskip.
Note:Notallsimplequeriessupportcounting.Inthiscasenullisreturned.
Examples
Ignoreanylimit:
arangosh>db.five.all().limit(2).count();
null
Countinganylimitorskip:
arangosh>db.five.all().limit(2).count(true);
2
SimpleQueries
674
Pagination
ItisrecommendedtouseAQLinstead,seetheLIMIToperation.
If,forexample,youdisplaytheresultofausersearch,thenyouareingeneralnotinterestedinthecompletedresultset,butonlythefirst10orsodocuments.Ormaybethenext10documentsforthesecondpage.Inthiscase,youcantheskipandlimitoperators.TheseoperatorsworklikeLIMITinMySQL.
skipusedtogetherwithlimitcanbeusedtoimplementpagination.Theskipoperatorskipsoverthefirstndocuments.So,inordertocreateresultpageswith10resultdocumentsperpage,youcanuseskip(n*10).limit(10)toaccessthe10documentsonthenthpage.Thisresultshouldbesorted,sothatthepaginationworksinapredicableway.
Limit
limitquery.limit(number)
Limitsaresulttothefirstnumberdocuments.Specifyingalimitof0willreturnnodocumentsatall.Ifyoudonotneedalimit,justdonotaddthelimitoperator.Thelimitmustbenon-negative.
Ingeneraltheinputtolimitshouldbesorted.Otherwiseitwillbeunclearwhichdocumentswillbeincludedintheresultset.
Examples
arangosh>db.five.all().toArray();
arangosh>db.five.all().limit(2).toArray();
showexecutionresults
Skip
skipquery.skip(number)
Skipsthefirstnumberdocuments.Ifnumberispositive,thenthisnumberofdocumentsareskippedbeforereturningthequeryresults.
Ingeneraltheinputtoskipshouldbesorted.Otherwiseitwillbeunclearwhichdocumentswillbeincludedintheresultset.
Note:usingnegativeskipvaluesisdeprecatedasofArangoDB2.6andwillnotbesupportedinfutureversionsofArangoDB.
Examples
arangosh>db.five.all().toArray();
arangosh>db.five.all().skip(3).toArray();
showexecutionresults
SimpleQueries
675
ModificationQueries
ItisrecommendedtouseAQLinstead,seeDataModificationQueries.
ArangoDBalsoallowsremoving,replacing,andupdatingdocumentsbasedonanexampledocument.Everydocumentinthecollectionwillbecomparedagainstthespecifiedexampledocumentandbedeleted/replaced/updatedifallattributesmatch.
Thesemethodshouldbeusedwithcautionastheyareintendedtoremoveormodifylotsofdocumentsinacollection.
Allmethodscanoptionallyberestrictedtoaspecificnumberofoperations.However,ifalimitisspecificbutislessthanthenumberofmatches,itwillbeundefinedwhichofthematchingdocumentswillgetremoved/modified.RemovebyExample,ReplacebyExampleandUpdatebyExamplearedescribedwithexamplesinthesubchapterCollectionMethods.
SimpleQueries
676
GeoQueries
ItisrecommendedtouseAQLinstead,seeGeofunctions.
TheArangoDBallowstoselectdocumentsbasedongeographiccoordinates.Inorderforthistowork,ageo-spatialindexmustbedefined.ThisindexwilluseaveryelaboratealgorithmtolookupneighborsthatisamagnitudefasterthanasimpleR*index.
Ingeneralageocoordinateisapairoflatitudeandlongitude,whichmustbothbespecifiedasnumbers.Ageoindexcanbecreatedoncoordinatesthatarestoredinasinglelistattributewithtwoelementslike[-10,+30](latitudefirst,followedbylongitude)oroncoordinatesstoredintwoseparateattributes.
Forexample,toindexthefollowingdocuments,anindexcanbecreatedonthepositionattributeofthedocuments:
db.test.save({position:[-10,30]});
db.test.save({position:[10,45.5]});
db.test.ensureIndex({type:"geo",fields:["position"]});
Ifcoordinatesarestoredintwodistinctattributes,theindexmustbecreatedonthetwoattributes:
db.test.save({latitude:-10,longitude:30});
db.test.save({latitude:10,longitude:45.5});
db.test.ensureIndex({type:"geo",fields:["latitude","longitude"]});
Inordertofindalldocumentswithinagivenradiusaroundacoordinateusethewithinoperator.Inordertofindalldocumentsnearagivendocumentusethenearoperator.
Itispossibletodefinemorethanonegeo-spatialindexpercollection.Inthiscaseyoumustgiveahintusingthegeooperatorwhichofindexesshouldbeusedinaquery.
Near
constructsanearqueryforacollectioncollection.near(latitude,longitude)
Thereturnedlistissortedaccordingtothedistance,withthenearestdocumenttothecoordinate(latitude,longitude)comingfirst.Ifthereareneardocumentsofequaldistance,documentsarechosenrandomlyfromthissetuntilthelimitisreached.Itispossibletochangethelimitusingthelimitoperator.
Inordertousethenearoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.
Note:neardoesnotsupportnegativeskips.//However,youcanstilluselimitfollowedtoskip.
collection.near(latitude,longitude).limit(limit)
Limitstheresulttolimitdocumentsinsteadofthedefault100.
Note:Unlikewithmultipleexplicitlimits,limitwillraisetheimplicitdefaultlimitimposedbywithin.
collection.near(latitude,longitude).distance()
Thiswilladdanattributedistancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.
collection.near(latitude,longitude).distance(name)
Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.
Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.
SimpleQueries
677
Note:thenearsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthenearoperatoristousetheAQLNEARfunctioninanAQLqueryasfollows:
FORdocINNEAR(@@collection,@latitude,@longitude,@limit)
RETURNdoc
Examples
Togetthenearesttwolocations:
arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});
arangosh>for(vari=-90;i<=90;i+=10){
........>for(varj=-180;j<=180;j+=10){
........>db.geo.save({
........>name:"Name/"+i+"/"+j,
........>loc:[i,j]});
........>}}
arangosh>db.geo.near(0,0).limit(2).toArray();
showexecutionresultsIfyouneedthedistanceaswell,thenyoucanusethedistanceoperator:
arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});
arangosh>for(vari=-90;i<=90;i+=10){
........>for(varj=-180;j<=180;j+=10){
........>db.geo.save({
........>name:"Name/"+i+"/"+j,
........>loc:[i,j]});
........>}}
arangosh>db.geo.near(0,0).distance().limit(2).toArray();
showexecutionresults
Within
constructsawithinqueryforacollectioncollection.within(latitude,longitude,radius)
Thiswillfindalldocumentswithinagivenradiusaroundthecoordinate(latitude,longitude).Thereturnedarrayissortedbydistance,beginningwiththenearestdocument.
Inordertousethewithinoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.
collection.within(latitude,longitude,radius).distance()
Thiswilladdanattribute_distancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.
collection.within(latitude,longitude,radius).distance(name)
Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.
Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.
Note:thewithinsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthewithinoperatoristousetheAQLWITHINfunctioninanAQLqueryasfollows:
FORdocINWITHIN(@@collection,@latitude,@longitude,@radius,@distanceAttributeName)
SimpleQueries
678
RETURNdoc
Examples
Tofindalldocumentswithinaradiusof2000kmuse:
arangosh>for(vari=-90;i<=90;i+=10){
........>for(varj=-180;j<=180;j+=10){
........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});}}
arangosh>db.geo.within(0,0,2000*1000).distance().toArray();
showexecutionresults
Geo
constructsageoindexselectioncollection.geo(location-attribute)
Looksupageoindexdefinedonattributelocation_attribute.
Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.
Thisisusefulforcollectionswithmultipledefinedgeoindexes.
collection.geo(location_attribute,true)
Looksupageoindexonacompoundattributelocation_attribute.
Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.
collection.geo(latitude_attribute,longitude_attribute)
Looksupageoindexdefinedonthetwoattributeslatitude_attributeandlongitude-attribute.
Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.
Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.
Note:thegeosimplequeryhelperfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforrunninggeoqueriesistousetheirAQLequivalents.
Examples
Assumeyouhavealocationstoredaslistintheattributehomeandadestinationstoredintheattributework.Thenyoucanusethegeooperatortoselectwhichgeo-spatialattributes(andthuswhichindex)touseinanearquery.
arangosh>for(i=-90;i<=90;i+=10){
........>for(j=-180;j<=180;j+=10){
........>db.complex.save({name:"Name/"+i+"/"+j,
........>home:[i,j],
........>work:[-i,-j]});
........>}
........>}
........>
arangosh>db.complex.near(0,170).limit(5);
arangosh>db.complex.ensureIndex({type:"geo",fields:["home"]});
arangosh>db.complex.near(0,170).limit(5).toArray();
arangosh>db.complex.geo("work").near(0,170).limit(5);
arangosh>db.complex.ensureIndex({type:"geo",fields:["work"]});
arangosh>db.complex.geo("work").near(0,170).limit(5).toArray();
SimpleQueries
679
showexecutionresults
Relatedtopics
OtherArangoDBgeographicfeaturesaredescribedin:
AQLGeofunctionsGeoindexes
SimpleQueries
680
Fulltextqueries
ItisrecommendedtouseAQLinstead,seeFulltextfunctions.
ArangoDBallowstorunqueriesontextcontainedindocumentattributes.Tousethis,afulltextindexmustbedefinedfortheattributeofthecollectionthatcontainsthetext.Creatingtheindexwillparsethetextinthespecifiedattributeforalldocumentsofthecollection.Onlydocumentswillbeindexedthatcontainatextualvalueintheindexedattribute.Forsuchdocuments,thetextvaluewillbeparsed,andtheindividualwordswillbeinsertedintothefulltextindex.
Whenafulltextindexexists,itcanbequeriedusingafulltextquery.
Fulltext
queriesthefulltextindexcollection.fulltext(attribute,query)
Thefulltextsimplequeryfunctionsperformsafulltextsearchonthespecifiedattributeandthespecifiedquery.
Detailsaboutthefulltextquerysyntaxcanbefoundbelow.
Note:thefulltextsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforexecutingfulltextqueriesistouseanAQLqueryusingtheFULLTEXTAQLfunctionasfollows:
FORdocINFULLTEXT(@@collection,@attributeName,@queryString,@limit)
RETURNdoc
Examples
arangosh>db.emails.ensureFulltextIndex("content");
arangosh>db.emails.save({content:
........>"HelloAlice,howareyoudoing?Regards,Bob"});
arangosh>db.emails.save({content:
........>"HelloCharlie,doAliceandBobknowaboutit?"});
arangosh>db.emails.save({content:"Ithinktheydon'tknow.Regards,Eve"});
arangosh>db.emails.fulltext("content","charlie,|eve").toArray();
showexecutionresults
FulltextSyntax:
Inthesimplestform,afulltextquerycontainsjustthesoughtword.Ifmultiplesearchwordsaregiveninaquery,theyshouldbeseparatedbycommas.AllsearchwordswillbecombinedwithalogicalANDbydefault,andonlysuchdocumentswillbereturnedthatcontainallsearchwords.Thisdefaultbehaviorcanbechangedbyprovidingtheextracontrolcharactersinthefulltextquery,whichare:
+:logicalAND(intersection)|:logicalOR(union)-:negation(exclusion)
Examples:
"banana":searchesfordocumentscontaining"banana""banana,apple":searchesfordocumentscontainingboth"banana"AND"apple""banana,|orange":searchesfordocumentscontainingeither"banana"OR"orange"ORboth"banana,-apple":searchesfordocumentsthatcontains"banana"butNOT"apple".
Logicaloperatorsareevaluatedfromlefttoright.
SimpleQueries
681
Eachsearchwordcanoptionallybeprefixedwithcomplete:orprefix:,withcomplete:beingthedefault.Thisallowssearchingforcompletewordsorforwordprefixes.Suffixsearchesoranyotherformsarepartial-wordmatchingarecurrentlynotsupported.
Examples:
"complete:banana":searchesfordocumentscontainingtheexactword"banana""prefix:head":searchesfordocumentswithwordsthatstartwithprefix"head""prefix:head,banana":searchesfordocumentscontainwordsstartingwithprefix"head"andthatalsocontaintheexactword"banana".
Completematchandprefixsearchoptionscanbecombinedwiththelogicaloperators.
Pleasenotethatonlywordswithaminimumlengthwillgetindexed.Thisminimumlengthcanbedefinedwhencreatingthefulltextindex.Forwordstokenization,thelibicutextboundaryanalysisisused,whichtakesintoaccountthedefaultasdefinedatserverstartup(--server.default-languagestartupoption).Generally,thewordboundaryanalysiswillfilteroutpunctuationbutwillnotdomuchmore.
Especiallynowordnormalization,stemming,orsimilarityanalysiswillbeperformedwhenindexingorsearching.Ifanyofthesefeaturesisrequired,itissuggestedthattheuserdoesthetextnormalizationontheclientside,andprovidesforeachdocumentanextraattributecontainingjustacomma-separatedlistofnormalizedwords.Thisattributecanthenbeindexedwithafulltextindex,andtheusercansendfulltextqueriesforthisindex,withthefulltextqueriesalsocontainingthestemmedornormalizedversionsofwordsasrequiredbytheuser.
SimpleQueries
682
ArangoDB'sActions
ItisrecommendedtouseFoxxinstead.
IntroductiontoUserActions
InsomewaysthecommunicationlayeroftheArangoDBserverbehaveslikeaWebserver.UnlikeaWebserver,itnormallyrespondstoHTTPrequestsbydeliveringJSONobjects.Remember,documentsinthedatabasearejustJSONobjects.So,mostofthetimetheHTTPresponsewillcontainaJSONdocumentfromthedatabaseasbody.YoucanextractthedocumentsstoredinthedatabaseusingHTTPGET.YoucanstoredocumentsusingHTTPPOST.
However,thereissomethingmore.Youcanwritesmallsnippets-socalledactions-toextendthedatabase.TheideaofactionsisthatsometimesitisbettertostorepartsofthebusinesslogicwithinArangoDB.
Thesimplestexampleistheageofaperson.Assumeyoustoreinformationaboutpeopleinyourdatabase.Itisananti-patterntostoretheage,becauseitchangeseverynowandthen.Therefore,younormallystorethebirthdayandlettheclientdecidewhattodowithit.However,ifyouhavemanydifferentclients,itmightbeeasiertoenrichthepersondocumentwiththeageusingactionsonceontheserverside.
Or,forinstance,ifyouwanttoapplysomestatisticstolargedata-setsandyoucannoteasilyexpressthisasquery.Youcandefineaactioninsteadoftransferringthewholedatatotheclientanddothecomputationontheclient.
Actionsarealsousefulifyouwanttorestrictandfilterdataaccordingtosomecomplexpermissionsystem.
TheArangoDBservercandeliverallkindsofinformation,JSONbeingonlyonepossibleformat.YoucanalsogenerateHTMLorimages.However,aWebserverisnormallybettersuitedforthetaskasitalsoimplementsvariouscachingstrategies,languageselection,compressionandsoon.Havingsaidthat,therearestillsituationswhereitmightbesuitabletousetheArangoDBtodeliverHTMLpages-staticordynamic.Asimpleexampleisthebuilt-inadministrationinterface.YoucanaccessitusinganymodernbrowserandthereisnoneedforaseparateApacheorIIS.
IngeneralyouwilluseFoxxtoeasilyextendthedatabasewithbusinesslogic.Foxxprovidesansimpletouseinterfacetoactions.
Thefollowingsectionswillexplainthelow-levelactionswithinArangoDBonwhichFoxxisbuiltandshowhowtodefinethem.TheexamplesstartwithdeliveringstaticHTMLpages-evenifthisisnottheprimaryuse-caseforactions.ThelatersectionswillthenshowyouhowtocodesomepiecesofyourbusinesslogicandreturnJSONobjects.
TheinterfaceislooselymodeledaftertheJavaScriptclassesforHTTPrequestandresponsesfoundinnode.jsandthemiddleware/routingaspectsofconnect.jsandexpress.js.
Notethatunlikenode.js,ArangoDBismulti-threadedandthereisnoeasywaytosharestatebetweenqueriesinsidetheJavaScriptengine.Ifsuchstateinformationisrequired,youneedtousethedatabaseitself.
Actions
683
AHelloWorldExample
TheclientAPIorbrowsersendsaHTTPrequesttotheArangoDBserverandtheserverreturnsaHTTPresponsetotheclient.AHTTPrequestconsistsofamethod,normallyGETorPOSTwhenusingabrowser,andarequestpathlike/hello/world.ForarealWebserverthereareazillionofotherthingtoconsider,wewillignorethisforthemoment.TheHTTPresponsecontainsacontenttype,describinghowtointerpretthereturneddata,andthedataitself.
Inthefollowingexample,wewanttodefineanactioninArangoDB,sothattheserverreturnstheHTMLdocument
<html>
<body>
HelloWorld
</body>
</html>
ifaskedGET/hello/world.
Theserverneedstoknowwhatfunctiontocallorwhatdocumenttodeliverifitreceivesarequest.Thisiscalledrouting.AlltheroutinginformationofArangoDBisstoredinacollection_routing.Eachentryinthiscollectionsdescribeshowtodealwithaparticularrequestpath.
Fortheaboveexample,addthefollowingdocumenttothe_routingcollection:
arangosh>db._routing.save({
........>url:{
........>match:"/hello/world"
........>},
........>content:{
........>contentType:"text/html",
........>body:"<html><body>HelloWorld</body></html>"
........>}
........>});
showexecutionresultsInordertoactivatethenewrouting,youmusteitherrestarttheserverorcalltheinternalreloadfunction.
arangosh>require("internal").reloadRouting()
Nowusethebrowserandaccesshttp://localhost:8529/hello/world
YoushouldseetheHelloWorldinourbrowser:
shell>curl--dump-http://localhost:8529/hello/world
HTTP/1.1200OK
content-type:text/html
x-content-type-options:nosniff
"HelloWorld"
MatchingaURLTherearealotofoptionsfortheurlattribute.Ifyoudefinedifferentroutingforthesamepath,thenthefollowingsimpleruleisappliedinordertodeterminewhichmatchwins:Iftherearetwomatches,thenthemorespecificwins.I.e,ifthereisawildcardmatchandanexactmatch,theexactmatchispreferred.Ifthereisashortandalongmatch,thelongermatchwins.
Actions
684
ExactMatch
Ifthedefinitionis
{
url:{
match:"/hello/world"
}
}
thenthematchmustbeexact.Onlytherequestfor/hello/worldwillmatch,everythingelse,e.g./hello/world/myor/hello/world2,willnotmatch.
Thefollowingdefinitionisashort-cutforanexactmatch.
{
url:"/hello/world"
}
Note:WhilethetwodefinitionswillresultinthesameURLmatching,thereisasubtledifferencebetweenthem:
Theformerdefinition(definingurlasanobjectwithamatchattribute)willresultintheURLbeingaccessibleviaallsupportedHTTPmethods(e.g.GET,POST,PUT,DELETE,...),whereasthelatterdefinition(providingastringurlattribute)willresultintheURLbeingaccessibleviaHTTPGETandHTTPHEADonly,withallotherHTTPmethodsbeingdisabled.CallingaURLwithanunsupportedordisabledHTTPmethodwillresultinanHTTP501(notimplemented)error.
PrefixMatch
Ifthedefinitionis
{
url:{
match:"/hello/world/*"
}
}
thenthematchcanbeaprefixmatch.Therequestsfor/hello/world,/hello/world/my,and/hello/world/how/are/youwillallmatch.However/hello/world2doesnotmatch.PrefixmatcheswithinaURLpart,i.e./hello/world*,arenotallowed.Thewildcardmustoccurattheend,i.e.
/hello/*/world
isalsodisallowed.
Ifyoudefinetworoutes
{url:{match:"/hello/world/*"}}
{url:{match:"/hello/world/emil"}}
thenthesecondroutewillbeusedfor/hello/world/emilbecauseitismorespecific.
ParameterizedMatch
Aparameterizedmatchissimilartoaprefixmatch,buttheparametersarealsoallowedinsidetheURLpath.
Ifthedefinitionis
{
url:{
match:"/hello/:name/world"
}
}
Actions
685
thentheURLmusthavethreeparts,thefirstpartbeinghelloandthethirdpartworld.Forexample,/hello/emil/worldwillmatch,while/hello/emil/meyer/worldwillnot.
ConstraintMatch
Aconstraintmatchissimilartoaparameterizedmatch,buttheparameterscancarryconstraints.
Ifthedefinitionis
{
url:{
match:"/hello/:name/world",
constraint:{
name:"/[a-z]+/"
}
}
}
thentheURLmusthavethreeparts,thefirstpartbeinghelloandthethirdpartworld.Thesecondpartmustbealllowercase.
ItispossibletousemorethenoneconstraintforthesameURLpart.
{
url:{
match:"/hello/:name|:id/world",
constraint:{
name:"/[a-z]+/",id:"/[0-9]+/"
}
}
}
OptionalMatch
Anoptionalmatchissimilartoaparameterizedmatch,butthelastparameterisoptional.
Ifthedefinitionis
{
url:{
match:"/hello/:name?",
constraint:{
name:"/[a-z]+/"
}
}
}
thentheURL/helloand/hello/emilwillmatch.
Ifthedefinitionsare
{url:{match:"/hello/world"}}
{url:{match:"/hello/:name",constraint:{name:"/[a-z]+/"}}}
{url:{match:"/hello/*"}}
thentheURL/hello/worldwillbematchedbythefirstroute,becauseitisthemostspecific.TheURL/hello/youwillbematchedbythesecondroute,becauseitismorespecificthantheprefixmatch.
MethodRestriction
YoucanrestrictthematchtospecificHTTPmethods.
Ifthedefinitionis
{
url:{
Actions
686
match:"/hello/world",
methods:["post","put"]
}
}
thenonlyHTTPPOSTandPUTrequestswillmatch.CallingwithadifferentHTTPmethodwillresultinanHTTP501error.
Pleasenotethatifurlisdefinedasasimplestring,thenonlytheHTTPmethodsGETandHEADwillbeallowed,anallothermethodswillbedisabled:
{
url:"/hello/world"
}
MoreonMatching
Rememberthatthemorespecificmatchwins.
Amatchwithoutparameterorwildcardismorespecificthanamatchwithparametersorwildcard.Amatchwithparameterismorespecificthanamatchwithawildcard.Ifthereismorethanoneparameter,specificityisappliedfromlefttoright.
Considerthefollowingdefinitions
arangosh>db._routing.save({
........>url:{match:"/hello/world"},
........>content:{contentType:"text/plain",body:"MatchNo1"}});
arangosh>db._routing.save({
........>url:{match:"/hello/:name",constraint:{name:"/[a-z]+/"}},
........>content:{contentType:"text/plain",body:"MatchNo2"}});
arangosh>db._routing.save({
........>url:{match:"/:something/world"},
........>content:{contentType:"text/plain",body:"MatchNo3"}});
arangosh>db._routing.save({
........>url:{match:"/hi/*"},
........>content:{contentType:"text/plain",body:"MatchNo4"}});
arangosh>require("internal").reloadRouting()
showexecutionresultsThen
shell>curl--dump-http://localhost:8529/hello/world
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
"MatchNo1"
shell>curl--dump-http://localhost:8529/hello/emil
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
"MatchNo2"
shell>curl--dump-http://localhost:8529/your/world
HTTP/1.1200OK
Actions
687
content-type:text/plain
x-content-type-options:nosniff
"MatchNo3"
shell>curl--dump-http://localhost:8529/hi/you
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
"MatchNo4"
Youcanwritethefollowingdocumentintothe_routingcollectiontotesttheaboveexamples.
{
routes:[
{url:{match:"/hello/world"},content:"route1"},
{url:{match:"/hello/:name|:id",constraint:{name:"/[a-z]+/",id:"/[0-9]+/"}},content:"route2"},
{url:{match:"/:something/world"},content:"route3"},
{url:{match:"/hello/*"},content:"route4"},
]
}
Actions
688
AHelloWorldExampleforJSON
Ifyouchangetheexampleslightly,thenaJSONobjectwillbedelivered.
arangosh>db._routing.save({
........>url:"/hello/json",
........>content:{
........>contentType:"application/json",
........>body:'{"hello":"world"}'
........>}
........>});
arangosh>require("internal").reloadRouting()
showexecutionresultsAgaincheckwithyourbrowserorcURLhttp://localhost:8529/hello/json
Dependingonyourbrowserandinstalledadd-onsyouwilleitherseetheJSONobjectoradownloaddialog.IfyourbrowserwantstoopenanexternalapplicationtodisplaytheJSONobject,youcanchangethecontentTypeto"text/plain" fortheexample.Thismakesiteasiertochecktheexampleusingabrowser.Orusecurltoaccesstheserver.
shell>curl--dump-http://localhost:8529/hello/json
HTTP/1.1200OK
content-type:application/json;charset=utf-8
x-content-type-options:nosniff
{
"hello":"world"
}
DeliveringContent
Therearealotofdifferentwaysonhowtodelivercontent.Wehavealreadyseenthesimplestone,wherestaticcontentisdelivered.Thefun,however,startswhendeliveringdynamiccontent.
StaticContent
Youcanspecifyabodyandacontent-type.
arangosh>db._routing.save({
........>url:"/hello/contentType",
........>content:{
........>contentType:"text/html",
........>body:"<html><body>HelloWorld</body></html>"
........>}
........>});
arangosh>require("internal").reloadRouting()
showexecutionresults
shell>curl--dump-http://localhost:8529/hello/contentType
Actions
689
HTTP/1.1200OK
content-type:text/html
x-content-type-options:nosniff
"HelloWorld"
Ifthecontenttypeistext/plainthenyoucanusetheshort-cut
{
content:"HelloWorld"
}
ASimpleAction
Thesimplestdynamicactionis:
{
action:{
do:"@arangodb/actions/echoRequest"
}
}
Itisnotadvisabletostorefunctionsdirectlyintheroutingtable.Itisbettertocallfunctionsdefinedinmodules.IntheaboveexamplethefunctioncanbeaccessedfromJavaScriptas:
require("@arangodb/actions").echoRequest
ThefunctionechoRequestispre-defined.Ittakestherequestobjectsandechositintheresponse.
Thesignatureofsuchafunctionmustbe
function(req,res,options,next)
Examples
arangosh>db._routing.save({
........>url:"/hello/echo",
........>action:{
........>do:"@arangodb/actions/echoRequest"
........>}
........>});
showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo
Youshouldseesomethinglike
arangosh>arango.GET("/hello/echo")
showexecutionresults
Therequestmightcontainpath,prefix,suffix,andurlParametersattributes.pathisthecompletepathassuppliedbytheuserandalwaysavailable.Ifaprefixwasmatched,thenthisprefixisstoredintheattributeprefixandtheremainingURLpartsarestoredasanarrayinsuffix.Ifoneormoreparameterswerematched,thentheparametervaluesarestoredinurlParameters.
Actions
690
Forexample,iftheurldescriptionis
{
url:{
match:"/hello/:name/:action"
}
}
andyourequestthepath/hello/emil/jump,thentherequestobjectwillcontainthefollowingattribute
urlParameters:{
name:"emil",
action:"jump"
}
ActionController
Asanalternativetothesimpleaction,youcanusecontrollers.Acontrollerisamodule,definesthefunctionget,put,post,delete,head,patch.Ifarequestofthecorrespondingtypeismatched,thefunctionwillbecalled.
Examples
arangosh>db._routing.save({
........>url:"/hello/echo",
........>action:{
........>controller:"@arangodb/actions/echoController"
........>}
........>});
showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo:
arangosh>arango.GET("/hello/echo")
showexecutionresults
PrefixActionController
Thecontrollerisselectedwhenthedefinitionisread.Thereisamoreflexible,butslowerandmaybeinsecurevariant,theprefixcontroller.
Assumethattheurlisaprefixmatch
{
url:{
match:/hello/*"
}
}
Youcanuse
{
action:{
prefixController:"@arangodb/actions"
}
}
todefineaprefixcontroller.IftheURL/hello/echoControllerisgiven,thenthemodule@arangodb/actions/echoControllerisused.
Ifyouuseaprefixcontroller,youshouldmakecertainthatnounwantedactionsareavailableundertheprefix.
Actions
691
Thedefinition
{
action:"@arangodb/actions"
}
isashort-cutforaprefixcontrollerdefinition.
FunctionAction
Youcanalsostoreafunctiondirectlyintheroutingtable.
Examples
arangosh>db._routing.save({
........>url:"/hello/echo",
........>action:{
........>callback:"function(req,res){res.statusCode=200;res.body='Hello'}"
........>}
........>});
showexecutionresults
arangosh>arango.GET("hello/echo")
arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/hello/echo'REMOVEroute
in_routing")
[objectArangoQueryCursor,count:0,cached:false,hasMore:false]
arangosh>require("internal").reloadRouting()
RequestsandResponses
Thecontrollermustdefinehandlerfunctionswhichtakearequestobjectandfilltheresponseobject.
AverysimpleexampleisthefunctionechoRequestdefinedinthemodule@arangodb/actions.
function(req,res,options,next){
varresult;
result={request:req,options:options};
res.responseCode=exports.HTTP_OK;
res.contentType="application/json";
res.body=JSON.stringify(result);
}
Installitvia:
arangosh>db._routing.save({
........>url:"/echo",
........>action:{
........>do:"@arangodb/actions/echoRequest"
........>}
........>})
showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo
Youshouldseesomethinglike
Actions
692
arangosh>arango.GET("/hello/echo")
arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/hello/echo'REMOVEroute
in_routing")
arangosh>require("internal").reloadRouting()
showexecutionresultsYoumayalsopassoptionstothecalledfunction:
arangosh>db._routing.save({
........>url:"/echo",
........>action:{
........>do:"@arangodb/actions/echoRequest",
........>options:{
........>"Hello":"World"
........>}
........>}
........>});
showexecutionresultsYounowseetheoptionsintheresult:
arangosh>arango.GET("/echo")
arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/echo'REMOVEroutein
_routing")
arangosh>require("internal").reloadRouting()
showexecutionresults
Actions
693
ModifyingRequestandResponse
Aswe'veseeninthepreviousexamples,actionsgetcalledwiththerequestandresponseobjects(namedreqandresintheexamples)passedasparameterstotheirhandlerfunctions.
ThereqobjectcontainstheincomingHTTPrequest,whichmightormightnothavebeenmodifiedbyapreviousaction(ifactionswerechained).
Ahandlercanmodifytherequestobjectinplaceifdesired.Thismightbeusefulwhenwritingmiddleware(seebelow)thatisusedtointerceptincomingrequests,modifythemandpassthemtotheactualhandlers.
Whilemodifyingtherequestobjectmightnotbethatrelevantfornon-middlewareactions,modifyingtheresponseobjectdefinitelyis.Modifyingtheresponseobjectisanaction'sonlywaytoreturndatatothecalleroftheaction.
We'vealreadyseenhowtosettheHTTPstatuscode,thecontenttype,andtheresultbody.Theresobjecthasthefollowingpropertiesforthese:
contentType:MIMEtypeofthebodyasdefinedintheHTTPstandard(e.g.text/html,text/plain,application/json,...)responsecode:theHTTPstatuscodeoftheresponseasdefinedintheHTTPstandard.Commonvaluesforactionsthatsucceedare200or201.PleaserefertotheHTTPstandardformoreinformation.body:theactualresponsedata
Tosetormodifyarbitraryheadersoftheresponseobject,theheaderspropertycanbeused.Forexample,toaddauser-definedheadertotheresponse,thefollowingcodewilldo:
res.headers=res.headers||{};//headersmightormightnotbepresent
res.headers['X-Test']='someValue';//setheaderX-Testto"someValue"
ThiswillsettheadditionalHTTPheaderX-TesttovaluesomeValue.Otherheaderscanbesetaswell.NotethatArangoDBmightchangethecaseoftheheadernamestolowercasewhenassemblingtheoverallresponsethatissenttothecaller.
ItisnotnecessarytoexplicitlysetaContent-LengthheaderfortheresponseasArangoDBwillcalculatethecontentlengthautomaticallyandaddthisheaderitself.ArangoDBmightalsoaddaConnectionheaderitselftohandleHTTPkeep-alive.
ArangoDBalsosupportsautomatictransformationofthebodydatatoanotherformat.Currently,theonlysupportedtransformationsarebase64-encodingandbase64-decoding.Usingthetransformations,anactioncancreateabase64encodedbodyandstillletArangoDBsendthenon-encodedversion,forexample:
res.body='VGhpcyBpcyBhIHRlc3Q=';
res.transformations=res.transformations||[];//initialize
res.transformations.push('base64decode');//willbase64decodetheresponsebody
WhenArangoDBprocessestheresponse,itwillbase64-decodewhat'sinres.bodyandsettheHTTPheaderContent-Encoding:binary.Theoppositecanbeachievedwiththebase64encodetransformation:ArangoDBwillthenautomaticallybase64-encodethebodyandsetaContent-Encoding:base64HTTPheader.
WritingdynamicactionhandlersTowriteyourowndynamicactionhandlers,youmustputthemintomodules.
Modulesareameansoforganizingactionhandlersandmakingthemloadableunderspecificnames.
Tostart,we'lldefineasimpleactionhandlerinamodule/ownTest:
arangosh>db._modules.save({
........>path:"/db:/ownTest",
........>content:
........>"exports.do=function(req,res,options,next){"+
........>"res.body='test';"+
Actions
694
........>"res.responseCode=200;"+
........>"res.contentType='text/plain';"+
........>"};"
........>});
showexecutionresultsThisdoesnothingbutregisteradoactionhandlerinamodule/ownTest.Theactionhandlerisnotyetcallable,butmustbemappedtoaroutefirst.Tomaptheactiontotheroute/ourtest,executethefollowingcommand:
arangosh>db._routing.save({
........>url:"/ourtest",
........>action:{
........>controller:"db://ownTest"
........>}
........>});
arangosh>require("internal").reloadRouting()
showexecutionresultsNowusethebrowserorcURLandaccesshttp://localhost:8529/ourtest:
shell>curl--dump-http://localhost:8529/ourtest
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
"test"
Youwillseethatthemodule'sdofunctionhasbeenexecuted.
AWordaboutCachingSometimesitmightseemthatyourchangedonottakeeffect.Inthiscasetheculpritcouldbetheroutingcaches:
Theroutingcachestorestheroutinginformationcomputedfromthe_routingcollection.Wheneveryouchangethiscollectionmanually,youneedtocall
arangosh>require("internal").reloadRouting()
inordertorebuildthecache.
AdvancedUsagesFordetailedinformationseethereferencemanual.
Redirects
Usethefollowingforapermanentredirect:
arangosh>db._routing.save({
........>url:"/redirectMe",
........>action:{
........>do:"@arangodb/actions/redirectRequest",
........>options:{
Actions
695
........>permanently:true,
........>destination:"/somewhere.else/"
........>}
........>}
........>});
arangosh>require("internal").reloadRouting()
showexecutionresults
shell>curl--dump-http://localhost:8529/redirectMe
HTTP/1.1301MovedPermanently
x-content-type-options:nosniff
content-type:text/html
location:/somewhere.else/
"<html><head><title>Moved</title></head><body><h1>Moved</h1><p>Thispagehasmovedto<a
href=\"/somewhere.else/\">/somewhere.else/</a>.</p></body></html>"
RoutingBundles
Insteadofaddingallroutesforpackageseparately,youcanspecifyabundle:
arangosh>db._routing.save({
........>routes:[
........>{
........>url:"/url1",
........>content:"route1"
........>},
........>{
........>url:"/url2",
........>content:"route2"
........>},
........>{
........>url:"/url3",
........>content:"route3"
........>}
........>]
........>});
arangosh>require("internal").reloadRouting()
showexecutionresults
shell>curl--dump-http://localhost:8529/url2
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
"route2"
shell>curl--dump-http://localhost:8529/url3
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
Actions
696
"route3"
Theadvantageis,thatyoucanputallyourroutesintoonedocumentanduseacommonprefix.
arangosh>db._routing.save({
........>urlPrefix:"/test",
........>routes:[
........>{
........>url:"/url1",
........>content:"route1"
........>},
........>{
........>url:"/url2",
........>content:"route2"
........>},
........>{
........>url:"/url3",
........>content:"route3"
........>}
........>]
........>});
arangosh>require("internal").reloadRouting()
showexecutionresultswilldefinetheURL/test/url1,/test/url2,and/test/url3:
shell>curl--dump-http://localhost:8529/test/url1
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
"route1"
shell>curl--dump-http://localhost:8529/test/url2
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
"route2"
shell>curl--dump-http://localhost:8529/test/url3
HTTP/1.1200OK
content-type:text/plain
x-content-type-options:nosniff
"route3"
WritingMiddleware
Assume,youwanttologeveryrequestinyournamespacetotheconsole.(ifArangoDBisrunningasadaemon,thiswillendupinthelogfile).InthiscaseyoucaneasilydefineanactionfortheURL/subdirectory.Thisactionsimplylogstherequests,callsthenextinline,andlogstheresponse:
arangosh>db._modules.save({
Actions
697
........>path:"/db:/OwnMiddlewareTest",
........>content:
........>"exports.logRequest=function(req,res,options,next){"+
........>"console=require('console');"+
........>"console.log('receivedrequest:%s',JSON.stringify(req));"+
........>"next();"+
........>"console.log('producedresponse:%s',JSON.stringify(res));"+
........>"};"
........>});
showexecutionresultsThisfunctionwillnowbeavailableasdb://OwnMiddlewareTest/logRequest.YouneedtotellArangoDBthatitisshoulduseaprefixmatchandthattheshortestmatchshouldwininthiscase:
arangosh>db._routing.save({
........>middleware:[
........>{
........>url:{
........>match:"/subdirectory/*"
........>},
........>action:{
........>do:"db://OwnMiddlewareTest/logRequest"
........>}
........>}
........>]
........>});
showexecutionresultsWhenyoucallnext()inthataction,thenextspecificroutingwillbeusedfortheoriginalURL.EvenifyoumodifytheURLintherequestobjectreq,thiswillnotcausethenext()tojumptotheroutingdefinedforthisnextURL.IfproceedsoccurringtheoriginURL.However,ifyouusenext(true),theroutingwillstopandrequesthandlingisstartedwiththenewURL.Youmustensurethatnext(true)isnevercalledwithoutmodifyingtheURLintherequestobjectreq.Otherwiseanendlessloopwilloccur.
Nowweaddsomeothersimpleroutingstotestallthis:
arangosh>db._routing.save({
........>url:"/subdirectory/ourtest/1",
........>action:{
........>do:"@arangodb/actions/echoRequest"
........>}
........>});
arangosh>db._routing.save({
........>url:"/subdirectory/ourtest/2",
........>action:{
........>do:"@arangodb/actions/echoRequest"
........>}
........>});
arangosh>db._routing.save({
........>url:"/subdirectory/ourtest/3",
........>action:{
........>do:"@arangodb/actions/echoRequest"
........>}
........>});
arangosh>require("internal").reloadRouting()
showexecutionresults
Actions
698
Thenwesendsomecurlrequeststothesesampleroutes:
shell>curl--dump-http://localhost:8529/subdirectory/ourtest/1
HTTP/1.1200OK
content-type:application/json;charset=utf-8
x-content-type-options:nosniff
showresponsebodyandtheconsole(and/orthelogfile)willshowrequestsandreplies.Notethatloggingdoesn'twarrantthesequenceinwhichtheselineswillappear.
ApplicationDeploymentUsingsingleroutesorbundlescanbebecomeabitmessyinlargeapplications.Kaerushaswrittenadeploymenttoolinnode.js.
NotethatthereisalsoFoxxforbuildingapplicationswithArangoDB.
CommonPitfallswhenusingActions
Caching
IfyoumadeanychangestotheroutingbutthechangesdoesnothaveanyeffectwhencallingthemodifiedactionsURL,youmighthavebeenhitbysomecachingissues.
Afteranymodificationtotheroutingoractions,itisthusrecommendedtomakethechanges"live"bycallingthefollowingfunctionsfromwithinarangosh:
Youmightalsobeaffectedbyclient-sidecaching.BrowserstendtocachecontentandalsoredirectionURLs.Youmightneedtoclearordisablethebrowsercacheinsomecasestoseeyourchangesineffect.
Datatypes
Whenprocessingtherequestdatainanaction,pleasebeawarethatthedatatypeofallqueryparametersisstring.ThisisbecausethewholeURLisastringandwhentheindividualpartsareextracted,theywillalsobestrings.
Forexample,whencallingtheURLhttp://localhost:8529/hello/world?value=5
theparametervaluewillhaveavalueof(string)5,not(number)5.ThismightbetroublesomeifyouuseJavaScript's===operatorwhencheckingrequestparametervalues.
ThesameproblemoccurswithincomingHTTPheaders.WhensendingthefollowingheaderfromaclienttoArangoDB
X-My-Value:5
thentheheaderX-My-Valuewillhaveavalueof(string)5andnot(number)5.
404NotFound
IfyoudefinedaURLintheroutingandtheURLisaccessiblefineviaHTTPGETbutreturnsanHTTP501(notimplemented)forotherHTTPmethodssuchasPOST,PUTorDELETE,thenyoumighthavebeenhitbysomedefaults.
Bydefault,URLsdefinedlikethis(simplestringurlattribute)areaccessibleviaHTTPGETandHEADonly.TomakesuchURLsaccessibleviaotherHTTPmethods,extendtheURLdefinitionwiththemethodsattribute.
Forexample,thisdefinitiononlyallowsaccessviaGETandHEAD:
Actions
699
{
url:"/hello/world"
}
whereasthisdefinitionallowsHTTPGET,POST,andPUT:
arangosh>db._routing.save({
........>url:{
........>match:"/hello/world",
........>methods:["get","post","put"]
........>},
........>action:{
........>do:"@arangodb/actions/echoRequest"
........>}
........>});
arangosh>require("internal").reloadRouting()
showexecutionresults
shell>curl--dump-http://localhost:8529/hello/world
HTTP/1.1200OK
content-type:application/json;charset=utf-8
x-content-type-options:nosniff
showresponsebodyTheformerdefinition(definingurlasanobjectwithamatchattribute)willresultintheURLbeingaccessibleviaallsupportedHTTPmethods(e.g.GET,POST,PUT,DELETE,...),whereasthelatterdefinition(providingastringurlattribute)willresultintheURLbeingaccessibleviaHTTPGETandHTTPHEADonly,withallotherHTTPmethodsbeingdisabled.CallingaURLwithanunsupportedordisabledHTTPmethodwillresultinanHTTP404error.
Actions
700
Errorcodesandmeanings
Generalerrors
0-ERROR_NO_ERRORNoerrorhasoccurred.
1-ERROR_FAILEDWillberaisedwhenageneralerroroccurred.
2-ERROR_SYS_ERRORWillberaisedwhenoperatingsystemerroroccurred.
3-ERROR_OUT_OF_MEMORYWillberaisedwhenthereisamemoryshortage.
4-ERROR_INTERNALWillberaisedwhenaninternalerroroccurred.
5-ERROR_ILLEGAL_NUMBERWillberaisedwhenanillegalrepresentationofanumberwasgiven.
6-ERROR_NUMERIC_OVERFLOWWillberaisedwhenanumericoverflowoccurred.
7-ERROR_ILLEGAL_OPTIONWillberaisedwhenanunknownoptionwassuppliedbytheuser.
8-ERROR_DEAD_PIDWillberaisedwhenaPIDwithoutalivingprocesswasfound.
9-ERROR_NOT_IMPLEMENTEDWillberaisedwhenhittinganunimplementedfeature.
10-ERROR_BAD_PARAMETERWillberaisedwhentheparameterdoesnotfulfilltherequirements.
11-ERROR_FORBIDDENWillberaisedwhenyouaremissingpermissionfortheoperation.
12-ERROR_OUT_OF_MEMORY_MMAPWillberaisedwhenthereisamemoryshortage.
13-ERROR_CORRUPTED_CSVWillberaisedwhenencounteringacorruptcsvline.
14-ERROR_FILE_NOT_FOUNDWillberaisedwhenafileisnotfound.
15-ERROR_CANNOT_WRITE_FILEWillberaisedwhenafilecannotbewritten.
16-ERROR_CANNOT_OVERWRITE_FILEWillberaisedwhenanattemptismadetooverwriteanexistingfile.
17-ERROR_TYPE_ERRORWillberaisedwhenatypeerrorisunencountered.
18-ERROR_LOCK_TIMEOUTWillberaisedwhenthere'satimeoutwaitingforalock.
19-ERROR_CANNOT_CREATE_DIRECTORYWillberaisedwhenanattempttocreateadirectoryfails.
Errorcodesandmeanings
701
20-ERROR_CANNOT_CREATE_TEMP_FILEWillberaisedwhenanattempttocreateatemporaryfilefails.
21-ERROR_REQUEST_CANCELEDWillberaisedwhenarequestiscanceledbytheuser.
22-ERROR_DEBUGWillberaisedintentionallyduringdebugging.
25-ERROR_IP_ADDRESS_INVALIDWillberaisedwhenthestructureofanIPaddressisinvalid.
27-ERROR_FILE_EXISTSWillberaisedwhenafilealreadyexists.
28-ERROR_LOCKEDWillberaisedwhenaresourceoranoperationislocked.
29-ERROR_DEADLOCKWillberaisedwhenadeadlockisdetectedwhenaccessingcollections.
30-ERROR_SHUTTING_DOWNWillberaisedwhenacallcannotsucceedbecauseaservershutdownisalreadyinprogress.
31-ERROR_ONLY_ENTERPRISEWillberaisedwhenanenterprise-featureisrequestedfromthecommunityedition.
32-ERROR_RESOURCE_LIMITWillberaisedwhentheresourcesusedbyanoperationexceedtheconfiguredmaximumvalue.
HTTPerrorstatuscodes400-ERROR_HTTP_BAD_PARAMETERWillberaisedwhentheHTTPrequestdoesnotfulfilltherequirements.
401-ERROR_HTTP_UNAUTHORIZEDWillberaisedwhenauthorizationisrequiredbuttheuserisnotauthorized.
403-ERROR_HTTP_FORBIDDENWillberaisedwhentheoperationisforbidden.
404-ERROR_HTTP_NOT_FOUNDWillberaisedwhenanURIisunknown.
405-ERROR_HTTP_METHOD_NOT_ALLOWEDWillberaisedwhenanunsupportedHTTPmethodisusedforanoperation.
406-ERROR_HTTP_NOT_ACCEPTABLEWillberaisedwhenanunsupportedHTTPcontenttypeisusedforanoperation,orifarequestisnotacceptableforaleaderorfollower.
412-ERROR_HTTP_PRECONDITION_FAILEDWillberaisedwhenapreconditionforanHTTPrequestisnotmet.
500-ERROR_HTTP_SERVER_ERRORWillberaisedwhenaninternalserverisencountered.
503-ERROR_HTTP_SERVICE_UNAVAILABLEWillberaisedwhenaserviceistemporarilyunavailable.
504-ERROR_HTTP_GATEWAY_TIMEOUTWillberaisedwhenaservicecontactedbyArangoDBdoesnotrespondinatimelymanner.
Errorcodesandmeanings
702
HTTPprocessingerrors
600-ERROR_HTTP_CORRUPTED_JSONWillberaisedwhenastringrepresentationofaJSONobjectiscorrupt.
601-ERROR_HTTP_SUPERFLUOUS_SUFFICESWillberaisedwhentheURLcontainssuperfluoussuffices.
InternalArangoDBstorageerrorsForerrorsthatoccurbecauseofaprogrammingerror.
1000-ERROR_ARANGO_ILLEGAL_STATEInternalerrorthatwillberaisedwhenthedatafileisnotintherequiredstate.
1002-ERROR_ARANGO_DATAFILE_SEALEDInternalerrorthatwillberaisedwhentryingtowritetoadatafile.
1004-ERROR_ARANGO_READ_ONLYInternalerrorthatwillberaisedwhentryingtowritetoaread-onlydatafileorcollection.
1005-ERROR_ARANGO_DUPLICATE_IDENTIFIERInternalerrorthatwillberaisedwhenaidentifierduplicateisdetected.
1006-ERROR_ARANGO_DATAFILE_UNREADABLEInternalerrorthatwillberaisedwhenadatafileisunreadable.
1007-ERROR_ARANGO_DATAFILE_EMPTYInternalerrorthatwillberaisedwhenadatafileisempty.
1008-ERROR_ARANGO_RECOVERYWillberaisedwhenanerroroccurredduringWALlogfilerecovery.
1009-ERROR_ARANGO_DATAFILE_STATISTICS_NOT_FOUNDWillberaisedwhenarequireddatafilestatisticsobjectwasnotfound.
ExternalArangoDBstorageerrors
Forerrorsthatoccurbecauseofanoutsideevent.
1100-ERROR_ARANGO_CORRUPTED_DATAFILEWillberaisedwhenacorruptionisdetectedinadatafile.
1101-ERROR_ARANGO_ILLEGAL_PARAMETER_FILEWillberaisedifaparameterfileiscorruptedorcannotberead.
1102-ERROR_ARANGO_CORRUPTED_COLLECTIONWillberaisedwhenacollectioncontainsoneormorecorrupteddatafiles.
1103-ERROR_ARANGO_MMAP_FAILEDWillberaisedwhenthesystemcallmmapfailed.
1104-ERROR_ARANGO_FILESYSTEM_FULLWillberaisedwhenthefilesystemisfull.
1105-ERROR_ARANGO_NO_JOURNALWillberaisedwhenajournalcannotbecreated.
1106-ERROR_ARANGO_DATAFILE_ALREADY_EXISTSWillberaisedwhenthedatafilecannotbecreatedorrenamedbecauseafileofthesamenamealreadyexists.
1107-ERROR_ARANGO_DATADIR_LOCKEDWillberaisedwhenthedatabasedirectoryislockedbyadifferentprocess.
Errorcodesandmeanings
703
1108-ERROR_ARANGO_COLLECTION_DIRECTORY_ALREADY_EXISTSWillberaisedwhenthecollectioncannotbecreatedbecauseadirectoryofthesamenamealreadyexists.
1109-ERROR_ARANGO_MSYNC_FAILEDWillberaisedwhenthesystemcallmsyncfailed.
1110-ERROR_ARANGO_DATADIR_UNLOCKABLEWillberaisedwhentheservercannotlockthedatabasedirectoryonstartup.
1111-ERROR_ARANGO_SYNC_TIMEOUTWillberaisedwhentheserverwaitedtoolongforadatafiletobesyncedtodisk.
GeneralArangoDBstorageerrors
Forerrorsthatoccurwhenfulfillingauserrequest.
1200-ERROR_ARANGO_CONFLICTWillberaisedwhenupdatingordeletingadocumentandaconflicthasbeendetected.
1201-ERROR_ARANGO_DATADIR_INVALIDWillberaisedwhenanon-existingdatabasedirectorywasspecifiedwhenstartingthedatabase.
1202-ERROR_ARANGO_DOCUMENT_NOT_FOUNDWillberaisedwhenadocumentwithagivenidentifierorhandleisunknown.
1203-ERROR_ARANGO_COLLECTION_NOT_FOUNDWillberaisedwhenacollectionwiththegivenidentifierornameisunknown.
1204-ERROR_ARANGO_COLLECTION_PARAMETER_MISSINGWillberaisedwhenthecollectionparameterismissing.
1205-ERROR_ARANGO_DOCUMENT_HANDLE_BADWillberaisedwhenadocumenthandleiscorrupt.
1206-ERROR_ARANGO_MAXIMAL_SIZE_TOO_SMALLWillberaisedwhenthemaximalsizeofthejournalistoosmall.
1207-ERROR_ARANGO_DUPLICATE_NAMEWillberaisedwhenanameduplicateisdetected.
1208-ERROR_ARANGO_ILLEGAL_NAMEWillberaisedwhenanillegalnameisdetected.
1209-ERROR_ARANGO_NO_INDEXWillberaisedwhennosuitableindexforthequeryisknown.
1210-ERROR_ARANGO_UNIQUE_CONSTRAINT_VIOLATEDWillberaisedwhenthereisauniqueconstraintviolation.
1211-ERROR_ARANGO_VIEW_NOT_FOUNDWillberaisedwhenaviewwiththegivenidentifierornameisunknown.
1212-ERROR_ARANGO_INDEX_NOT_FOUNDWillberaisedwhenanindexwithagivenidentifierisunknown.
1213-ERROR_ARANGO_CROSS_COLLECTION_REQUESTWillberaisedwhenacross-collectionisrequested.
1214-ERROR_ARANGO_INDEX_HANDLE_BADWillberaisedwhenaindexhandleiscorrupt.
1216-ERROR_ARANGO_DOCUMENT_TOO_LARGEWillberaisedwhenthedocumentcannotfitintoanydatafilebecauseofitistoolarge.
Errorcodesandmeanings
704
1217-ERROR_ARANGO_COLLECTION_NOT_UNLOADEDWillberaisedwhenacollectionshouldbeunloaded,buthasadifferentstatus.
1218-ERROR_ARANGO_COLLECTION_TYPE_INVALIDWillberaisedwhenaninvalidcollectiontypeisusedinarequest.
1219-ERROR_ARANGO_VALIDATION_FAILEDWillberaisedwhenthevalidationofanattributeofastructurefailed.
1220-ERROR_ARANGO_ATTRIBUTE_PARSER_FAILEDWillberaisedwhenparsinganattributenamedefinitionfailed.
1221-ERROR_ARANGO_DOCUMENT_KEY_BADWillberaisedwhenadocumentkeyiscorrupt.
1222-ERROR_ARANGO_DOCUMENT_KEY_UNEXPECTEDWillberaisedwhenauser-defineddocumentkeyissuppliedforcollectionswithautokeygeneration.
1224-ERROR_ARANGO_DATADIR_NOT_WRITABLEWillberaisedwhentheserver'sdatabasedirectoryisnotwritableforthecurrentuser.
1225-ERROR_ARANGO_OUT_OF_KEYSWillberaisedwhenakeygeneratorrunsoutofkeys.
1226-ERROR_ARANGO_DOCUMENT_KEY_MISSINGWillberaisedwhenadocumentkeyismissing.
1227-ERROR_ARANGO_DOCUMENT_TYPE_INVALIDWillberaisedwhenthereisanattempttocreateadocumentwithaninvalidtype.
1228-ERROR_ARANGO_DATABASE_NOT_FOUNDWillberaisedwhenanon-existingdatabaseisaccessed.
1229-ERROR_ARANGO_DATABASE_NAME_INVALIDWillberaisedwhenaninvaliddatabasenameisused.
1230-ERROR_ARANGO_USE_SYSTEM_DATABASEWillberaisedwhenanoperationisrequestedinadatabaseotherthanthesystemdatabase.
1231-ERROR_ARANGO_ENDPOINT_NOT_FOUNDWillberaisedwhenthereisanattempttodeleteanon-existingendpoint.
1232-ERROR_ARANGO_INVALID_KEY_GENERATORWillberaisedwhenaninvalidkeygeneratordescriptionisused.
1233-ERROR_ARANGO_INVALID_EDGE_ATTRIBUTEwillberaisedwhenthe_fromor_tovaluesofanedgeareundefinedorcontainaninvalidvalue.
1234-ERROR_ARANGO_INDEX_DOCUMENT_ATTRIBUTE_MISSINGWillberaisedwhenanattempttoinsertadocumentintoanindexiscausedbyinthedocumentnothavingoneormoreattributeswhichtheindexisbuilton.
1235-ERROR_ARANGO_INDEX_CREATION_FAILEDWillberaisedwhenanattempttocreateanindexhasfailed.
1236-ERROR_ARANGO_WRITE_THROTTLE_TIMEOUTWillberaisedwhentheserveriswrite-throttledandawriteoperationhaswaitedtoolongfortheservertoprocessqueuedoperations.
1237-ERROR_ARANGO_COLLECTION_TYPE_MISMATCHWillberaisedwhenacollectionhasadifferenttypefromwhathasbeenexpected.
1238-ERROR_ARANGO_COLLECTION_NOT_LOADEDWillberaisedwhenacollectionisaccessedthatisnotyetloaded.
Errorcodesandmeanings
705
1239-ERROR_ARANGO_DOCUMENT_REV_BADWillberaisedwhenadocumentrevisioniscorruptorismissingwhereneeded.
CheckedArangoDBstorageerrors
Forerrorsthatoccurbutareanticipated.
1300-ERROR_ARANGO_DATAFILE_FULLWillberaisedwhenthedatafilereachesitslimit.
1301-ERROR_ARANGO_EMPTY_DATADIRWillberaisedwhenencounteringanemptyserverdatabasedirectory.
1302-ERROR_ARANGO_TRY_AGAINWillberaisedwhenanoperationshouldberetried.
1303-ERROR_ARANGO_BUSYWillberaisedwhenstorageengineisbusy.
1304-ERROR_ARANGO_MERGE_IN_PROGRESSWillberaisedwhenstorageenginehasadatafilemergeinprogressandcannotcompletetheoperation.
1305-ERROR_ARANGO_IO_ERRORWillberaisedwhenstorageengineencountersanI/Oerror.
ArangoDBreplicationerrors
1400-ERROR_REPLICATION_NO_RESPONSEWillberaisedwhenthereplicationapplierdoesnotreceiveanyoranincompleteresponsefromthemaster.
1401-ERROR_REPLICATION_INVALID_RESPONSEWillberaisedwhenthereplicationapplierreceivesaninvalidresponsefromthemaster.
1402-ERROR_REPLICATION_MASTER_ERRORWillberaisedwhenthereplicationapplierreceivesaservererrorfromthemaster.
1403-ERROR_REPLICATION_MASTER_INCOMPATIBLEWillberaisedwhenthereplicationapplierconnectstoamasterthathasanincompatibleversion.
1404-ERROR_REPLICATION_MASTER_CHANGEWillberaisedwhenthereplicationapplierconnectstoadifferentmasterthanbefore.
1405-ERROR_REPLICATION_LOOPWillberaisedwhenthereplicationapplierisaskedtoconnecttoitselfforreplication.
1406-ERROR_REPLICATION_UNEXPECTED_MARKERWillberaisedwhenanunexpectedmarkerisfoundinthereplicationlogstream.
1407-ERROR_REPLICATION_INVALID_APPLIER_STATEWillberaisedwhenaninvalidreplicationapplierstatefileisfound.
1408-ERROR_REPLICATION_UNEXPECTED_TRANSACTIONWillberaisedwhenanunexpectedtransactionidisfound.
1410-ERROR_REPLICATION_INVALID_APPLIER_CONFIGURATIONWillberaisedwhentheconfigurationforthereplicationapplierisinvalid.
1411-ERROR_REPLICATION_RUNNINGWillberaisedwhenthereisanattempttoperformanoperationwhilethereplicationapplierisrunning.
1412-ERROR_REPLICATION_APPLIER_STOPPEDSpecialerrorcodeusedtoindicatethereplicationapplierwasstoppedbyauser.
Errorcodesandmeanings
706
1413-ERROR_REPLICATION_NO_START_TICKWillberaisedwhenthereplicationapplierisstartedwithoutaknownstarttickvalue.
1414-ERROR_REPLICATION_START_TICK_NOT_PRESENTWillberaisedwhenthereplicationapplierfetchesdatausingastarttick,butthatstarttickisnotpresentontheloggerserveranymore.
ERROR_REPLICATION_WRONG_CHECKSUM_FORMAT,1415,"thechecksumformatiswrong","Willberaisedwhentheformatofthechecksumiswrong")
1416-ERROR_REPLICATION_WRONG_CHECKSUMWillberaisedwhenanewbornfollowersubmitsawrongchecksum
1417-ERROR_REPLICATION_SHARD_NONEMPTYWillberaisedwhenashardisnotemptyandthefollowertriesashortcut
ArangoDBclustererrors1450-ERROR_CLUSTER_NO_AGENCYWillberaisedwhennoneoftheagencyserverscanbeconnectedto.
1451-ERROR_CLUSTER_NO_COORDINATOR_HEADERWillberaisedwhenaDBserverinaclusterreceivesaHTTPrequestwithoutacoordinatorheader.
1452-ERROR_CLUSTER_COULD_NOT_LOCK_PLANWillberaisedwhenacoordinatorinaclustercannotlockthePlanhierarchyintheagency.
1453-ERROR_CLUSTER_COLLECTION_ID_EXISTSWillberaisedwhenacoordinatorinaclustertriestocreateacollectionandthecollectionIDalreadyexists.
1454-ERROR_CLUSTER_COULD_NOT_CREATE_COLLECTION_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewcollectioninthePlanhierarchyintheagency.
1455-ERROR_CLUSTER_COULD_NOT_READ_CURRENT_VERSIONWillberaisedwhenacoordinatorinaclustercannotreadtheVersionentryintheCurrenthierarchyintheagency.
1456-ERROR_CLUSTER_COULD_NOT_CREATE_COLLECTIONWillberaisedwhenacoordinatorinaclusternoticesthatsomeDBServersreportproblemswhencreatingshardsforanewcollection.
1457-ERROR_CLUSTER_TIMEOUTWillberaisedwhenacoordinatorinaclusterrunsintoatimeoutforsomeclusterwideoperation.
1458-ERROR_CLUSTER_COULD_NOT_REMOVE_COLLECTION_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanentryforacollectioninthePlanhierarchyintheagency.
1459-ERROR_CLUSTER_COULD_NOT_REMOVE_COLLECTION_IN_CURRENTWillberaisedwhenacoordinatorinaclustercannotremoveanentryforacollectionintheCurrenthierarchyintheagency.
1460-ERROR_CLUSTER_COULD_NOT_CREATE_DATABASE_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewdatabaseinthePlanhierarchyintheagency.
1461-ERROR_CLUSTER_COULD_NOT_CREATE_DATABASEWillberaisedwhenacoordinatorinaclusternoticesthatsomeDBServersreportproblemswhencreatingdatabasesforanewclusterwidedatabase.
1462-ERROR_CLUSTER_COULD_NOT_REMOVE_DATABASE_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanentryforadatabaseinthePlanhierarchyintheagency.
1463-ERROR_CLUSTER_COULD_NOT_REMOVE_DATABASE_IN_CURRENTWillberaisedwhenacoordinatorinaclustercannotremoveanentryforadatabaseintheCurrenthierarchyintheagency.
1464-ERROR_CLUSTER_SHARD_GONEWillberaisedwhenacoordinatorinaclustercannotdeterminetheshardthatisresponsibleforagivendocument.
Errorcodesandmeanings
707
1465-ERROR_CLUSTER_CONNECTION_LOSTWillberaisedwhenacoordinatorinaclusterlosesanHTTPconnectiontoaDBserverintheclusterwhilsttransferringdata.
1466-ERROR_CLUSTER_MUST_NOT_SPECIFY_KEYWillberaisedwhenacoordinatorinaclusterfindsthatthe_keyattributewasspecifiedinashardedcollectiontheusesnotonly_keyasshardingattribute.
1467-ERROR_CLUSTER_GOT_CONTRADICTING_ANSWERSWillberaisedifacoordinatorinaclustergetsconflictingresultsfromdifferentshards,whichshouldneverhappen.
1468-ERROR_CLUSTER_NOT_ALL_SHARDING_ATTRIBUTES_GIVENWillberaisedifacoordinatortriestofindoutwhichshardisresponsibleforapartialdocument,butcannotdothisbecausenotallshardingattributesarespecified.
1469-ERROR_CLUSTER_MUST_NOT_CHANGE_SHARDING_ATTRIBUTESWillberaisedifthereisanattempttoupdatethevalueofashardattribute.
1470-ERROR_CLUSTER_UNSUPPORTEDWillberaisedwhenthereisanattempttocarryoutanoperationthatisnotsupportedinthecontextofashardedcollection.
1471-ERROR_CLUSTER_ONLY_ON_COORDINATORWillberaisedifthereisanattempttorunacoordinator-onlyoperationonadifferenttypeofnode.
1472-ERROR_CLUSTER_READING_PLAN_AGENCYWillberaisedifacoordinatororDBservercannotreadthePlanintheagency.
1473-ERROR_CLUSTER_COULD_NOT_TRUNCATE_COLLECTIONWillberaisedifacoordinatorcannottruncateallshardsofaclustercollection.
1474-ERROR_CLUSTER_AQL_COMMUNICATIONWillberaisediftheinternalcommunicationoftheclusterforAQLproducesanerror.
1475-ERROR_ARANGO_DOCUMENT_NOT_FOUND_OR_SHARDING_ATTRIBUTES_CHANGEDWillberaisedwhenadocumentwithagivenidentifierorhandleisunknown,oriftheshardingattributeshavebeenchangedinaREPLACEoperationinthecluster.
1476-ERROR_CLUSTER_COULD_NOT_DETERMINE_IDWillberaisedifaclusterserveratstartupcouldnotdetermineitsownIDfromthelocalinfoprovided.
1477-ERROR_CLUSTER_ONLY_ON_DBSERVERWillberaisedifthereisanattempttorunaDBserver-onlyoperationonadifferenttypeofnode.
1478-ERROR_CLUSTER_BACKEND_UNAVAILABLEWillberaisedifarequireddbservercan'tbereached.
1479-ERROR_CLUSTER_UNKNOWN_CALLBACK_ENDPOINTAnendpointcouldn'tbefound
1480-ERROR_CLUSTER_AGENCY_STRUCTURE_INVALIDThestructureintheagencyisinvalid
1481-ERROR_CLUSTER_AQL_COLLECTION_OUT_OF_SYNCWillberaisedifacollectionneededduringqueryexecutionisoutofsync.Thiscurrentlycanonlyhappenwhenusingsatellitecollections
1482-ERROR_CLUSTER_COULD_NOT_CREATE_INDEX_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewindexinthePlanhierarchyintheagency.
1483-ERROR_CLUSTER_COULD_NOT_DROP_INDEX_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanindexfromthePlanhierarchyintheagency.
1484-ERROR_CLUSTER_CHAIN_OF_DISTRIBUTESHARDSLIKEWillberaisedifonetriestocreateacollectionwithadistributeShardsLikeattributewhichpointstoanothercollectionthatalsohasone.
Errorcodesandmeanings
708
1485-ERROR_CLUSTER_MUST_NOT_DROP_COLL_OTHER_DISTRIBUTESHARDSLIKEWillberaisedifonetriestodropacollectiontowhichanothercollectionpointswithitsdistributeShardsLikeattribute.
1486-ERROR_CLUSTER_UNKNOWN_DISTRIBUTESHARDSLIKEWillberaisedifonetriestocreateacollectionwhichpointstoanunknowncollectioninitsdistributeShardsLikeattribute.
1487-ERROR_CLUSTER_INSUFFICIENT_DBSERVERSWillberaisedifonetriestocreateacollectionwithareplicationFactorgreaterthantheavailablenumberofDBServers.
1488-ERROR_CLUSTER_COULD_NOT_DROP_FOLLOWERWillberaisedifafollowerthatoughttobedroppedcouldnotbedroppedintheagency(underCurrent).
1489-ERROR_CLUSTER_SHARD_LEADER_REFUSES_REPLICATIONWillberaisedifareplicationoperationisrefusedbyashardleader.
1490-ERROR_CLUSTER_SHARD_FOLLOWER_REFUSES_OPERATIONWillberaisedifanon-replicationoperationisrefusedbyashardfollower.
1491-ERROR_CLUSTER_SHARD_LEADER_RESIGNEDWillberaisedifanon-replicationoperationisrefusedbyaformershardleaderthathasfoundoutthatitisnolongertheleader.
1492-ERROR_CLUSTER_AGENCY_COMMUNICATION_FAILEDWillberaisedifaftervariousretriesanagencyoperationcouldnotbeperformedsuccessfully.
1493-ERROR_CLUSTER_DISTRIBUTE_SHARDS_LIKE_REPLICATION_FACTORWillberaisedifintendedreplicationfactordoesnotmatchthatoftheprototypeshardgivenindistributeShardsLikeparameter.
1494-ERROR_CLUSTER_DISTRIBUTE_SHARDS_LIKE_NUMBER_OF_SHARDSWillberaisedifintendednumberofshardsdoesnotmatchthatoftheprototypeshardgivenindistributeShardsLikeparameter.
1495-ERROR_CLUSTER_LEADERSHIP_CHALLENGE_ONGOINGWillberaisedwhenserversarecurrentlycompetingforleadership,andtheresultisstillunknown.
1496-ERROR_CLUSTER_NOT_LEADERWillberaisedwhenanoperationissenttoanon-leadingserver.
ArangoDBqueryerrors
1500-ERROR_QUERY_KILLEDWillberaisedwhenarunningqueryiskilledbyanexplicitadmincommand.
1501-ERROR_QUERY_PARSEWillberaisedwhenqueryisparsedandisfoundtobesyntacticallyinvalid.
1502-ERROR_QUERY_EMPTYWillberaisedwhenanemptyqueryisspecified.
1503-ERROR_QUERY_SCRIPTWillberaisedwhenaruntimeerroriscausedbythequery.
1504-ERROR_QUERY_NUMBER_OUT_OF_RANGEWillberaisedwhenanumberisoutsidetheexpectedrange.
1510-ERROR_QUERY_VARIABLE_NAME_INVALIDWillberaisedwhenaninvalidvariablenameisused.
1511-ERROR_QUERY_VARIABLE_REDECLAREDWillberaisedwhenavariablegetsre-assignedinaquery.
1512-ERROR_QUERY_VARIABLE_NAME_UNKNOWNWillberaisedwhenanunknownvariableisusedorthevariableisundefinedthecontextitisused.
1521-ERROR_QUERY_COLLECTION_LOCK_FAILEDWillberaisedwhenareadlockonthecollectioncannotbeacquired.
Errorcodesandmeanings
709
1522-ERROR_QUERY_TOO_MANY_COLLECTIONSWillberaisedwhenthenumberofcollectionsinaqueryisbeyondtheallowedvalue.
1530-ERROR_QUERY_DOCUMENT_ATTRIBUTE_REDECLAREDWillberaisedwhenadocumentattributeisre-assigned.
1540-ERROR_QUERY_FUNCTION_NAME_UNKNOWNWillberaisedwhenanundefinedfunctioniscalled.
1541-ERROR_QUERY_FUNCTION_ARGUMENT_NUMBER_MISMATCHWillberaisedwhenthenumberofargumentsusedinafunctioncalldoesnotmatchtheexpectednumberofargumentsforthefunction.
1542-ERROR_QUERY_FUNCTION_ARGUMENT_TYPE_MISMATCHWillberaisedwhenthetypeofanargumentusedinafunctioncalldoesnotmatchtheexpectedargumenttype.
1543-ERROR_QUERY_INVALID_REGEXWillberaisedwhenaninvalidregexargumentvalueisusedinacalltoafunctionthatexpectsaregex.
1550-ERROR_QUERY_BIND_PARAMETERS_INVALIDWillberaisedwhenthestructureofbindparameterspassedhasanunexpectedformat.
1551-ERROR_QUERY_BIND_PARAMETER_MISSINGWillberaisedwhenabindparameterwasdeclaredinthequerybutthequeryisbeingexecutedwithnovalueforthatparameter.
1552-ERROR_QUERY_BIND_PARAMETER_UNDECLAREDWillberaisedwhenavaluegetsspecifiedforanundeclaredbindparameter.
1553-ERROR_QUERY_BIND_PARAMETER_TYPEWillberaisedwhenabindparameterhasaninvalidvalueortype.
1560-ERROR_QUERY_INVALID_LOGICAL_VALUEWillberaisedwhenanon-booleanvalueisusedinalogicaloperation.
1561-ERROR_QUERY_INVALID_ARITHMETIC_VALUEWillberaisedwhenanon-numericvalueisusedinanarithmeticoperation.
1562-ERROR_QUERY_DIVISION_BY_ZEROWillberaisedwhenthereisanattempttodividebyzero.
1563-ERROR_QUERY_ARRAY_EXPECTEDWillberaisedwhenanon-arrayoperandisusedforanoperationthatexpectsanarrayargumentoperand.
1569-ERROR_QUERY_FAIL_CALLEDWillberaisedwhenthefunctionFAIL()iscalledfrominsideaquery.
1570-ERROR_QUERY_GEO_INDEX_MISSINGWillberaisedwhenageorestrictionwasspecifiedbutnosuitablegeoindexisfoundtoresolveit.
1571-ERROR_QUERY_FULLTEXT_INDEX_MISSINGWillberaisedwhenafulltextqueryisperformedonacollectionwithoutasuitablefulltextindex.
1572-ERROR_QUERY_INVALID_DATE_VALUEWillberaisedwhenavaluecannotbeconvertedtoadate.
1573-ERROR_QUERY_MULTI_MODIFYWillberaisedwhenanAQLquerycontainsmorethanonedata-modifyingoperation.
1574-ERROR_QUERY_INVALID_AGGREGATE_EXPRESSIONWillberaisedwhenanAQLquerycontainsaninvalidaggregateexpression.
1575-ERROR_QUERY_COMPILE_TIME_OPTIONSWillberaisedwhenanAQLdata-modificationquerycontainsoptionsthatcannotbefiguredoutatquerycompiletime.
1576-ERROR_QUERY_EXCEPTION_OPTIONSWillberaisedwhenanAQLdata-modificationquerycontainsaninvalidoptionsspecification.
Errorcodesandmeanings
710
1577-ERROR_QUERY_COLLECTION_USED_IN_EXPRESSIONWillberaisedwhenacollectionisusedasanoperandinanAQLexpression.
1578-ERROR_QUERY_DISALLOWED_DYNAMIC_CALLWillberaisedwhenadynamicfunctioncallismadetoafunctionthatcannotbecalleddynamically.
1579-ERROR_QUERY_ACCESS_AFTER_MODIFICATIONWillberaisedwhencollectiondataareaccessedafteradata-modificationoperation.
AQLuserfunctionerrors1580-ERROR_QUERY_FUNCTION_INVALID_NAMEWillberaisedwhenauserfunctionwithaninvalidnameisregistered.
1581-ERROR_QUERY_FUNCTION_INVALID_CODEWillberaisedwhenauserfunctionisregisteredwithinvalidcode.
1582-ERROR_QUERY_FUNCTION_NOT_FOUNDWillberaisedwhenauserfunctionisaccessedbutnotfound.
1583-ERROR_QUERY_FUNCTION_RUNTIME_ERRORWillberaisedwhenauserfunctionthrowsaruntimeexception.
AQLqueryregistryerrors
1590-ERROR_QUERY_BAD_JSON_PLANWillberaisedwhenanHTTPAPIforaquerygotaninvalidJSONobject.
1591-ERROR_QUERY_NOT_FOUNDWillberaisedwhenanIdofaqueryisnotfoundbytheHTTPAPI.
1592-ERROR_QUERY_IN_USEWillberaisedwhenanIdofaqueryisfoundbytheHTTPAPIbutthequeryisinuse.
ArangoDBcursorerrors1600-ERROR_CURSOR_NOT_FOUNDWillberaisedwhenacursorisrequestedviaitsidbutacursorwiththatidcannotbefound.
1601-ERROR_CURSOR_BUSYWillberaisedwhenacursorisrequestedviaitsidbutaconcurrentrequestisstillusingthecursor.
ArangoDBtransactionerrors
1650-ERROR_TRANSACTION_INTERNALWillberaisedwhenawrongusageoftransactionsisdetected.thisisaninternalerrorandindicatesabuginArangoDB.
1651-ERROR_TRANSACTION_NESTEDWillberaisedwhentransactionsarenested.
1652-ERROR_TRANSACTION_UNREGISTERED_COLLECTIONWillberaisedwhenacollectionisusedinthemiddleofatransactionbutwasnotregisteredattransactionstart.
1653-ERROR_TRANSACTION_DISALLOWED_OPERATIONWillberaisedwhenadisallowedoperationiscarriedoutinatransaction.
1654-ERROR_TRANSACTION_ABORTEDWillberaisedwhenatransactionwasaborted.
Errorcodesandmeanings
711
Usermanagementerrors
1700-ERROR_USER_INVALID_NAMEWillberaisedwhenaninvalidusernameisused.
1701-ERROR_USER_INVALID_PASSWORDWillberaisedwhenaninvalidpasswordisused.
1702-ERROR_USER_DUPLICATEWillberaisedwhenausernamealreadyexists.
1703-ERROR_USER_NOT_FOUNDWillberaisedwhenausernameisupdatedthatdoesnotexist.
1704-ERROR_USER_CHANGE_PASSWORDWillberaisedwhentheusermustchangehispassword.
1705-ERROR_USER_EXTERNALWillberaisedwhentheuserisauthenicatedbyanexternalserver.
Servicemanagementerrors(legacy)
ThesehavebeensupercededbytheFoxxmanagementerrorsinpublicAPIs.
1750-ERROR_SERVICE_INVALID_NAMEWillberaisedwhenaninvalidservicenameisspecified.
1751-ERROR_SERVICE_INVALID_MOUNTWillberaisedwhenaninvalidmountisspecified.
1752-ERROR_SERVICE_DOWNLOAD_FAILEDWillberaisedwhenaservicedownloadfromthecentralrepositoryfailed.
1753-ERROR_SERVICE_UPLOAD_FAILEDWillberaisedwhenaserviceuploadfromtheclienttotheArangoDBserverfailed.
LDAPerrors
1800-ERROR_LDAP_CANNOT_INITcannotinitaLDAPconnection
1801-ERROR_LDAP_CANNOT_SET_OPTIONcannotsetaLDAPoption
1802-ERROR_LDAP_CANNOT_BINDcannotbindtoaLDAPserver
1803-ERROR_LDAP_CANNOT_UNBINDcannotunbindfromaLDAPserver
1804-ERROR_LDAP_CANNOT_SEARCHcannotsearchtheLDAPserver
1805-ERROR_LDAP_CANNOT_START_TLScannotstaraTLSLDAPsession
1806-ERROR_LDAP_FOUND_NO_OBJECTSLDAPdidn'tfoundanyobjectswiththespecifiedsearchquery
1807-ERROR_LDAP_NOT_ONE_USER_FOUNDLDAPfoundzerooremorethanoneuser
Errorcodesandmeanings
712
1808-ERROR_LDAP_USER_NOT_IDENTIFIEDLDAPfoundauser,butitsnotthedesiredone
1820-ERROR_LDAP_INVALID_MODEcantdistinguishavalidmodeforprovidedldapconfiguration
Taskerrors
1850-ERROR_TASK_INVALID_IDWillberaisedwhenataskiscreatedwithaninvalidid.
1851-ERROR_TASK_DUPLICATE_IDWillberaisedwhenataskidiscreatedwithaduplicateid.
1852-ERROR_TASK_NOT_FOUNDWillberaisedwhenataskwiththespecifiedidcouldnotbefound.
Graph/traversalerrors
1901-ERROR_GRAPH_INVALID_GRAPHWillberaisedwhenaninvalidnameispassedtotheserver.
1902-ERROR_GRAPH_COULD_NOT_CREATE_GRAPHWillberaisedwhenaninvalidname,verticesoredgesispassedtotheserver.
1903-ERROR_GRAPH_INVALID_VERTEXWillberaisedwhenaninvalidvertexidispassedtotheserver.
1904-ERROR_GRAPH_COULD_NOT_CREATE_VERTEXWillberaisedwhenthevertexcouldnotbecreated.
1905-ERROR_GRAPH_COULD_NOT_CHANGE_VERTEXWillberaisedwhenthevertexcouldnotbechanged.
1906-ERROR_GRAPH_INVALID_EDGEWillberaisedwhenaninvalidedgeidispassedtotheserver.
1907-ERROR_GRAPH_COULD_NOT_CREATE_EDGEWillberaisedwhentheedgecouldnotbecreated.
1908-ERROR_GRAPH_COULD_NOT_CHANGE_EDGEWillberaisedwhentheedgecouldnotbechanged.
1909-ERROR_GRAPH_TOO_MANY_ITERATIONSWillberaisedwhentoomanyiterationsaredoneinagraphtraversal.
1910-ERROR_GRAPH_INVALID_FILTER_RESULTWillberaisedwhenaninvalidfilterresultisreturnedinagraphtraversal.
1920-ERROR_GRAPH_COLLECTION_MULTI_USEanedgecollectionmayonlybeusedonceinoneedgedefinitionofagraph.,
1921-ERROR_GRAPH_COLLECTION_USE_IN_MULTI_GRAPHSisalreadyusedbyanothergraphinadifferentedgedefinition.,
1922-ERROR_GRAPH_CREATE_MISSING_NAMEagraphnameisrequiredtocreateagraph.,
1923-ERROR_GRAPH_CREATE_MALFORMED_EDGE_DEFINITIONtheedgedefinitionismalformed.Ithastobeanarrayofobjects.,
1924-ERROR_GRAPH_NOT_FOUNDagraphwiththisnamecouldnotbefound.,
Errorcodesandmeanings
713
1925-ERROR_GRAPH_DUPLICATEagraphwiththisnamealreadyexists.,
1926-ERROR_GRAPH_VERTEX_COL_DOES_NOT_EXISTthespecifiedvertexcollectiondoesnotexistorisnotpartofthegraph.,
1927-ERROR_GRAPH_WRONG_COLLECTION_TYPE_VERTEXthecollectionisnotavertexcollection.,
1928-ERROR_GRAPH_NOT_IN_ORPHAN_COLLECTIONVertexcollectionnotinorphancollectionofthegraph.,
1929-ERROR_GRAPH_COLLECTION_USED_IN_EDGE_DEFThecollectionisalreadyusedinanedgedefinitionofthegraph.,
1930-ERROR_GRAPH_EDGE_COLLECTION_NOT_USEDTheedgecollectionisnotusedinanyedgedefinitionofthegraph.,
1932-ERROR_GRAPH_NO_GRAPH_COLLECTIONcollection_graphsdoesnotexist.,
1933-ERROR_GRAPH_INVALID_EXAMPLE_ARRAY_OBJECT_STRINGInvalidexampletype.HastobeString,ArrayorObject.,
1934-ERROR_GRAPH_INVALID_EXAMPLE_ARRAY_OBJECTInvalidexampletype.HastobeArrayorObject.,
1935-ERROR_GRAPH_INVALID_NUMBER_OF_ARGUMENTSInvalidnumberofarguments.Expected:,
1936-ERROR_GRAPH_INVALID_PARAMETERInvalidparametertype.,
1937-ERROR_GRAPH_INVALID_IDInvalidid,
1938-ERROR_GRAPH_COLLECTION_USED_IN_ORPHANSThecollectionisalreadyusedintheorphansofthegraph.,
1939-ERROR_GRAPH_EDGE_COL_DOES_NOT_EXISTthespecifiededgecollectiondoesnotexistorisnotpartofthegraph.,
1940-ERROR_GRAPH_EMPTYTherequestedgraphhasnoedgecollections.
Sessionerrors1950-ERROR_SESSION_UNKNOWNWillberaisedwhenaninvalid/unknownsessionidispassedtotheserver.
1951-ERROR_SESSION_EXPIREDWillberaisedwhenasessionisexpired.
SimpleClienterrors
2000-SIMPLE_CLIENT_UNKNOWN_ERRORThiserrorshouldnothappen.
2001-SIMPLE_CLIENT_COULD_NOT_CONNECTWillberaisedwhentheclientcouldnotconnecttotheserver.
2002-SIMPLE_CLIENT_COULD_NOT_WRITEWillberaisedwhentheclientcouldnotwritedata.
Errorcodesandmeanings
714
2003-SIMPLE_CLIENT_COULD_NOT_READWillberaisedwhentheclientcouldnotreaddata.
Communicatorerrors
2100-COMMUNICATOR_REQUEST_ABORTEDRequestwasaborted.
2101-COMMUNICATOR_DISABLEDCommunicationwasdisabled.
Foxxmanagementerrors
3000-ERROR_MALFORMED_MANIFEST_FILETheservicemanifestfileisnotwell-formedJSON.
3001-ERROR_INVALID_SERVICE_MANIFESTTheservicemanifestcontainsinvalidvalues.
3002-ERROR_SERVICE_FILES_MISSINGTheservicefolderorbundledoesnotexistonthisserver.
3003-ERROR_SERVICE_FILES_OUTDATEDThelocalservicebundledoesnotmatchthechecksuminthedatabase.
3004-ERROR_INVALID_FOXX_OPTIONSTheserviceoptionscontaininvalidvalues.
3007-ERROR_INVALID_MOUNTPOINTTheservicemountpathcontainsinvalidcharacters.
3009-ERROR_SERVICE_NOT_FOUNDNoservicefoundatthegivenmountpath.
3010-ERROR_SERVICE_NEEDS_CONFIGURATIONTheserviceismissingconfigurationordependencies.
3011-ERROR_SERVICE_MOUNTPOINT_CONFLICTAservicealreadyexistsatthegivenmountpath.
3012-ERROR_SERVICE_MANIFEST_NOT_FOUNDTheservicedirectorydoesnotcontainamanifestfile.
3013-ERROR_SERVICE_OPTIONS_MALFORMEDTheserviceoptionsarenotwell-formedJSON.
3014-ERROR_SERVICE_SOURCE_NOT_FOUNDThesourcepathdoesnotmatchafileordirectory.
3015-ERROR_SERVICE_SOURCE_ERRORThesourcepathcouldnotberesolved.
3016-ERROR_SERVICE_UNKNOWN_SCRIPTTheservicedoesnothaveascriptwiththisname.
JavaScriptmoduleloadererrors3100-ERROR_MODULE_NOT_FOUNDThemodulepathcouldnotberesolved.
Errorcodesandmeanings
715
3101-ERROR_MODULE_SYNTAX_ERRORThemodulecouldnotbeparsedbecauseofasyntaxerror.
3103-ERROR_MODULE_FAILUREFailedtoinvokethemoduleinitscontext.
Enterpriseerrors
4000-ERROR_NO_SMART_COLLECTIONTherequestedcollectionneedstobesmart,butitain't
4001-ERROR_NO_SMART_GRAPH_ATTRIBUTEThegivendocumentdoesnothavethesmartgraphattributeset.
4002-ERROR_CANNOT_DROP_SMART_COLLECTIONThissmartcollectioncannotbedropped,itdictatesshardinginthegraph.
4003-ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTEInasmartvertexcollection_keymustbeprefixedwiththevalueofthesmartgraphattribute.
4004-ERROR_ILLEGAL_SMART_GRAPH_ATTRIBUTEThegivensmartGraphattributeisillegalandconnotbeusedforsharding.Allsystemattributesareforbidden.
Clusterrepairerrors
5000-ERROR_CLUSTER_REPAIRS_FAILEDGeneralerrorduringclusterrepairs
5001-ERROR_CLUSTER_REPAIRS_NOT_ENOUGH_HEALTHYWillberaisedwhen,duringrepairDistributeShardsLike,theremustbeafreedbservertomoveashard,butthereisnocandidateornoneishealthy.
5002-ERROR_CLUSTER_REPAIRS_REPLICATION_FACTOR_VIOLATEDWillberaisedonvariousinconsistenciesregardingthereplicationfactor
5003-ERROR_CLUSTER_REPAIRS_NO_DBSERVERSWillberaisedifacollectionthatisfixedhassomeshardwithoutDBServers
5004-ERROR_CLUSTER_REPAIRS_MISMATCHING_LEADERSWillberaisedifashardincollectionanditsprototypeinthecorrespondingdistributeShardsLikecollectionhavemismatchingleaders(whentheyshouldalreadyhavebeenfixed)
5005-ERROR_CLUSTER_REPAIRS_MISMATCHING_FOLLOWERSWillberaisedifashardincollectionanditsprototypeinthecorrespondingdistributeShardsLikecollectiondon'thavethesamefollowers(whentheyshouldalreadyhavebeenadjusted)
5006-ERROR_CLUSTER_REPAIRS_INCONSISTENT_ATTRIBUTESWillberaisedifacollectionthatisfixeddoes(not)havedistributeShardsLikewhenitisexpected,ordoes(not)haverepairingDistributeShardsLikewhenitisexpected
5007-ERROR_CLUSTER_REPAIRS_MISMATCHING_SHARDSWillberaisedifinacollectionanditsdistributeShardsLikeprototypecollectionsomeshardanditsprototypehaveanunequalnumberofDBServers
5008-ERROR_CLUSTER_REPAIRS_JOB_FAILEDWillberaisedifamoveshardjobintheagencyfailedduringclusterrepairs
5009-ERROR_CLUSTER_REPAIRS_JOB_DISAPPEAREDWillberaisedifamoveshardjobintheagencycannotbefoundanymorebeforeitfinished
5010-ERROR_CLUSTER_REPAIRS_OPERATION_FAILEDWillberaisedifanagencytransactionfailedduringeithersendingorexecutingit.
Errorcodesandmeanings
716
Agencyerrors
20001-ERROR_AGENCY_INQUIRY_SYNTAXInquiryhandlesalistofstringclientIds:[,...].
20011-ERROR_AGENCY_INFORM_MUST_BE_OBJECTTheinformmessageintheagencymustbeanobject.
20012-ERROR_AGENCY_INFORM_MUST_CONTAIN_TERMTheinformmessageintheagencymustcontainauintparameter'term'.
20013-ERROR_AGENCY_INFORM_MUST_CONTAIN_IDTheinformmessageintheagencymustcontainastringparameter'id'.
20014-ERROR_AGENCY_INFORM_MUST_CONTAIN_ACTIVETheinformmessageintheagencymustcontainanarray'active'.
20015-ERROR_AGENCY_INFORM_MUST_CONTAIN_POOLTheinformmessageintheagencymustcontainanobject'pool'.
20016-ERROR_AGENCY_INFORM_MUST_CONTAIN_MIN_PINGTheinformmessageintheagencymustcontainanobject'minping'.
20017-ERROR_AGENCY_INFORM_MUST_CONTAIN_MAX_PINGTheinformmessageintheagencymustcontainanobject'maxping'.
20018-ERROR_AGENCY_INFORM_MUST_CONTAIN_TIMEOUT_MULTTheinformmessageintheagencymustcontainanobject'timeoutMult'.
20020-ERROR_AGENCY_INQUIRE_CLIENT_ID_MUST_BE_STRINGInquirybyclientIdfailed
20021-ERROR_AGENCY_CANNOT_REBUILD_DBSWillberaisedifthereadDBorthespearHeadcannotberebuiltfromthereplicatedlog.
Supervisionerrors
20501-ERROR_SUPERVISION_GENERAL_FAILUREGeneralsupervisionfailure.
Dispatchererrors
21001-ERROR_DISPATCHER_IS_STOPPINGWillbereturnedifashutdownisinprogress.
21002-ERROR_QUEUE_UNKNOWNWillbereturnedifaqueuewiththisnamedoesnotexist.
21003-ERROR_QUEUE_FULLWillbereturnedifaqueuewiththisnameisfull.
Errorcodesandmeanings
717
Glossary
Collection
Acollectionconsistsofdocuments.Itisuniquelyidentifiedbyitscollectionidentifier.Italsohasauniquenamethatclientsshouldusetoidentifyandaccessit.Collectionscanberenamed.Itwillchangethecollectionname,butnotthecollectionidentifier.Collectionscontaindocumentsofaspecifictype.Therearecurrentlytwotypes:document(default)andedge.Thetypeisspecifiedbytheuserwhenthecollectioniscreated,andcannotbechangedlater.
CollectionIdentifier
Acollectionidentifieridentifiesacollectioninadatabase.Itisastringvalueandisuniquewithinthedatabase.UptoincludingArangoDB1.1,thecollectionidentifierhasbeenaclient'sprimarymeanstoaccesscollections.StartingwithArangoDB1.2,clientsshouldinsteaduseacollection'suniquenametoaccessacollectioninsteadofitsidentifier.
ArangoDBcurrentlyuses64bitunsignedintegervaluestomaintaincollectionidsinternally.Whenreturningcollectionidstoclients,ArangoDBwillputthemintoastringtoensurethecollectionidisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreatthecollectionidsreturnedbyArangoDBasopaquestringswhentheystoreoruseitlocally.
CollectionName
Acollectionnameidentifiesacollectioninadatabase.Itisastringandisuniquewithinthedatabase.Unlikethecollectionidentifieritissuppliedbythecreatorofthecollection.Thecollectionnamemustconsistofletters,digits,andthe_(underscore)and-(dash)charactersonly.PleaserefertoNamingConventionsformoreinformationonvalidcollectionnames.
Database
ArangoDBcanhandlemultipledatabasesinthesameserverinstance.Databasescanbeusedtologicallygroupandseparatedata.AnArangoDBdatabaseconsistsofcollectionsanddedicateddatabase-specificworkerprocesses.
Adatabasecontainsitsowncollections(whichcannotbeaccessedfromotherdatabases),Foxxapplications,andreplicationloggersandappliers.EachArangoDBdatabasecontainsitsownsystemcollections(e.g._users,_replication,...).
TherewillalwaysbeatleastonedatabaseinArangoDB.Thisisthedefaultdatabase,named_system.Thisdatabasecannotbedropped,andprovidesspecialoperationsforcreating,dropping,andenumeratingdatabases.Userscancreateadditionaldatabasesandgivethemuniquenamestoaccessthemlater.Databasemanagementoperationscannotbeinitiatedfromoutofuser-defineddatabases.
WhenArangoDBisaccessedviaitsHTTPRESTAPI,thedatabasenameisreadfromthefirstpartoftherequestURIpath(e.g./_db/_system/...).IftherequestURIdoesnotcontainadatabasename,thedatabasenameisautomaticallyderivedfromtheendpoint.PleaserefertoDatabaseEndpointformoreinformation.
DatabaseName
AsingleArangoDBinstancecanhandlemultipledatabasesinparallel.Whenmultipledatabasesareused,eachdatabasemustbegivenauniquename.Thisnameisusedtouniquelyidentifyadatabase.ThedefaultdatabaseinArangoDBisnamed_system.
Thedatabasenameisastringconsistingofonlyletters,digitsandthe_(underscore)and-(dash)characters.User-defineddatabasenamesmustalwaysstartwithaletter.Databasenamesiscase-sensitive.
DatabaseOrganization
AsingleArangoDBinstancecanhandlemultipledatabasesinparallel.Bydefault,therewillbeatleastonedatabase,whichisnamed_system.
Databasesarephysicallystoredinseparatesub-directoriesunderneaththedatabasedirectory,whichitselfresidesintheinstance'sdatadirectory.
Glossary
718
Eachdatabasehasitsownsub-directory,nameddatabase-.Thedatabasedirectorycontainssub-directoriesforthecollectionsofthedatabase,andafilenamedparameter.json.Thisfilecontainsthedatabaseidandname.
InanexampleArangoDBinstancewhichhastwodatabases,thefilesystemlayoutcouldlooklikethis:
data/#theinstance'sdatadirectory
databases/#sub-directorycontainingalldatabases'data
database-<id>/#sub-directoryforasingledatabase
parameter.json#filecontainingdatabaseidandname
collection-<id>/#directorycontainingdataaboutacollection
database-<id>/#sub-directoryforanotherdatabase
parameter.json#filecontainingdatabaseidandname
collection-<id>/#directorycontainingdataaboutacollection
collection-<id>/#directorycontainingdataaboutacollection
Foxxapplicationsarealsoorganizedindatabase-specificdirectoriesinsidetheapplicationpath.Thefilesystemlayoutcouldlooklikethis:
apps/#theinstance'sapplicationdirectory
system/#systemapplications(canbeignored)
_db/#sub-directorycontainingdatabase-specificapplications
<database-name>/#sub-directoryforasingledatabase
<mountpoint>/APP#sub-directoryforasingleapplication
<mountpoint>/APP#sub-directoryforasingleapplication
<database-name>/#sub-directoryforanotherdatabase
<mountpoint>/APP#sub-directoryforasingleapplication
Document
DocumentsinArangoDBareJSONobjects.Theseobjectscanbenested(toanydepth)andmaycontainarrays.Eachdocumentisuniquelyidentifiedbyitsdocumenthandle.
DocumentEtag
Thedocumentrevision(_revvalue)enclosedindoublequotes.TherevisionisreturnedbyseveralHTTPAPImethodsintheEtagHTTPheader.
DocumentHandle
Adocumenthandleuniquelyidentifiesadocumentinthedatabase.Itisastringandconsistsofthecollection'snameandthedocumentkey(_keyattribute)separatedby/.Thedocumenthandleisstoredinadocument's_idattribute.
DocumentKey
Adocumentkeyisastringthatuniquelyidentifiesadocumentinagivencollection.Itcanandshouldbeusedbyclientswhenspecificdocumentsaresearched.Documentkeysarestoredinthe_keyattributeofdocuments.ThekeyvaluesareautomaticallyindexedbyArangoDBinacollection'sprimaryindex.Thuslookingupadocumentbyitskeyisregularlyafastoperation.The_keyvalueofadocumentisimmutableoncethedocumenthasbeencreated.
Bydefault,ArangoDBwillauto-generateadocumentkeyifno_keyattributeisspecified,andusetheuser-specified_keyvalueotherwise.
Thisbehaviorcanbechangedonaper-collectionlevelbycreatingcollectionswiththekeyOptionsattribute.
UsingkeyOptionsitispossibletodisallowuser-specifiedkeyscompletely,ortoforceaspecificregimeforauto-generatingthe_keyvalues.
Therearesomerestrictionsforuser-definedkeys(seeNamingConventionsfordocumentkeys).
DocumentRevision
Glossary
719
AsArangoDBsupportsMVCC,documentscanexistinmorethanonerevision.ThedocumentrevisionistheMVCCtokenusedtoidentifyaparticularrevisionofadocument.Itisastringvaluecurrentlycontaininganintegernumberandisuniquewithinthelistofdocumentrevisionsforasingledocument.Documentrevisionscanbeusedtoconditionallyupdate,replaceordeletedocumentsinthedatabase.Inordertofindaparticularrevisionofadocument,youneedthedocumenthandleandthedocumentrevision.
Thedocumentrevisionisstoredinthe_revattributeofadocument,andissetandupdatedbyArangoDBautomatically.The_revvaluecannotbesetfromtheoutside.
ArangoDBcurrentlyuses64bitunsignedintegervaluestomaintaindocumentrevisionsinternally.Whenreturningdocumentrevisionstoclients,ArangoDBwillputthemintoastringtoensuretherevisionidisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreattherevisionidreturnedbyArangoDBasanopaquestringwhentheystoreoruseitlocally.ThiswillallowArangoDBtochangetheformatofrevisionidslaterifthisshouldberequired.Clientscanuserevisionsidstoperformsimpleequality/non-equalitycomparisons(e.g.tocheckwhetheradocumenthaschangedornot),buttheyshouldnotuserevisionidstoperformgreater/lessthancomparisonswiththemtocheckifadocumentrevisionisolderthanoneanother,evenifthismightworkforsomecases.
Edge
Edgesarespecialdocumentsusedforconnectingotherdocumentsintoagraph.Anedgedescribestheconnectionbetweentwodocumentsusingtheinternalattributes:_fromand_to.Thesecontaindocumenthandles,namelythestart-pointandtheend-pointoftheedge.
EdgeCollection
Edgecollectionsarecollectionsthatstoreedges.
EdgeDefinition
Edgedefinitionsarepartsofthedefinitionofnamedgraphs.Theydescribewhichedgecollectionsconnectwhichvertexcollections.
GeneralGraph
Modulemaintaininggraphsetupinthe_graphscollection-akanamedgraphs.Configureswhichedgecollectionsrelatetowhichvertexcollections.Ensuresgraphconsistencyinmodificationqueries.
NamedGraphs
Namedgraphsenforceconsistencybetweenedgecollectionsandvertexcollections,soifyouremoveavertex,edgespointingtoitwillberemovedtoo.
Index
Indexesareusedtoallowfastaccesstodocumentsinacollection.Allcollectionshaveaprimaryindex,whichisthedocument's_keyattribute.Thisindexcannotbedroppedorchanged.
Edgecollectionswillalsohaveanautomaticallycreatededgesindex,whichcannotbemodified.Thisindexprovidesquickaccesstodocumentsviathe_fromand_toattributes.
Mostuser-landindexescanbecreatedbydefiningthenamesoftheattributeswhichshouldbeindexed.Someindextypesallowindexingjustoneattribute(e.g.fulltextindex)whereasotherindextypesallowindexingmultipleattributes.
Indexingthesystemattribute_idinuser-definedindexesisnotsupportedbyanyindextype.
EdgesIndex
Anedgesindexisautomaticallycreatedforedgecollections.Itcontainsconnectionsbetweenvertexdocumentsandisinvokedwhentheconnectingedgesofavertexarequeried.Thereisnowaytoexplicitlycreateordeleteedgesindexes.
FulltextIndex
Glossary
720
Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.Afulltextindexcanbedefinedononeattributeonly,andwillincludeallwordscontainedindocumentsthathaveatextualvalueintheindexattribute.SinceArangoDB2.6theindexwillalsoincludewordsfromtheindexattributeiftheindexattributeisanarrayofstrings,oranobjectwithstringvaluemembers.
Forexample,givenafulltextindexonthetranslationsattributeandthefollowingdocuments,thensearchingforлисаusingthefulltextindexwouldreturnonlythefirstdocument.SearchingfortheindexfortheexactstringFoxwouldreturnthefirsttwodocuments,andsearchingforprefix:Foxwouldreturnallthreedocuments:
{translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}}
{translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"}
{translations:["ArangoDB","document","database","Foxx"]}
Iftheindexattributeisneitherastring,anobjectoranarray,itscontentswillnotbeindexed.Whenindexingthecontentsofanarrayattribute,anarraymemberwillonlybeincludedintheindexifitisastring.Whenindexingthecontentsofanobjectattribute,anobjectmembervaluewillonlybeincludedintheindexifitisastring.Otherdatatypesareignoredandnotindexed.
Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.Wordsareindexedintheirlower-casedform.Theindexsupportscompletematchqueries(fullwords)andprefixqueries.
GeoIndex
Ageoindexisusedtofindplacesonthesurfaceoftheearthfast.
IndexHandle
Anindexhandleuniquelyidentifiesanindexinthedatabase.Itisastringandconsistsofacollectionnameandanindexidentifierseparatedby/.
HashIndex
Ahashindexisusedtofinddocumentsbasedonexamples.Ahashindexcanbecreatedforoneormultipledocumentattributes.
Ahashindexwillonlybeusedbyqueriesifallindexedattributesarepresentintheexampleorsearchquery,andifallattributesarecomparedusingtheequality(==operator).Thatmeansthehashindexdoesnotsupportrangequeries.
AuniquehashindexhasanamortizedcomplexityofO(1)forlookup,insert,update,andremoveoperations.Thenon-uniquehashindexissimilar,butamortizedlookupperformanceisO(n),withnbeingthenumberofindexentrieswiththesamelookupvalue.
SkiplistIndex
Askiplistisasortedindextypethatcanbeusedtofindrangesofdocuments.
AnonymousGraphs
Youmayuseedgecollectionswithvertexcollectionswithoutthegraphmanagementfacilities.However,graphconsistencyisnotenforcedbythese.Ifyouremovevertices,youhavetoensurebyyourselvesedgespointingtothisvertexareremoved.Anonymousgraphsmaynotbebrowsedusinggraphviewerinthewebinterface.Thismaybefasterinsomescenarios.
Glossary
721