Creating a simplified global unique file catalogue
description
Transcript of Creating a simplified global unique file catalogue
![Page 1: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/1.jpg)
Creating a simplified global unique file catalogue
Miguel Martinez PedreiraPablo Saiz
![Page 2: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/2.jpg)
2
Motivations Reaching memory limits in the actual catalogue database machine(s) Catalogue is now ~700 Gigabytes Some tables are too large (in G#L and L#L)
e.g. G132L – 55 GB, tried to split Conversion old-catalogue to new-catalogue takes too long
Between 15 and 20 hours Stuck on some of these large tables Several mysql configurations (threads, memory, pools...) Tried on alientest03 and alientest07 Helped to find things that shouldn’t be there
In search of better performance
![Page 3: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/3.jpg)
3
Motivations Tendency points to even bigger increase
![Page 4: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/4.jpg)
4
Previous status Catalogue implementing several-host database
In order to separate the database into different machines Never used in practice: users and data in the same host
Hosts Index No referential integrity
![Page 5: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/5.jpg)
5
Current status (production) Only references between G#L_REF and G#L_PFN
with G#L, and them with SE Found some entries that shouldn’t be there
e.g. PFNs with GUIDs that don’t exist some unused or temporary tables (old
functionalities, L#L without _s) ~250L tables (x3) ~100 G tables (x4)
![Page 6: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/6.jpg)
6
Improvements Complete referential integrity: FKs all around
consistent database InnoDB tables row level locking using ids (instead of text fields) for referencing
normalized data, surrogated keys
Cleaning tables, renaming fields Fixing some keys
Lead to v2-20 version (currently in PANDA)
![Page 7: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/7.jpg)
7
Link to documentation!
x 100 x 250
![Page 8: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/8.jpg)
8
How to improve New (Pablo’s) idea: Filesystem catalogue No database, instead the catalogue is a UNIX-like filesystem Using metadata files to keep the necessary information
aggregating this information was quite similar to having the DB itself...
Tried/discussed about different available filesystems dealing with inodes, file limits, performance...
Performance was measured unfortunately, it was too slow
What else? -> guidless catalogue
![Page 9: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/9.jpg)
9
GUIDs GUID = Global Unique IDentifier
First decided to have GUIDs, because it was expected that LFNs would change the name very often, and GUIDs were something static
They give flexibility and functionalities
Very nice for mirroring
Permissions per LFN and per GUID e.g. for having reduced permissions on links
Being used for the quotas too
BUT: not really used in practice, and adds quite a lot of complexity compared to the benefits
LFN
LFNGUID PFN
PFN
PFN
![Page 10: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/10.jpg)
10
Deleting GUIDs We plan to use the LFN combined with the
timestamp to maintain its uniqueness So you can differ between them when they are
deleted and a new LFN with the same name is registered
File naming for storage in SEs was based on GUIDs now is LFN+timestamp: more human-readable
Simplicity 240GB directly gone (~35% DB)
![Page 11: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/11.jpg)
11
Links At this moment we have the ‘special’ PFN ‘guid://’ for
handling links A link is a LFN pointing to other LFN Using GUIDs is more flexible (- consistent)
Our design implies pointing inside the same index: but is the use case in production (job outputs and archives).
This change helps us to get rid of 65% of the PFN entries-> 65% of 160GB = 100GB
Slightly different from UNIX symlinks, AliEn links point to part of an archive, not the full file
![Page 12: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/12.jpg)
12
linkId
Link to documentation!
x 100 x 250
![Page 13: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/13.jpg)
13
Expectations GUIDs gone, LFNs stay the same, PFN significantly reduced
if there was consistency, GUIDs should be below LFNs
35% PFNs (350000) should be similar to the number of stored files in SEs (310000)
![Page 14: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/14.jpg)
14
Expectations We get rid of ~50% of the catalogue in terms of size (240GB from
GUIDs + 100GB from links) Performance based on all the changes expected to really increase
We’ve had good experience with the TaskQueue Looks like simple changes from a high-level view
but it involves a lot of effort to check and adapt the code GUIDs everywhere
Found old, miss working, improvable code When?
May-Jun all tests working (AliEn test set) ALICE_TEST prepared in July to ‘really’ test
![Page 15: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/15.jpg)
15
Testing Desired a production-like environment
Not only for the new catalogue, but any change in AliEn Going back to ALICE_TEST
Already had first stages of trains and standard jobs running We want to synchronize the test catalogue with the
production one And to execute production jobs We know is not that easy and beautiful (e.g. API)
Aiming to have newer versions of AliEn in production In a ‘safe’ way Test cases are very hard to cover
![Page 16: Creating a simplified global unique file catalogue](https://reader035.fdocuments.us/reader035/viewer/2022062323/568164bd550346895dd6cc29/html5/thumbnails/16.jpg)
16
To sum up File Catalogue is getting unmaintainable Is a critical part in AliEn Some improvements implemented, we need more Moving to a GUIDless catalogue Planning to have production-like testing
and newer versions easier