Automating file transfers January 2015 webinar
-
Upload
filecatalyst -
Category
Software
-
view
69 -
download
1
Transcript of Automating file transfers January 2015 webinar
Introduction
• John Tkaczewski, President and Co-Founder
• Check out our blog: http://www.filecatalyst.com/blog
| © Copyright FileCatalyst, 20153
Overview
• FileCatalyst Introduction
• Challenges with Automated Workflows
• Demo HotFolder
• Final Remarks
• Questions
| © Copyright FileCatalyst, 20154
What is File Transfer Acceleration?
• File transfer acceleration technology allows large files, or “big data” to be sent quickly over large geographical distances without being impeded by network impairments such as latency and packet loss
• This is accomplished by transferring files using the UDP protocol instead of the TCP protocol, the protocol used for FTP
• The main difference between UDP and TCP is that UDP wastes no time communicating about the receipt of a block of data so that regardless of network latency, data transmission remains constant, thus overcoming the peak and valley effect that occurs with TCP based protocols on links with high latency
• Reliability and congestion control is added to UDP at the application layer thus it doesn’t sacrifice its other desirable properties
| © Copyright FileCatalyst, 20155
Automation, there is nothing to it!
Setup Cron Job using SCP… and I’m done right?
…. Well Perhaps Not!
| © Copyright FileCatalyst, 20157
Challenges of Automated Workflows
• Volume of data
• System Notifications (message fatigue)
• System properties (system clock, mounts, network drives)
• Interpreting logs and finding errors
• Folder Synchronization
• Cross platform transfers, ex: Windows to Linux
• Transfer Speed
• Data Verification (Checksum)| © Copyright FileCatalyst, 20158
Automation tools within FileCatalyst (Cont’d)
Hotfolder features useful for data replication:• Scheduler for tasks and bandwidth• File System Events• Delta (RSYNC)• Data-set rules• Monitoring and notifications• ZIP chunking (Single Archive)• MD5 • File transfer cache• Multi-Client (multiple concurrent file transfers)
| © Copyright FileCatalyst, 201510
Platform specific gotchas
Common file attributes for all platforms:
File name, Date Created, Date Modified, Owner
| © Copyright FileCatalyst, 201512
Operating System File Attributes
MacOS Extended attributes (xattr), Resource Forks, Symlinks, Permissions (Read Write Execute), Access Control Levels (ACL’s)
Windows Permissions, Attributes: Read Only – Archive - Hidden – System, Security Attributes
Linux Permissions (group, user, other)
Data replication across operating systems
• Extended File Attributes are LOST when moving a file between operating systems.
• But it’s OK because extended attributes may not be always needed.
| © Copyright FileCatalyst, 201513
File system events
The file system events provides a way for an application to ask for notification when the contents of a directory hierarchy are modified.
For example, an application can use this to quickly detect when the user modifies a file within a folder using another application.
• Avoid scanning large folders to detect changes! Use file system events
• File system events only work on local drives (no SMB) Most Enterprise network drive technology does support file system events
• Transfer the initial file set without file system events, THEN turn it on.
| © Copyright FileCatalyst, 201514
Upcoming Events
- FileCatalyst User Forum, London, UK (around BVE)
- BVE 2015, Excel, London, UK (Feb 24-26), stand R40
- CABSAT 2015, Dubai World Trade Centre (March 10-12), stand 220
- NAB Show 2015, Las Vegas, Nevada (April 13-16), stand SL8505
And of course don’t forget our monthly webinars... And …
… our email newsletter “The Transfer Express”
| © Copyright FileCatalyst, 201515