The File Synchronisation tool copies changed files between multiple servers, and is heavily optimised to perform very quickly on high latency networks with high security restrictions.

The Problem

At Standard & Poor’s (S&P), all client facing websites were hosted in New York to prevent arbitrage issues caused by timing delays of updates in different geographies. To minimise latency issues and to provide system resilience from network outages, all report content for these websites was generated to file on Australian systems and then copied to the NY servers.

As the business wanted to move to near-real-time publishing of report content, it became apparent that the overnight copying of files was unacceptable, and hourly or on-demand updates were required.

Traditional file copy or FTP between servers would take hours to copy all files, and various 3rd party file synchronisation tools (that would compare files and only copy changed files) were still taking 20-30 minutes. Even rsync (the Unix standard for synchronising files) was non-performant because it was not optimised for the 500ms network latency between Australia and New York.

The Solution

The custom built solution used advanced algorithms with network compression, and dual-hosted agents to minimise requests and latency. A full copy would take 2-3 minutes, and synchronisation of changes occurred in seconds.

The Benefits

This provided the business with high confidence that the correct data was always being shown on the client facing websites, and that corrections or updates could be reliably published in seconds.