Flood: Multi-Situate Responsive Big Data Supervision for Efficient Workflows on Clouds

The global deployment of cloud datacenters is enabling large scale scientific workflows to improve performance and deliver fast responses. Overflow advises a set of pluggable services, convened in a data scientist cloud kit. They provide the applications with the possibility to monitor the underlying infrastructure, to exploit smart data compression, duplication and geo-replication, to evaluate data supervision costs, to set a balance between money and time, and optimize the transfer tactic accordingly. The system was validated on the Microsoft Azure cloud across its 6 EU and US datacenters. The experiments were conducted on hundreds of nodes using fake levels and real-life bio-informatics applications (A-Brain, BLAST). The results show that our system is able to model accurately the cloud performance and to leverage this for efficient data diffusion, being able to reduce the fiscal costs and transfer time by up to three times.