DATA MIGRATION FOR LARGE SCIENTIFIC DATASETS IN CLOUDS
DOI:
https://doi.org/10.32010/26166127.2018.1.1.66.86Keywords:
Data management, Data transfer, Data migration, Cloud storageAbstract
Transferring large data files between various storages including cloud storages is an important task both for academic and commercial users. This should be done in an efficient and secure way. The paper describes Data Avenue that fulfills all these conditions. Data Avenue can efficiently transfer large files even in the range of TerraBytes among storages having very different access protocols (Amazon S3, OpenStack Swift, SFTP, SRM, iRODS, etc.). It can be used in personal, organizational and public deployment with all the security mechanisms required for these usage configurations. Data Avenue can be used by a GUI as well as by a REST API. The papers describes in detail all these features and usage modes of Data Avenue and also provides performance measurement results proving the efficiency of the tool that can be accessed and used via several public web pages.
Downloads
References
[1]. Allcock, W. (2003) GridLTP: Protocol Extensions to FTP for the Grid, Global Grid ForumGFD-R-P.020.
[2]. Shoshani, A. (2002) Storage Resource Management, GGF-4. Retrieved from: https://sdm.lbl.gov/srm-wg/doc/02.02.srm.joint.design/index.htm
[3]. Lemaitre, S., Frohner, A., Baud, J.P., Smith, D., Nienartowicz, K., Abadie, L., Mollon, R. (2007) Recent developments in LFC. CHEP07.
[4]. Hajnal, A., Marton, I., Farkas, Z., Kacsuk, P., Remote storage management in science gateways via data bridging, Concurrency and Computation: Practice and Experience, 27 (16)., 4398-4411.
[5]. Kacsuk, P, Farkas, Z., Kozlovszky, M., Herman, G., Balasko, A., Karoczkai, K., Marton, I. (2012) WS-PGRADE/gUSE generic DCI gateway framework for a large variety of user communities”, Journal of Grid Computing, 10(4).
[6]. Kiss, T., Kacsuk, P., Kovacs, J., Rakoczi,B., Hajnal, A., Farkas, A., Gesmier, G., Terstyanszky, G. (2017) ’MiCADOMicroservice-based Cloud Application-level Dynamic Orchestrator, Future Generation Computer Systems.
[7]. MTA Cloud website https://cloud.mta.hu/
[8]. HBONE website https://www.niif.hu/en/hbone_hbone [9]. Docker website https://www.docker.com/
[10]. Data Avenue website https://data-avenue.eu/
[11]. Ceph website https://ceph.com/
[12].AWS website https://aws.amazon.com/
[13].Apache Tomcat website https://tomcat.apache.org
[14].A. William, J. Bresnahan, R. Kettimuthu, M. Link, C. Dumitrescu, I. Raicu, and I. Foster, The Globus striped GridFTP framework and server, in proceedings of the 2005 ACM/IEEE conference on Supercomputing, IEEE Computer Society, p. 54, 2005.
[15]. Havard Heido Holm, Jon M. Hjelmervik, Volkan Gezer, ’’CloudFlowAnlnfrastructureforEngineeringWorkowsintheCloud”, UBICOMM 2016 : The 24 Tenth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (2016)
[16]. COLA project website https://project-cola.eu/
[17]. Jozsef Kovacs, Peter Kacsuk, ’’Occopus: a Multi-Cloud Orchestrator to Deployand Manage Complex Scientific Infrastructures”, Journal of Grid Computing, Volume 16, Issue 1, pp 1937, 2018
[18]. Occopus website http://occopus.lpds.sztaki.hu/
