Capture - AWS DataSync - Deployment Guide

Overview

This guide provides an overview of AWS DataSync, and a walkthrough for deploying a DataSync OVA template within a VMware ESXi Hypervisor environment. 

TLDR;

1. Download the DataSync Agent (OVA) 
2. Deploy the DataSync Agent 
3. Complete Network Configuration. 
4. Provide Nuclei with activation key
5. Provide Nuclei with the LAN IP address and path of the network share 

What is AWS DataSync?

AWS DataSync makes it simple and fast to move large amounts of data online between on-premises storage and Amazon S3, Amazon Elastic File System (Amazon EFS), or Amazon FSx for Windows File Server. Manual tasks related to data transfers can slow down migrations and burden IT operations. DataSync eliminates or automatically handles many of these tasks, including scripting copy jobs, scheduling and monitoring transfers, validating data, and optimizing network utilization.

How does it work? 

The DataSync software agent connects to your Network File System (NFS) and Server Message Block (SMB) storage, so you don’t have to modify your applications. DataSync can transfer hundreds of terabytes and millions of files at speeds up to 10 times faster than open-source tools, over the internet or AWS Direct Connect links. You can use DataSync to migrate active data sets or archives to AWS, transfer data to the cloud for timely analysis and processing, or replicate data to AWS for business continuity. 

If you have on-premises systems generating or using data that needs to move into or out of AWS for processing, you can use DataSync to accelerate and schedule the transfers. It can help speed up critical hybrid cloud workflows in industries that need to move active files into AWS quickly, including video production in media and entertainment, seismic research in oil and gas, machine learning in life science, and big data analytics in finance.

More information on AWS DataSync is available here. 

Prerequisites

DataSync Agent 

DataSync runs on-premise as a virtual machine (VM).

DataSync supports the following hypervisor versions and hosts:

Virtual Machine Requirements

When deploying AWS DataSync on-premises, make sure that the underlying hardware where you deploy the DataSync VM can dedicate the following minimum resources:

Network Requirements

Network Requirements to Connect to your Self-Managed Storage

Deploy the DataSync agent close to the source file system to minimize the distance traveled using native protocols such as Network File System (NFS). Doing this ensures that files travel over the network using our purpose-built, accelerated protocol. It keeps the distance traveled by using native, slower protocols to a minimum, and significantly speeds up transfers.

The following ports are required for communication between the DataSync agent and your NFS, Server Message Block (SMB), or Amazon S3 API compatible storage.

Network Requirements When Using Public Service Endpoints or FIPS Endpoints

Your agent VM requires access to the following endpoints to communicate with AWS when using public service endpoints, or when using FIPS endpoints. Enabling this access is not necessary when using DataSync with VPC endpoints.

If you use a firewall or router to filter or limit network traffic, configure your firewall or router to allow these service endpoints. They're required to enable outbound communication between your network and AWS.

activation.datasync.$region.amazonaws.com

activation.datasync-fips.$region.amazonaws.com

datasync.$region.amazonaws.com

$taskId.datasync-dp.$region.amazonaws.com cp.datasync.$region.amazonaws.com

cp.datasync-fips.$region.amazonaws.com

repo.$region.amazonaws.com repo.default.amazonaws.com packages.$region.amazonaws.com

54.201.223.107

0.amazon.pool.ntp.org
1.amazon.pool.ntp.org
2.amazon.pool.ntp.org
3.amazon.pool.ntp.org 
                                

The following illustration provides an overview of the ports required by DataSync:

Gettings Setup 

Download the DataSync Agent OVA template

As a first step, download the VMware OVA for the AWS DataSync Agent:

Download: AWS DataSync Agent VMWare

Deploy the DataSync Agent (VMware ESXi Hypervisor)

Select "Deploy OVF Template" to deploy the image on the host: 

Select "Local file" and browse to the image you downloaded previously.

Select a name for the virtual machine:

Select the compute resource that should be used for the template:

Review details and ensure that all is correct: 

Select the storage volume that should be used by the VM: 

Select the networks that should be used by the VM:

Ensure that all information is correct, and select "Finish" to deploy template: 

Confirm successful import (using Recent Tasks): 

Complete Network Configuration

Start the VM, and afterwards login to the DataSync console:

Default credentials: 

login: admin
password: password

Select "Network Configuration" to configure the IP address of the DataSync Agent

Test successful network connectivity by selecting "Test Network Connectivity":

Final Steps

Congratulations! You've successfully deployed the DataSync Agent. 🎉

To connect this virtual appliance to Nuclei infrastructure, we will need the activation key. This should be option 0 from the main menu.

Be aware that there is a 30 minute timer for the activation key so we need to make sure the handover is coordinated.

As a final step, you will need to share the path and credentials to your network share with Nuclei. This is the share in your environment where the recordings have been exported to.

Once provided, Nuclei will complete the configuration of the appliance to facilitate transfer of data from your local network share to Nuclei's Amazon S3 resources:

Need Help? 

We're here! Please contact support@nuclei.ai if you find you need help deploying the AWS DataSync Agent. 

Many thanks!! ✌️