About the Company
Implementation of Amazon EFS for ideaForge Technology Ltd
- Client: ideaForge Technology Ltd
ideaForge is a design-focused UAV manufacturer developing drone solutions for a variety of application .
- Industry Vertical : Drone Solution and UAV Manufacturer
Galaxy Office Automation Team was tasked with implementing Amazon Elastic File System (EFS) to enhance the data storage capabilities for their client, ideaForge Technology Ltd. The primary objective was to set up a robust, scalable, and secure storage system on AWS EC2 instances that could seamlessly integrate with the client’s on-premises infrastructure using a site-to-site VPN.
Challenge
1. Data Migration: Optimizing the migration process to minimize downtime and ensure
data integrity while transferring large volumes of data (100 GB) securely and
efficiently from on-premises servers to AWS.
2. Network Integration: Configuring and managing a robust network infrastructure to
establish secure and reliable connections between on-premises data centers and AWS
infrastructure, ensuring minimal latency and maximum uptime.
3. Scalability: Designing and implementing a scalable storage architecture that can
seamlessly accommodate the expected growth of data volumes into terabytes, while
ensuring high availability and performance.
4. Performance: Maintaining high data availability and performance consistency across
distributed networks.
Optimizing the migration process to minimize downtime and ensure data integrity while transferring large volumes of data (100 GB) securely and efficiently from on-premises servers to AWS.
Configuring and managing a robust network infrastructure to
establish secure and reliable connections between on-premises data centers and AWS infrastructure, ensuring minimal latency and maximum uptime.
Designing and implementing a scalable storage architecture that can seamlessly accommodate the expected growth of data volumes into terabytes, while ensuring high availability and performance
Maintaining high data availability and performance consistency across distributed networks
Solution
The Galaxy Office Automation Team designed and implemented a solution using Amazon EFS for scalable file storage connected to EC2 instances in the AWS cloud. The steps and technologies involved included:
Configured Amazon EFS to provide a scalable file storage system. EFS was chosen for its ease of use, scalability, and performance.
Deployed several EC2 instances that would connect to the EFS for storing and retrieving data. These instances were configured to scale based on the load and data access patterns
Implemented an AWS Site-to-Site VPN to securely connect the client’s on-premises data centre to the AWS VPC. This ensured encrypted data
transfers and maintained data integrity.
Migrated 100 GB of data from the on-premises servers to the Amazon EFS using secure and reliable data transfer methods. Initial tests were
conducted to ensure data integrity and performance.
Utilized Systems Manager
Session Manager to automate and streamline the management of EC2 instances and other resources within the AWS environment.
We have Set up AWS CloudWatch alarm for
monitoring the Storage of the EFS for threshold of 80 % and EC2 instances CPU utilization set to 80 % threshold and using SNS service we will get Email notification
about this alarm.
Implemented AWS security best practices, including network access control lists, security groups, and IAM policies to ensure the data is protected against
unauthorized access.
Amazon Elastic File System (EFS)
Amazon Elastic File System (EFS) is a cloud-based file storage service provided by Amazon Web Services (AWS) that offers several significant advantages for businesses and developers. Here are some of the key reasons why EFS is important and beneficial:
EFS is designed to scale on demand to petabytes without disrupting
applications, making it ideal for workloads and applications that require large amounts of data storage. This automatic scaling eliminates the need for manual intervention in storage provisioning and management.
EFS is easy to use and can be set up in minutes. It eliminates the
complexity associated with deploying, scaling, and maintaining a distributed file system. Users can simply create an EFS file system and start using it without detailed knowledge of the underlying infrastructure
EFS offers high-performance file storage with low latencies, which is
crucial for performance-sensitive applications. It supports thousands of concurrent
NFS connections and delivers consistent performance, which is vital for applications
with high throughput needs
EFS is designed to be highly durable and available. It
automatically replicates files across multiple availability zones to prevent data loss
due to failures of individual components or an entire data centre.
The storage capacity used and the performance scale automatically with
the amount of data stored, which means you pay only for the storage you use. This can lead to cost savings compared to provisioning storage with a fixed capacity that
may not be fully utilized.
EFS allows multiple EC2 instances to access the file system
simultaneously, making it a great solution for applications and workloads that require
file storage accessible by multiple instances. This is particularly useful for SaaS
applications and content management systems.
EFS integrates well with other AWS services such as Amazon EC2 and
AWS Lambda, allowing businesses to build and deploy a wide range of applications
and services. It supports standard file system interfaces and permissions, which makes
it easy to integrate with existing applications.
EFS provides robust security features that allow users to control access to
files using POSIX permissions. It supports AWS Identity and Access Management (IAM) for managing access to the EFS API and can be used with Virtual Private
Cloud (VPC) to isolate file system network traffic.
With its pay-as-you-go model, EFS can be more cost-effective than
on-premises solutions, especially when factoring in the costs associated with
hardware maintenance, power, cooling, and administration.
EFS integrates with AWS Backup, making it easy to create and manage backups of file systems. These backups can be used for disaster recovery
purposes, ensuring that critical data can be restored quickly and reliably
Success Metrics
Amazon Elastic File System (EFS) is a cloud-based file storage service provided by Amazon Web Services (AWS) that offers several significant advantages for businesses and developers. Here are some of the key reasons why EFS is important and beneficial:
• Following the implementation of Amazon EFS, we observed a significant reduction in data access times. Previously, accessing approximately 10-15 GB of data took 15-20 minutes, but after transitioning to EFS, this time was reduced to just 2-3 minutes.
• Initially, SATA Hard Disk Drives (HDDs) typically deliver an IOPS range of about 80-150. In contrast, Amazon Elastic File System (EFS), even in its General-Purpose mode, provides substantially higher IOPS. For instance, with 100 GB of stored data, EFS can deliver around 5,000 IOPS. This performance can increase significantly when configured in Max I/O mode and as more data is stored,
• No Upfront Costs: With EFS, there are no upfront costs or investments required for purchasing hardware or provisioning storage infrastructure. This eliminates the need for significant upfront CapEx expenditures typically associated with building and managing on-premises storage solutions.
• Managed Service: Amazon EFS drives operational expenditure (OpEx) savings by offering a fully managed storage solution, sparing the need for dedicated staff, hardware upkeep, and ongoing management. With its automated scalability and seamless integration with AWS services, EFS streamlines operations, allowing organizations to focus on core activities while ensuring cost-effectiveness in their storage infrastructure.
Comparison of storage costs before and after implementing EFS, considering scalability and pay-as-you-go features.
Reduction in operational costs related to maintenance of on-premises storage solutions.
On-Premises Storage Costs:
• Initial Setup Costs: $500 to $800
• Ongoing Yearly Operational Costs: $100 to $300
Total cost for the first year (assuming the minimum setup cost):
• Initial Setup: $500
• Yearly Operational: $100
• Total for 1 year: $500 + $100 = $600
AWS EFS Costs:
• Monthly Cost: $10
• Yearly Cost: $10 x 12 = $120
Cost Comparison
1. Minimum On-Premises Cost for 1 Year: $600
2. AWS EFS Cost for 1 Year: $120
Percentage Comparison
• AWS EFS is significantly cheaper than on-premises storage, costing only about 10.91% to 20% of the on-premises setup, depending on the initial investment and
operational costs
1. Ability to scale storage capacity up or down easily without significant
downtime or manual intervention.
2. Number of workloads or applications successfully migrated to EFS without
disruption.
How Galaxy Successfully Solved the Company’s Challenges
• Data Volume: Initial audits showed 20 TB of data, predominantly consisting of large media files and application data.
• Performance Requirements: The existing system experienced access delays (15-20 minutes for large data batches), which needed significant improvement.
• Cost Structure: Ongoing maintenance and hardware costs were escalating, necessitating a more cost-efficient solution.
• EFS Configuration: Chose the General Purpose performance mode and the bursting throughput mode to optimize for the company’s workload, which involves frequent access to media files.
• Network Design: Established a secure AWS Site-to-Site VPN connection between the on-premises data center and AWS to ensure secure data transfer.
• VPN Setup: Configured the AWS Site-to-Site VPN for a secure and reliable connection to facilitate the data transfer.
• Secure Access: Use Systems Manager Session Manager for secure, encrypted shell access to EC2 instances, eliminating the need for managing SSH keys.
• Data Integrity Checks: Conducted comprehensive tests to ensure data integrity and completeness post-migration.
• The implementation of Amazon EFS for ideaForge Technology Ltd resulted in several immediate benefits:
• Scalability: The solution proved to be highly scalable, handling the initial load of 100 GB efficiently with provisions to scale up to several terabytes in the future.
• Performance: There was a noticeable improvement in data access speeds and reliability, facilitating smoother operations for the client.
• Security: Enhanced security measures ensured that all data transferred remained secure across both on-premises and cloud environments.
• Cost-Effectiveness: By using Amazon EFS, the client could leverage a pay-as-you-go model, saving on upfront capital expenditures while benefiting from AWS’s scalable
infrastructure.
Future Plans
The Galaxy Office Automation Team has laid down a scalable foundation that can efficiently handle an increase in data storage needs. In the future, the data stored on Amazon EFS is expected to grow into terabytes, and the infrastructure is designed to accommodate this growth seamlessly. Further integrations and optimizations are planned as the data and access patterns evolve.
We have successfully migrated important production data from on-premises storage to Amazon EFS. We plan to migrate the remaining data, approximately 4 to 5 TB, in the next few months.
The successful implementation of Amazon EFS for ideaForge Technology Ltd by the Galaxy Office Automation Team showcased migration to EFS, the organization streamlined data management across multiple AWS services, achieving an elasticity that allowed storage to automatically adjust from 100 GB to 500 GB based on user demand. This integration facilitated a 40% reduction in latency and a 20% cost savings compared to their previous on-premises solution.
“Galaxy Office Automation’s expertise in AWS and their efficient migration process made our transition to EBS volumes seamless and painless. EBS has transformed our storage infrastructure, providing us with the performance, scalability, and cost-efficiency we need to support our continued growth”
IT Manager
EXFO India Pvt Ltd