AWS Workflow: S3 to EMR

Hello Sir, Greeting for the day, Hope you are doing well We are Group of professional having more than 8 years’ experience in Windows/Linux Server administration , Website Hosting (Cpanel/Plesk) migration, Mail server configuration management ,CISCO Router Switches, Firewall(ASA,PIX,Sonicwall, cyberoam) ,Cloud Computing(AWS/Azure,office365), We provide support for small and medium companies to build their successful online businesses utilizing different platforms like Microsoft (Asp.net, SQL Server, C#, VB.net), LAMP, CMS like DNN, CATALLOK, Magneto, Word Press. We provide solutions focused web professional with comprehensive background orchestrating the design and development of cutting-edge web sites and user experiences that strategically leverage and maximize online presence. Please initiate chat to discuss your requirement in details. Thanks karun

We can automate the entire workflow. Trigger new EMR, the Bootstrap step does the environment setup, do the required work using Steps: these steps copy the data from S3 to HDFS, Mapreduces the data using Hive (or Spark), uses Hive tables to store intermediate results and at the end copies the reduced results back to HDFS and then to S3. Have worked on complex data pipelines that involved Spark Streaming, Kinesis, Flume, Kafka, and S3. Let me know if we could discuss this today. I'm relatively free for next3 days. My Skype handle: kiranbm Good day!

My team includes experienced sysadmins & developers who manage Cloud infrastructure(on AWS, Google Cloud, Rackspace) for various clients across globe for last 9 years. Other programming skills like ROR, Python, AngularJS, PHP. etc and automation tools used like Jenkins, chef etc. and Databases handled like Mysql, Postgresql, mongodb etc. I have understood the requirement. Please share the furtherdetails and will get started immediately. Feel free to contact me for further discussion.

A proposal has not yet been provided

Hi, We are growing IT Company in India Having expertise in amazon web services. We can setup aws ec2, aws s3 and other amazon products with best configuration. We have expert knowledge of as below techniques:- 1. Amazon Elastic computer cloud(EC2) 2. Amazon Simple storage services (Amozan S3) 3. Amazon Cloud front 4. Amazon Relational Database Services(RDS) 5. Amazon Simple DB 6. Amazon Simple Notification Services (SNS) 7. Simple Queue Services (SQS) 8. Virtual Private Cloud (VPC) 9. EBS 10. Route53 11. ELB 12. Auto Scaling 13. DNS Failover 14. Load Balancing Thanks.

Firstly, I have used AWS SWF to do exactly this kind of operation. A simple description about me, I am an expert AWS Big Data engineer with solid experience on Map Reduce, HIVE, HBASE, Pig, Amazon Web Services(EC2, EMR, VPC, IAM, S3, Glacier, DynamoDB, CloudWatch, Kinesis, Data Pipeline, SQS, SNS, SES, SWF, API Gateway). I have expert level coding skills in Java, Scala and Node Js. I have worked on multiple projects with the mentioned skills and have expert level understanding of Hadoop Stack. I have experience of setting up of 100 nodes cluster on EC2 with Cloudera Enterprise edition.

A proposal has not yet been provided

A proposal has not yet been provided

A proposal has not yet been provided

Diligere is a trusted IT service who provide Managed IT services,Cloud services, Cloud Infrastructure, Software Development,CRM development and Support, etc. We strongly believe in Client satisfaction through quality of work. Customer is the King for us

Hello, I've been working with AWS for two years, however I was not focusing on EMR specifically. I think your project requirements are pretty clear with very few unknowns such as: * programming language (mind Java?) * as far as I understand, SQS can't create an EMR instance per se. You'll need to either run a service on an EC2 instance which will be polling for this message and spin off EMR cluster or use something else (e.g. AWS Lambda) to initiate EMR cluster creation. * what is the expected load/S3 events frequency? if there are too many events triggered you will run out of your EMR quota unless you maintain an EMR instance and reuse it (that is much more complex solution) * how big is the data set and how much time it will take to finish the job(s)? I hope, my question reflect the level of my knowledge and way of thinking. Let me know the details and whether you're interested in my services. P.S. I am NOT very experienced with HDFS, Hive, Spark. Your guidance/code samples will be very appreciated. Depending on how much I will reuse from existing code base, I can make a "discount". I'd rather say, the price can be negotiated during the solution architecture discussion. Thank you! Have a great day, -- Igor