Establishing the connection between On-premises Oracle DB to AWS VPC
Requirement1: (AWS +EMR)
1. Setting up the AWS EMR cluster using VPC and once the EMR cluster is up.
2. Connecting to Hadoop Master node in AWS using Putty.
3. Run the query sqoop to list the tables in the on premises oracle DB.
4. Details will be provided the jdbc connection string, IP, port no, user id and password in the sqoop query.
5. Import the one database table from Oracle DB which is in Dubai HO and one Branch oracle DB.
Observations:
[login to view URL] are getting the error and did not connect to the oracle DB.
Note: The oracle JDBC driver has placed manually in the /usr/lib/sqoop/lib directory.
Error:
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/12/03 10:34:00 INFO [login to view URL]: Running Sqoop version: 1.4.6
17/12/03 10:34:01 WARN [login to view URL]: Setting your password on the command-line is insecure. Consider using -P instead.
17/12/03 10:34:01 INFO [login to view URL]: Data Connector for Oracle and Hadoop is disabled.
17/12/03 10:34:01 INFO [login to view URL]: Using default fetchSize of 1000
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/[login to view URL]!/org/slf4j/impl/[login to view URL]]
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/[login to view URL]!/org/slf4j/impl/[login to view URL]]
SLF4J: See [login to view URL] for an explanation.
SLF4J: Actual binding is of type [[login to view URL]]
17/12/03 10:35:02 ERROR [login to view URL]: Got exception running Sqoop: [login to view URL]
[login to view URL]
at [login to view URL]([login to view URL])
at [login to view URL]([login to view URL])
at [login to view URL]([login to view URL])
at [login to view URL]([login to view URL])
at [login to view URL]([login to view URL])
at [login to view URL]([login to view URL])
at [login to view URL]([login to view URL])
at [login to view URL]([login to view URL])
Requirement2: (Qubole ‘s QDS Business edition)
[login to view URL] to create a Bastion Node On-premises/AWS and white list the IP address mentioned by the Qubole ‘s QDS which is required as part of importing the on-premises Oracle DB to AWS S3.
* Please ensure that the following ip(s) '[login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL], [login to view URL]' have access privileges to your database.
- If your database is in a virtual network please check the bastion node flag and enter the bastion node details, this will enable us to connect to this database.
2. Create a datastore using Qubole’s QDS and see that able to create active datastore by using the details like ip, user id and password.
3. After creating the active datastore we are able to see the tables in Oracle DB to import the selected table to AWS S3 using Qubole’s import panel.
Hello,
We can help you to fix the issue with Oracle DB and make all your requirements working properly.
We are a team of server administrators having many years of experience in the field of Linux/Windows administration, Cloud Management Services etc.
Kindly accept our proposal and we can have a detailed discussion through chat. Hope to hear from you soon.
Thank you
I have worked on similar project where we used to import our Dimensional data stored in Oracle DB to Amazon S3 using Sqoop.
I have extensively worked on AWS with posses good knowledge of AWS VPC, EMR, S3.
I have good knowledge of handling networking in Amazon VPC ( setting up bastion host in public subnet , using NAT gateway)