Store large log files - best database suggestion and deployment

Hello, we can put it into hadoop and I have done something like that before. for more information contact me.

Hi there, My suggestion is as follow: - Upload log files to AWS S3 using AWS Kinesis Firehose - Load data from S3 to RDS or Elasticsearch(Recommended) If you need to load log data into database in real time, we can switch to use AWS Kinesis Data Stream. I would love to chat to finalize the solution for your project. Thank you, Wang

Hi This is a typical use case - the need of storing large log files onto file systems / database systems. Before I suggest any solution path, few questions - 1. what kind of log files are these ? whats the source of these log files ? are these structured - in the sense that - are these log files (lines in there) parsable ? is there a requirement to process these log files / lines before storing further ? 2. what's the end goal of keeping them in a database system ? reporting ? whats your typical retention period for this data ? depending upon your consumption use case, we should look at the storage solution. 3. What's the current technical environment at the organization ? is it primarily cloud environment ? or hybrid, or a localized - on premises env ? where do most of the organization database reside ? MS SQL Server ? Oracle ? Others ? all these questions / responses will help understand the problem landscape better. Any solution should consider these aspects before tooling can be discussed / suggested. The rate / time indicated in the bid here are indicative. thanks raghav