Write some Software

Bigdata solution steps for implementation. Explained: The main idea is to have all messages coming from the positioning module published to Kafka topics on the local server. Kafka connect will stream the messages to Hadoop (HDFS) running on a cloud server. MapReduce jobs will run on the data in the HDFS and create new files for very long period analytics. Spark will consume messages from one of the topics on the local server in real-time, apply calculations and save results on a MongoDB database. Every 4 hours, those analytics will be saved on the cloud for backup. Technologies involved: Kafka, Hadoop (HDFS + MapReduce), JAVA, MongoDB, Spark, Kafka Connect, Zookeeper. Steps involved: 1- Installing and running zookeeper Zookeeper will run on the local server. Kafka depends on zookeeper to run. 2- Installing and running Kafka. Kafka will be installed on the local server and run on zookeeper All messages that comes from the positioning system will be sent to a Kafka topic. A message would contain the clamp id, the timestamp, models of items in the clamp, the speed of the clamp and the location coordinates. 3- Installing and running Hadoop. Hadoop will be installed on the cloud server. It will hold all data received from the positioning module and it will keep all data since day 1. 4- Configuration and running a Kafka broker. Kafka brokers will need to run on the zookeeper server. 5- Single Node-Single Broker Configuration. These configurations will be made on the local server 6- Creating 2 topics: 2 topics will need to be created. -first topic to publish all data from the positioning system that must be sent to Hadoop on the cloud server for archive and long periods analytics -second topic to publish all data from the positioning system to Spark for real time analytics 7- Writing and running a Java Producer class: Producer classes will be needed on the side of the positioning system to send all readings to a Kafka topic. The reading will contain timestamp, clamp id, models in the clamp, speed of the clamp, and the clamp s real location 8- Installation and Use Kafka Connect: Kafka connect will send data from a Topic to a HDFS volume on cloud. The data will need to be saved into different folders with daily date as name. in each folder, we will need to have files representing each table. The file architecture will need to allow JOIN queries to be performed. 9- Developing java MapReduce classes: Map reduce classes will need to run on the cloud server. The will take input files from the cloud server in each period and they will produce useful data files that will be used for long term analytics. Those data files will need as well to be kept on the HDFS volume for reuse. They can contain frequencies, averages, counts …etc. the output in those files will need to be well classified to facilitate an easy visualisation 10- Sparks and MongoDB For real time analytics, alerts, errors and logs monitoring; Sparks and Mongo DB will need to be installed on the local server. This will allow real-time analytics. Spark will get data from the Kafka topic and after analytics, result will be saved on MongoDB. Every 4 hours, data in from the MongoDB server will need to be sent to the cloud for archive.