0Apache Spark is the most popular open source product today to work with Big Data. More and more Big Data developers are using Spark to generate solutions for Big Data problems. It is the de-facto standard tool today. But are there any tools/products which can claim as a close competitor to Apache Spark? Putting the question in another way - If I am given a choice, can I as a Big Data Architect can think beyond using Apache Spark as a tool which I can use for all my Big Data tasks?
I would like to analyse this question taking different use cases.
Firstly, the data which is to be considered. In this case, the data scientist gets data from some source. The data scientist or the user gets the data from somewhere, understands the data, cleans it up, correlate the data with other sources.The size of the data determines a lot here. If the data is few gigabytes (GBs), we have the option of choosing between R, MySQL, SQL Lite or a python notebook with Pandas. Spark is more useful when data is too large to process. Apache Spark is best for huge data, AWS Athena or Google BigQuery can be good competitors for Spark, but Spark has more enriched features. In such case, Spark steals over other competitors.
Secondly, for Data Visualization and creating Dashboards that provide monitoring and insights based on data streams. Here Spark does not come up to that level for this use case. BI tools like Tableau and SiSense provide much better support than Spark for streaming data within a certain range of the data set which is being used.
Thirdly as an ETL tool Spark works well especially when the data does scale up pretty high. But the user has to do a lot of work around Spark to make sure that everything is working smoothly. This usually means that when Spark is used for ETL, data is considerably delayed by several hours or even a day. Apache Flink and Spark streaming are two other alternatives for this use case, but the user needs to code a lot and manage the cluster.
Fourth and last when talking about Machine Learning as a use case to determine other alternatives for Apache Spark, we can analyse the entire process into the following steps-
1. Preparing your data set
2. Building your models and
3. Using your models in a production environment.
Spark is considered very good for the first two jobs - preparing the data sets and building the models. Apache Spark scores high over other tools on data discovery and manipulating the data. Spark has rich Machine learning libraries for building models. However key-value data store like Cassandra also required here which increases the complexity of the solution and running these data models in production for real-time predictions gives Spark the bumps and the process usually falls apart. Few alternatives to Spark for this particular use case are Google's Tensorflow and ScikitLearn.
