Learn Apache Spark with Free Lessons & Tips

Hive tables are very important when it comes to Hadoop and Spark as both can integrate and process the tables in Hive.

Let's see how we can create a hive table that internally stores the records in it in a parquet fashion.

Storing a hive table as a parquet file with a snappy compression in a traditional hive shell

1. Create a hive table called transaction and load it with records using the load command.

 create table transaction(no int,tdate string,userno int,amt int,pro string,city string,pay string) row format delimited fields terminated by ',';

load data local inpath '/home/cloudera/online/hive/transactions' into table transaction;

2. Create another hive table named tran_snappy with storage type as parquet and compression technique as snappy.

 create table tran_snappy(no int,tdate string,userno int,amt int,pro string,city string,pay string)  stored as parquet tblproperties('parquet.compression' = 'SNAPPY');

3. Insert the second table with records from the first table.

insert into table tran_snappy select * from transaction;

4. Go to the /user/hive/warehouse directory to check whether the file is in snappy format or not.

Storing a hive table as a parquet file with a snappy compression in spark sql

1.Import the hive context in the spark shell and create and load the hive table in a parquet format.

Import org.apache.spark.sql.hive.HiveContext

Val sqlContext = new HiveContext(sc)

Scala> sqlContext.sql(“create table transaction(no int,tdate string,userno int,amt int,pro string,city string,pay string) row format delimited fields terminated by ','

”)

2.Load the created table

Scala>sqlContext.sql(“load data local inpath '/home/cloudera/online/hive/transactions' into table transaction”)

3.Create a snappy compressed parquet table

Scala>sqlContext.sql(“create table tran_snappy(no int,tdate string,userno int,amt int,pro string,city string,pay string)  stored as parquet tblproperties('parquet.compression' = 'SNAPPY')”)

4.Load the table from the table gets created in the step 1.

Scala>val records_tran=sqlContext.sql(“select * from transaction”)

Scala>records_tran.insertInto(“tran_snappy”)

Now the records are inserted into the snappy compressed hive table. Go to the /user/hive/warehouse directory to check whether the parquet file gets generated for the corresponding table.

Apache Spark is the most popular open source product today to work with Big Data. More and more Big Data developers are using Spark to generate solutions for Big Data problems. It is the de-facto standard tool today. But are there any tools/products which can claim as a close competitor to Apache Spark? Putting the question in another way - If I am given a choice, can I as a Big Data Architect can think beyond using Apache Spark as a tool which I can use for all my Big Data tasks?

I would like to analyse this question taking different use cases.

Firstly, the data which is to be considered. In this case, the data scientist gets data from some source. The data scientist or the user gets the data from somewhere, understands the data, cleans it up, correlate the data with other sources.The size of the data determines a lot here. If the data is few gigabytes (GBs), we have the option of choosing between R, MySQL, SQL Lite or a python notebook with Pandas. Spark is more useful when data is too large to process. Apache Spark is best for huge data, AWS Athena or Google BigQuery can be good competitors for Spark, but Spark has more enriched features. In such case, Spark steals over other competitors.

Secondly, for Data Visualization and creating Dashboards that provide monitoring and insights based on data streams. Here Spark does not come up to that level for this use case. BI tools like Tableau and SiSense provide much better support than Spark for streaming data within a certain range of the data set which is being used.

Thirdly as an ETL tool Spark works well especially when the data does scale up pretty high. But the user has to do a lot of work around Spark to make sure that everything is working smoothly. This usually means that when Spark is used for ETL, data is considerably delayed by several hours or even a day. Apache Flink and Spark streaming are two other alternatives for this use case, but the user needs to code a lot and manage the cluster.

Fourth and last when talking about Machine Learning as a use case to determine other alternatives for Apache Spark, we can analyse the entire process into the following steps-
1. Preparing your data set
2. Building your models and
3. Using your models in a production environment.

Spark is considered very good for the first two jobs - preparing the data sets and building the models. Apache Spark scores high over other tools on data discovery and manipulating the data. Spark has rich Machine learning libraries for building models. However key-value data store like Cassandra also required here which increases the complexity of the solution and running these data models in production for real-time predictions gives Spark the bumps and the process usually falls apart. Few alternatives to Spark for this particular use case are Google's Tensorflow and ScikitLearn.

1. Introduction to Apache Spark:

It is a framework for performing general data analytics on distributed computing cluster like Hadoop.It provides in memory computations for increase speed and data process over mapreduce.It runs on top of existing hadoop cluster and access hadoop data store (HDFS), can also process structured data in Hive and Streaming data from HDFS, Flume, Kafka, Twitter.

2. Is Apache Spark going to replace Hadoop?

Hadoop is parallel data processing framework that has traditionally been used to run map/reduce jobs. These are long running jobs that take minutes or hours to complete. Spark has designed to run on top of Hadoop and it is an alternative to the traditional batch map/reduce model that can be used for real-time stream data processing and fast interactive queries that finish within seconds. So, Hadoop supports both traditional map/reduce and Spark.

We should look at Hadoop as a general purpose Framework that supports multiple models and we should look at Spark as an alternative to Hadoop MapReduce rather than a replacement to Hadoop.

3. Hadoop MapReduce vs. Spark: Which One to Choose?

Spark uses more RAM instead of network and disk I/O its relatively fast as compared to hadoop. But as it uses large RAM it needs a dedicated high end physical machine for producing effective results.

It all depends and the variables on which this decision depends keep on changing dynamically with time.

Internet of Things (IoT) is one of the booming topics these days among the software techies and the netizens, and is considered as the next big thing after Mobility, Cloud and Big Data.
Are you really aware of what actually Internet of Things is all about, and how it can transform your normal boring house into a smart home? If not, then you are at the right place. In this article, we are going to explain how the Internet of Things is going to digitalise our lives in the coming days.

What is IoT?

Internet of Things is a simple network of interconnected things that are linked with the sensors and internet-enabled devices, which allows them to gather and share the data responsively.  

This means IoT extends the internet connectivity beyond the regular devices like Computers, Smartphones and Tablets to a wide spectrum of electronic components and daily things, which will use the embedded technology via Internet, for interacting and communication with the external environment.

What IoT do?

Internet of Things makes it easier to manage our daily life aspects, without getting tensed or worried about anything. The power of inter-connectivity minimizes the chances for glitches and makes the smart devices to work for us. Just imagine how cool it would be, the moment when you wake up in the morning and the smart coffee machine prepares a bed coffee for you. It will definitely be going to make your day awesome.

Is IoT the future?

IoT is existing; there are already several IoT applications that are available in the market, which most of the people are already using them in their daily lives. In the coming days, the number will gradually increase as more people are likely to buy IoT products.

According to Gartner, 4.9 billion things were inter-connected in 2015, and the number is expected to reach 20.8 billion by 2020.

Source: Mario Morales, IDC

Switch to Smart Home with IoT:-
With IoT, you can transform your simple houses into the smart homes, where you can get the feasibility to control everything by using your Smartphone or a Tablet.

“A house is a machine for living in”

–  Le Corbusier

Le Corbusier around 80 years ago has referred a house as a machine designed for humans to stay and carry out their daily life aspects. From then on-wards, a number of attempts have been made by different techno-brains to transform this vision into reality. With the emergence of IoT, this vision of Le Corbusier finally seems to be fulfilling on a rapid pace.We are now already talking to our Smartphones (via personal assistants like Siri, Cortana etc), and in the coming days we will be doing the same with all the other internet-enabled devices that are present inside our smart homes. Smart homes with inter-connected devices are loaded with wide possibilities that makes our lives easier and more comfortable.

Let us imagine that you are returning back to the home from office, and you are in the need of an instant shower. Rather than turning the geyser on when you reach home and wait for the water to get heated up, you can simply use your smartphone while leaving the office, and tell the smart thermostat to heat the water up to a specified temperature.
No shortage of possibilities exists with IoT and analytics, because in future, ‘anything that can be connected will be connected’.

1.    Operate Remotely
With IoT, you can now have everything hooked up with the Internet, which allows you to operate any appliance from any part of the world – whether you want to turn on the bedroom AC or just need to switch off the bathroom light, anything and everything can be done on your fingertips.

2.      Intensified Home Security
Different devices are making the security aspects smarter for the people to always keep an eye on their homes. Whenever the security cameras detect any unexpected motion inside or outside your home, it will give you an instant alert message.

3.    Re-design Every Occasion
What will be more fabulous than the home lightening that changes with your personal tastes and occasions. With the inter-connectivity of bulbs and lights, you can switch between the various florescent colors, and can also adjust the brightness levels depending upon your current moods.

4.    Personalized Health
We are already using some smart wearable like health bands and wrist bands that monitors our heartbeat, pulse rate, blood pressure, sugar levels etc.  With IoT, your smart refrigerator now holds the ability to self-order for milk, vegetables and other food items when you start running low.

Conclusion:-

There is no doubt to say that, Internet of Things is the next great thing for us, and our lives are going to be greatly improved with it. Not only for house automation, IoT holds charge on the energy sectors, healthcare, manufacturing, transportation sectors and a lot more. However, the point that is to be noticed is the ‘smarter’ the devices inside our homes become, the higher the scope for misusing them will also prevail. Because, always being connected with the things around us will narrow the possibility of privacy and secrecy. Moreover, the chances for scams, frauds and cyber-attacks will also escalate, and also just think about what the advertisers are going to do.

In order to lay smart restrictions on these pitfalls, the big giants like Amazon, AT&T, Belkin, Bosch, GE, Cisco etc. have already started hunting for the talented and skillful IoT developers, who can smartly connect them. In order to meet these market requirements, Kovid Academy is offering the IoT online training and classroom training for the aspirants who holds an aggressive passion towards internet working technology. Come and learn ‘Internet of Things’ from Kovid, and start connecting your smartphone with the electronic devices around you.

For any business, the key success factor relies on its ability for finding the relevant information at the right time. In this digital world, it has become further crucial for the retailers to be aware of what decisions should be made, and how they will impact their financial and operational performances. The growing demand for these insights has kindled the growth of Big Data in retail industry. Today, the retailers are considering the Big Data and Analytics as the game changing path for their businesses in attaining a high competitive market advantage.

According to a survey by Gartner, it was predicted that the use of Big Data in retail industry will grab the $4.5 billion mark by 2019. The results have further informed that more than 500 businesses have already invested in Big Data and a lot more are expected to invest by the end of 2016.

“The future is going to be so personalized, you’ll know the customer as well as they know themselves” – Tom Ebling (President & CEO of Demandware Inc.)

In any retail landscape, Big Data can be effectively used for enhancing the supply chain figures, by accurately estimating the market demand, and pulling out large profits along with high customer satisfaction. The retailers with the key goal of reducing the operational costs and enhancing the profit margins are looking at the Big Data as a massive tool for vivid transformation possibilities.

For example, a leading fast-food retail chain can start analysing the data from social media for spotting the upcoming events in the local market. This will accordingly assist the retailer to be fully-prepared of meeting the demand with the required food supplies.
Big Data analytics offers the retailers to:

•    Deliver a smarter shopping experience to the customers.
•    Develop patterns for optimizing business operations.
•    Build supply chain decisions and product assortments.

With Big Data, the retailers get the ability to build smarter insights of their customer’s buying behaviours and also to understand the essence of 5R’s in the retail sector.

The Big Data analytics allows the retailers to interact with the right person (customer) about the right product using the right media with the right message at the right moment. However, deciding which of the customers are likely in the need of a specific product, and the best practice to go putting ahead is the key aspect here. The information gathered through the transactional records makes it easier for the retailers to forecast the products demand based on the individual geographic areas.

This digital revolution is serving the retailers with surplus opportunities for enhancing their overall customer experiences. Using the Big Data Analytics, the retailers are able to interpret and collate the vast amounts of data for extracting useful and meaningful information, which is delivering utmost value to the customers and enhanced profits to the retailers. This sort of forward-thinking among the retailers is helping them to boost in the areas like:

Concentrating on value

Retailers via Big Data are able to largely concentrate on the value aspects, rather than on the price factors. One best example that perfectly matches this criterion is the leading online retailer ‘Amazon’; it follows the smart effective dynamic pricing principle, which allows the Amazon to quickly change the prices of some of its products up to 10 times in a single day. This approach has largely helped Amazon in building a strong market reputation for delivering the best value and experience to its customers.

Customising the interactions

Based on the customer’s interactions (user’s browsing history), the retailers via Big Data Analysis are offering a list of recommendations of the similar products or additional accessories, which most of the customers has already purchased earlier. The best example can be stated is the Amazon’s recommendation feature ‘Customers who bought this product also bought …’ that uses the Big Data Analysis for seizing the customers on its web site. This recommendation feature has nearly increased the sales of Amazon by 30%.

Enhancing the customer experiences

With the rise in usage of e-commerce activities, customers are enjoying the feasibility of purchasing ‘anytime & anywhere’ across the globe. However, the chances for the cheating and fraudulent activities are also gaining a large scope. This can be barred by using the Big Data Analytics, which allows the retailers to gather the information shared by the customers while making their purchases, in determining the fraudulent transactions in the real-time.

Predicting the trends

Today, retailers are using their own resources and other social media platforms for gathering the data on customer’s purchasing history and preferences. This information not only helps the retailers in offering a tremendous shopping experience, but also helps them in analysing what to keep ready in their stock. Because, the present competitive market needs efficient patterns for estimating the future demands.

Conclusion

From the last couple of years, it has become quite apparent that Big Data program is rapidly transforming the way the retailers sell their products. However, to utilize the Big Data analytics to the fullest, the retailers are in need of skilled manpower who bound to be the experts in using the Big Data and Data Analytical tools. These factors are deriving a good career prospects for the Big Data aspirants and other software professionals, who are passionate of employing themselves with the canopy of Big Data Analytics. Accordingly, to fill these market gaps, Kovid Academy is offering Big Data Online and Classroom training programmes for transforming your skills to the next levels of expertise.