NashTech

Enable Huawei to implement different functionalities and integration support with presto and hive in CarbonData

Enable Huawei to implement different functionalities and integration support with presto and hive in CarbonData

Introduction

NashTech worked with Huawei to develop a file-format which is faster and efficient in processing and querying on big data. Now, Huawei clients able to speed up their system by utilising the features of CarbonData.

About Huawei

Apache CarbonData is an indexed columnar data format for fast analytics on big data platform, e.g. Apache Hadoop, Apache Spark, etc. NashTech enable Huawei to work in collaboration with them to implement different functionalities or integration support with different technologies including presto and hive in CarbonData.

The challenge

Huawei wants to explore a domain where backend, frontend, and continuous integration ensure backward compatibility of the older versions when the newer versions will be rolled out frequently. NashTech worked along with Huawei Team to help CarbonData in becoming an Apache-licensed project from an incubating project.

knoldus-carbon-diagram-1

The solution

NashTech worked closely with the Huawei team and helped in building the crucial functionalities, some of which are listed below:

  • Development of Dictionary Generation Tool for CarbonData.
  • Improved cost efficiencies: Automated cluster management reduced their operational costs by more than 50%.
  • CarbonData integration with Presto, Hive, Flink, and S3 technologies.
  • Setting up of continuous Integration via Jenkins.
  • Creation of Performance Testing tool to do benchmarking
  • Achieving zero bugs with Automation Testing.
  • Development of Apache CarbonData website and its maintenance.
  • Development and enhancement in core packages of CarbonData.
  • Benchmarking CarbonData against available file formats like Parquet and ORC, against frameworks like Spark, Presto, and Impala and against different storage systems like Hadoop, S3 and Kudu.

NashTech worked with Huawei to develop a file-format which is faster and efficient in processing and querying on big data. Now, Huawei clients able to speed up their system by utilising the features of CarbonData.

Our team also developed a proprietary performance benchmarking tool for CarbonData. This benchmark tool tests the performance of the CarbonData in comparison with its competitors like Parquet and ORC Format. The key functionality supported by the Benchmark tool are as follows:

  • Generating the TPCH benchmarking data depending on the cluster size driven by configuration.
  • Defining workloads as a configuration for particular datasets.
  • Loading the data into all the formats into the Hive Store like CarbonData, Parquet, and ORC.
  • Configuration based Tuning for Spark that included parallelism settings as well as spark configuration based on different workloads.
  • Executing the workloads and capturing the response time and results with respect to load for all the formats.
  • Comparison of the results in all the formats.
  • Generating an Excel report showing the comparison of the results as well as success and failures of test execution.

The outcome 

With the rapid development and concise code offered by Scala, NashTech could get the system into production in 4 months. The alerts are routed to different buckets based on rules defined and reach the consumers’ mailbox in a matter of seconds as soon as the news is broken. The product is being heavily used as a part of the infrastructure.

Read more case studies

Modernising legacy systems and driving efficiencies through partnership with RPS

Explore how NashTech help RPS modernise legacy systems and drive efficiencies through partnership

Supporting digital shelf analytics and unlocking eCommerce growth

Explore how NashTech help the digital shelf analytics and unlock growth with a world leading data insights and eCommerce solutions provider.

How THE OUT is disrupting the premium car rental industry

By working closely and collaboratively with the NashTech development team in Vietnam, they were able to build a high quality, digital first, luxury rental car service. Looking ahead into the future,...

GCP
GCP

Let's talk about your project

Our partnerships

Scroll to Top
PANEL DISCUSSION
Join the conversation and gain invaluable insights!
How Enterprises Can Leverage Cloud for Business Growth?
FREE WHITEPAPER
Unlock the power of knowledge with our new whitepaper
“Elevating User Experience for Product Owners”