Success Story for Manufacturing Client

  • by

Why Collaborate with Cuelebre

  • For Client, onboarding the data from different source systems, building
    data pipelines, deploying and orchestrating data pipelines is one of
    key areas in Data Science platform which helps globally to predicting
    sales forecasts and revenue projections.
  • We provided team of 5 members to support both development of ETL
    jobs and handling more than 200+ pipeline in productions and
    addressing stakeholders requests.

Challenges/Hidden Anomalies

  • Currently the DevOps team is manually monitoring the pipelines but there is
  • no visibility to know whether the pipeline is extracting and loading the data or
  • not. Many times the jobs run successfully but it fails to load data into the Data
  • Lake. So quality and puzzle is the biggest concern which we need to address.

Problems like:

  • Lack of data quality.
  • Required more manual time for monitoring and reporting.
  • Every day the DevOps team is manually updating the pipelines status
    to daily dashboard sheet.

Repeating data quality concerns from stakeholders which were affecting
the business.

Domain Handling Walkthrough

  • Cuelebre suggested and developed a Data Quality framework which is
    used to track the quality of the data being loaded into the Data Lake.
  • Data Quality framework is written in a generic form where it can do
    quality check for all the pipeline available in the lake.

Optimized Solutions/Results/Client Satisfaction

  • High assurance of data loaded into the Data Lake.
  • Reduced the pipelines monitoring time from 20h per week to 3h per
    week which saved huge manual effort and increased productivity on
    other areas.

Data Quality metrics for each pipeline is brought into visualization
which gave clear visibility to DevOps & stakeholders about the pipeline
and the quality of data being loaded.