Get in Touch

Azure Data Lake

Background

Microsoft Azure Data Lake is a secure cloud platform hosted in Azure that enables big data analytics and artificial intelligence (AI). ‘Data Lake’ specifically refers to a storage technology that is based on Azure Data Lake Storage Gen2.

With the volume of data increasing day by day, enterprises are looking for suitable digital infrastructure to store and apply data analytics. Providing unlimited storage space for largeamounts of data, including structured, semi-structured and unstructured data, Data Lake is the solution to all your storage needs.

Besides storage, Azure Data Lake also enables data scientists, marketers, and analysts toanalyze and understand data effectively. The backbone of this platform features the Hadoop file system (including HDFS and YARN), which provides massive petabyte-sized file computing.

Azure Data Lake consists of the following components:

  • Azure Data Lake Store: A storage system that stores structured, set-structured, and unstructured data.

  • Azure Data Lake Analytics: Analytical workspaces that provide contextual and actionable insights on data and business processes.

  • Azure HDInsight: A full-stack Hadoop platform & Azure service that supports a wide range of analytic engines, including Spark, Storm, and HBase.

Azure Data Lake Store

Azure Data Lake Storage (ADLS), a Hadoop-based repository, sits directly on top of Blob Storage, which means, files that are stored in Blob Storage are simultaneously available through ADLS. ADLS also enables all key Blob Storage functionality, including Azure AD-based permissions, encryption at rest, data tiering and lifecycle policies. ADLS provides a Hadoop-compatible file system (HDFS), hierarchical namespace (folders/metadata), and high-performance access to large volumes of data required for data analytics.

Azure Data Analytics

Azure Data Lake Analytics is a service built on Apache YARN. The analytics service can handle jobs of any scale instantly, instead of having to go through the processes of deploying, configuring and tuning hardware. Data Lake supports a range of tools and programming languages (SQL-like queries, and REST calls) that enable large amounts of data to be queried, transformed and reported.

Common Data Model Folders

Data stored in Data Lake complies with the Common Data Model folder standard. Common Data Model folders contain metadata definitions in addition to data files. Metadata definitions are kept in model files.

Metadata: Metadata description file contains semantic information about the entity records/attributes and links underlying data files.

Data files: The data files in the CMD folder have a well-defined structure and format. These files must be in CSV format.

The following illustration shows the Common Data Model folder structure for Dynamics D365 Finance and Operations apps.

Data Lake Features

  • Has the ability to store and analyze data of any kind and size.

  • Boasts multiple access methods including U-SQL, Spark, Hive, HBase, and Storm.

  • Equipped with dynamic scaling to match business priorities.

  • Armed with enterprise-grade security with Azure Active Directory

  • Contain support for standard and custom entities.

  • Maintains continuous full/delta updates for analytic scenarios.

How can it be used with D365 F&O?

Dynamics 365 products, such as Finance and Operations apps, use Data Lake for AI and analytics. Data from external sources can easily be ingested into Data Lake via hundreds of ready-made connectors that are available in tools such as Power BI dataflows and Azure Data Factory. Historical data and earlier data that is often inherited through the transition to Finance and Operations apps can be ingested directly into Data Lake.

Exporting D365 data to Azure Data Lake

D365 F&O allows you to select an entity store to export data from Finance and Operations into Azure Data Lake.

The recent features of D365, allow select tables and entities to include and export Azure Aata Lake. After selecting the tables and entities, the system will make an initial copy. The system then keeps the selected data up to date by applying changes, deletions, and additions. There may be a delay of a few minutes between data changes in your Finance & Operations apps instances and the time it takes for the data to reflect on your Data Lake.

 

Data Lake Features

  • Analytical workspaces provide contextual and actionable insights into Finance and Operations apps. They give users a bird’s-eye view of a business process so that they can immediately get relevant information and take appropriate action.

  • You can bring your own data into Data Lake to supplement the data that D365 Finance and Operations provide.

Summary

Data Lake provides unlimited space to store large amounts of data, this includes structured, semi structured and unstructured data. It enables data analysts to analyze and understand the data effectively for business forecasting. Therefore, customers can take advantage of the strengths and cost advances that this technology offers.

Share
H B Patil

“My Self H B Patil working as Lead Technical Consultant”, I have an experience in working with Microsoft Dynamics AX retail and commerce. I am ambitious to explore new features in Dynamic D365 for commerce and implementing the same, my hobbies are net surfing, playing badminton, Watching web series and movies.iption–

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from Youtube
Vimeo
Consent to display content from Vimeo
Google Maps
Consent to display content from Google
Spotify
Consent to display content from Spotify
Sound Cloud
Consent to display content from Sound
Connect With Us