Background
Microsoft Azure Data Lake is a secure cloud platform hosted in Azure that enables big data analytics and artificial intelligence (AI). ‘Data Lake’ specifically refers to a storage technology that is based on Azure Data Lake Storage Gen2.
With the volume of data increasing day by day, enterprises are looking for suitable digital infrastructure to store and apply data analytics. Providing unlimited storage space for largeamounts of data, including structured, semi-structured and unstructured data, Data Lake is the solution to all your storage needs.
Besides storage, Azure Data Lake also enables data scientists, marketers, and analysts toanalyze and understand data effectively. The backbone of this platform features the Hadoop file system (including HDFS and YARN), which provides massive petabyte-sized file computing.
Azure Data Lake consists of the following components:
Azure Data Lake Store
Azure Data Lake Storage (ADLS), a Hadoop-based repository, sits directly on top of Blob Storage, which means, files that are stored in Blob Storage are simultaneously available through ADLS. ADLS also enables all key Blob Storage functionality, including Azure AD-based permissions, encryption at rest, data tiering and lifecycle policies. ADLS provides a Hadoop-compatible file system (HDFS), hierarchical namespace (folders/metadata), and high-performance access to large volumes of data required for data analytics.
Azure Data Analytics
Azure Data Lake Analytics is a service built on Apache YARN. The analytics service can handle jobs of any scale instantly, instead of having to go through the processes of deploying, configuring and tuning hardware. Data Lake supports a range of tools and programming languages (SQL-like queries, and REST calls) that enable large amounts of data to be queried, transformed and reported.
Common Data Model Folders
Data stored in Data Lake complies with the Common Data Model folder standard. Common Data Model folders contain metadata definitions in addition to data files. Metadata definitions are kept in model files.
Metadata: Metadata description file contains semantic information about the entity records/attributes and links underlying data files.
Data files: The data files in the CMD folder have a well-defined structure and format. These files must be in CSV format.
The following illustration shows the Common Data Model folder structure for Dynamics D365 Finance and Operations apps.
Data Lake Features
How can it be used with D365 F&O?
Dynamics 365 products, such as Finance and Operations apps, use Data Lake for AI and analytics. Data from external sources can easily be ingested into Data Lake via hundreds of ready-made connectors that are available in tools such as Power BI dataflows and Azure Data Factory. Historical data and earlier data that is often inherited through the transition to Finance and Operations apps can be ingested directly into Data Lake.
Exporting D365 data to Azure Data Lake
D365 F&O allows you to select an entity store to export data from Finance and Operations into Azure Data Lake.
The recent features of D365, allow select tables and entities to include and export Azure Aata Lake. After selecting the tables and entities, the system will make an initial copy. The system then keeps the selected data up to date by applying changes, deletions, and additions. There may be a delay of a few minutes between data changes in your Finance & Operations apps instances and the time it takes for the data to reflect on your Data Lake.
Data Lake Features
Summary
Data Lake provides unlimited space to store large amounts of data, this includes structured, semi structured and unstructured data. It enables data analysts to analyze and understand the data effectively for business forecasting. Therefore, customers can take advantage of the strengths and cost advances that this technology offers.
