Let us start first by understanding what is Databricks. Databricks is a Multi-Cloud LakeHouse Platform based on Apache Spark. But what is really a Lake House? A data lakehouse is a unified analytics platform that combines the best elements of data lakes and data warehouses. Data LakeHouse delivers the openness, flexibility and machine learning support of data lakes along with the reliability, strong governance and performance of data warehouses. So in the LakeHouse we work on that engineering, analytics and AI all in one platform.
Let us have a look at the architecture of Databricks LakeHouse.
It is divided into three important layers:
First, the Cloud Service: Databricks is Multi-Cloud available on Microsoft Azure, Amazon Web Services and Google Cloud. Then there is the Databricks runtime, which is a set of core components like Apache Spark,Delta Lake and other system libraries. We will see Delta Lake in detail in the next module. Databricks uses the infrastructure of our cloud providers to provision virtual machines or nodes of a cluster. And this cluster comes pre-installed with Databricks runtime. On top of all this, there is the Databricks Workspace allowing us to interactively implement and run our data engineering, analytics and AI workloads.
Let us now understand how Databricks resources are deployed in our cloud provider. There are two high level components: the control plane and the data plane. The control plane resides in Databricks account while the data plane is in our own cloud subscription.
Whenever we create a Databricks Workspace, it is deployed in the control plane along with Databricks Services like Databricks UI, Cluster Manager, Workflow Service and Notebooks. We will have the chance during this course to see all these services. On the other hand, a storage account is deployed in the data plan in our own subscription. It is used for Databricks File System or DBFS.
In addition, when we want to set up a spark cluster, the cluster virtual machine will be also deployed in the Data Plane. So to summarize, the compute and the storage will be always in our own cloud account. Databricks will provide us with the tools we need to use and control our infrastructure.
Databricks has been founded by the same engineers that developed Spark. Because it is based on Apache Spark. The data is distributed and processed in-memory of multiple nodes in a cluster. Databricks supports all the languages supported by Spark. Which are Scala, Python, SQL, R and Java as well. It has also support for batch processing and stream processing in Spark.
In addition, on Databricks, we can process data no matter if it is structured, semi-structured,or even unstructured like images and videos. And since Apache Spark process data in a distributed manner, Databricks offers a native support of a distributed file system called Databricks File System or DBFS. So whenever we create a cluster in Databricks, it comes pre-installed with DBFS. We usually use file systems to persist data and files. However, DBFS is just an abstraction layer, while it uses the underlying cloud storage to persistthe data. To illustrate this,If we create a file in our cluster and store it in DBFS. This file is actually persisted in the underlying cloud storage. Like our Azure storage or our S3 buckets. So even after the cluster is terminated, all the data is saved in our cloud storage.
https://github.com/derar-alhussein/Databricks-Certified-Data-Engineer-Associate
The next step in setting up our environment is to create a cluster. Navigate to Compute in the left side bar.
From here, we can create and manage our clusters. A Cluster is a set of nodes or computers working together like a single entity. It consists of a master node called the driver and some other worker nodes. The driver node is responsible for coordinating the workers and their parallel execution of tasks.
Now, under All-purpose compute,click Create compute. On top,Click on the default name in order to change it.
Name the cluster as Demo Cluster, for example. Leave the policy Unrestricted to create fully configurable cluster. Our cluster could be multi node, that is having multiple workers, or simply a single node. A single node cluster, has no workers and run Spark jobs on the driver node.
For this course. A single node is enough. However, let us continue to see how to configure a multi node cluster.
For the access mode, we can allow our cluster to be shared by multiple users. However, only SQL and Python workloads will be supported. Instead, choose single user if we are the only one to use this cluster. Next, we need to select the Databricks Runtime version. Databricks Runtime is the virtual machine's image that comes with preinstalled libraries,which has a specific version of Spark, Scala and other libraries. Let's choose 11. 3 LTS,which is the latest version at the time of recording this course.
In addition, we can choose to activate Photon, which is a vectorized query engine developed in C++ to enhance Spark performance. Now we can go ahead and select the configuration of our worker nodes.
These are different virtual machine sizes provided by our cloud provider,in this case, Azure. Depending on our requirements of memory, cores and hard disk, we can select the configuration. Let's keep the default one. Then we can select the number of workers we need for our cluster. If we choose to enable auto scaling, then we need to provide a range for the number of workers. These allow databricks to resize our cluster automatically within this range. Otherwise disable auto scaling and just a fixed number of workers.
Let us select three here. After selecting the worker node configuration, we can now select the configuration for the driver node or simply keep it the same as the worker. Next, we can enable auto termination of the cluster. By providing the number of minutes, let's say 30 minutes. That is, if there is no activity for 30 minutes,the cluster will auto terminate.
On the right, we can see a summary of our cluster configuration.
Here we can see the number of DBUs. DBU stand for Databricks Unit and it is a unit of a processing capability per hour. Each configuration tells us how much DBUs would be consumed if virtual machine run for an hour and then pay for each DBU consumed. For example, if we have less number of workers or even a single node cluster, we will have less DBUs.
Let us confirm this. Select Single Node cluster. And click Confirm. Now with single node cluster, we are going to consume less DBUs. Now, we can hit the Create button to finish creating our cluster. Azure will now go ahead and provision the required virtual machine with a specific configuration and libraries as specified by Databricks runtime. Great. Our cluster now is up and running.
To access our cluster at any time. We can simply navigate to Compute in the left side bar. We can see our clusters are listed here with its current status running or terminated. From here, we can quickly terminate or start our cluster. In addition to deleting the cluster and editing its permissions. We can even edit the cluster by clicking on its name. Then click Edit. Remember, changing the cluster configuration may require a restart of the cluster. Let's cancel this. Here in the cluster page, you can notice two things. The Event log,that shows all the events that have happened with the cluster. For example, when the cluster was created or terminated. If it is edited or if it is running fine. This helps to track the activity on a cluster. And, in the Driver Log,we will get the log generated within the cluster notebooks and libraries.