Databricks : Databricks-Certified-Professional-Data-Engineer Questions & Answers

A data engineer is tasked with building a nightly batch ETL pipeline that processes very large volumes of raw JSON logs from a data lake into Delta tables for reporting. The data arrives in bulk once per day, and the pipeline takes several hours to complete. Cost efficiency is important, but performance and reliability of completing the pipeline are the highest priorities.
Which type of Databricks cluster should the data engineer configure?

• A. A high-concurrency cluster designed for interactive SQL workloads.
• B. A lightweight single-node cluster with low worker node count to reduce costs.
• C. An all-purpose cluster always kept running to ensure low-latency job startup times.
• D. A job cluster configured to autoscale across multiple workers during the pipeline run.

A data engineer needs to capture pipeline settings from an existing in the workspace, and use them to create and version a JSON file to create a new pipeline.
Which command should the data engineer enter in a web terminal configured with the Databricks CLI?

• A. Use list pipelines to get the specs for all pipelines; get the pipeline spec from the return results parse and use this to create a pipeline
• B. Stop the existing pipeline; use the returned settings in a reset command
• C. Use the get command to capture the settings for the existing pipeline; remove the pipeline_id and rename the pipeline; use this in a create command
• D. Use the alone command to create a copy of an existing pipeline; use the get JSON command to get the pipeline definition; save this to git

Each configuration below is identical to the extent that each cluster has 400 GB total of RAM, 160 total cores and only one Executor per VM.
Given a job with at least one wide transformation, which of the following cluster configurations will result in maximum performance?

• A. * Total VMs; 1
  * 400 GB per Executor
  * 160 Cores / Executor
• B. * Total VMs: 4
  * 100 GB per Executor
  * 40 Cores/Executor
• C. * Total VMs: 8
  * 50 GB per Executor
  * 20 Cores / Executor
• D. * Total VMs:2
  * 200 GB per Executor
  * 80 Cores / Executor

A junior data engineer is migrating a workload from a relational database system to the Databricks Lakehouse. The source system uses a star schema, leveraging foreign key constrains and multi-table inserts to validate records on write.
Which consideration will impact the decisions made by the engineer while migrating this workload?

• A. Foreign keys must reference a primary key field; multi-table inserts must leverage Delta Lake ' s upsert functionality.
• B. All Delta Lake transactions are ACID compliance against a single table, and Databricks does not enforce foreign key constraints.
• C. Committing to multiple tables simultaneously requires taking out multiple table locks and can lead to a state of deadlock.
• D. Databricks only allows foreign key constraints on hashed identifiers, which avoid collisions in highly- parallel writes.

A data team is automating a daily multi-task ETL pipeline in Databricks. The pipeline includes a notebook for ingesting raw data, a Python wheel task for data transformation, and a SQL query to update aggregates. They want to trigger the pipeline programmatically and see previous runs in the GUI. They need to ensure tasks are retried on failure and stakeholders are notified by email if any task fails.
Which two approaches will meet these requirements? (Choose 2 answers)

• A. Use Databricks Asset Bundles (DABs) to deploy the workflow, then trigger individual tasks directly by referencing each task's notebook or script path in the workspace.
• B. Trigger the job programmatically using the Databricks Jobs REST API (/jobs/run-now), the CLI (databricks jobs run-now), or one of the Databricks SDKs.
• C. Create a multi-task job using the UI, Databricks Asset Bundles (DABs), or the Jobs REST API (/jobs
  /create) with notebook, Python wheel, and SQL tasks. Configure task-level retries and email notifications in the job definition.
• D. Use the REST API endpoint /jobs/runs/submit to trigger each task individually as separate job runs and implement retries using custom logic in the orchestrator.
• E. Create a single orchestrator notebook that calls each step with dbutils.notebook.run(), defining a job for that notebook and configuring retries and notifications at the notebook level.