Associate-Developer-Apache-Spark Study Guide Brilliant Associate-Developer-Apache-Spark Exam Dumps PDF [Q64-Q83]

Associate-Developer-Apache-Spark Study Guide Brilliant Associate-Developer-Apache-Spark Exam Dumps PDF

View Associate-Developer-Apache-Spark Exam Question Dumps With Latest Demo

The advantages of taking the Databricks Associate Developer Apache Spark Exam?

There are many benefits to taking the Databricks Associate Developer Apache Spark Exam, including earning certification for your skills and knowledge. The exam can also help you develop new skills in areas such as data science, big data, and programming. This can help you advance in your career. The exam can also help you land a new job. Many companies use this as part of their hiring process. Databricks Associate Developer Apache Spark exam dumps will help you to get certified in your first attempt.

The Databricks Associate Developer Apache Spark Exam is offered by the Databricks team. The company offers this exam to its employees so that they can advance in their careers. You should take this exam to earn certification and get a good job. The company also offers the exam to students and people who want to learn about big data. The Databricks team will give you a free practice test before you take the real exam. The company will also provide you with a certificate of completion. This means that you will have passed the exam.

NEW QUESTION 64
Which of the following statements about broadcast variables is correct?

A. Broadcast variables are immutable.
B. Broadcast variables are occasionally dynamically updated on a per-task basis.
C. Broadcast variables are commonly used for tables that do not fit into memory.
D. Broadcast variables are serialized with every single task.
E. Broadcast variables are local to the worker node and not shared across the cluster.

Answer: A

Explanation:
Explanation
Broadcast variables are local to the worker node and not shared across the cluster.
This is wrong because broadcast variables are meant to be shared across the cluster. As such, they are never just local to the worker node, but available to all worker nodes.
Broadcast variables are commonly used for tables that do not fit into memory.
This is wrong because broadcast variables can only be broadcast because they are small and do fit into memory.
Broadcast variables are serialized with every single task.
This is wrong because they are cached on every machine in the cluster, precisely avoiding to have to be serialized with every single task.
Broadcast variables are occasionally dynamically updated on a per-task basis.
This is wrong because broadcast variables are immutable - they are never updated.
More info: Spark - The Definitive Guide, Chapter 14

NEW QUESTION 65
The code block shown below should write DataFrame transactionsDf to disk at path csvPath as a single CSV file, using tabs (\t characters) as separators between columns, expressing missing values as string n/a, and omitting a header row with column names. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__.write.__2__(__3__, " ").__4__.__5__(csvPath)

A. 1. csv
2. option
3. "sep"
4. option("emptyValue", "n/a")
5. path
* 1. repartition(1)
2. mode
3. "sep"
4. mode("nullValue", "n/a")
5. csv
B. 1. repartition(1)
2. option
3. "sep"
4. option("nullValue", "n/a")
5. csv
(Correct)
C. 1. coalesce(1)
2. option
3. "colsep"
4. option("nullValue", "n/a")
5. path
D. 1. coalesce(1)
2. option
3. "sep"
4. option("header", True)
5. path

Answer: B

Explanation:
Explanation
Correct code block:
transactionsDf.repartition(1).write.option("sep", "\t").option("nullValue", "n/a").csv(csvPath) It is important here to understand that the question specifically asks for writing the DataFrame as a single CSV file. This should trigger you to think about partitions. By default, every partition is written as a separate file, so you need to include repatition(1) into your call. coalesce(1) works here, too!
Secondly, the question is very much an invitation to search through the parameters in the Spark documentation that work with DataFrameWriter.csv (link below). You will also need to know that you need an option() statement to apply these parameters.
The final concern is about the general call structure. Once you have called accessed write of your DataFrame, options follow and then you write the DataFrame with csv. Instead of csv(csvPath), you could also use save(csvPath, format='csv') here.
More info: pyspark.sql.DataFrameWriter.csv - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

NEW QUESTION 66
Which of the following code blocks returns DataFrame transactionsDf sorted in descending order by column predError, showing missing values last?

A. transactionsDf.orderBy("predError").asc_nulls_last()
B. transactionsDf.sort("predError", ascending=False)
C. transactionsDf.sort(asc_nulls_last("predError"))
D. transactionsDf.orderBy("predError").desc_nulls_last()
E. transactionsDf.desc_nulls_last("predError")

Answer: B

Explanation:
Explanation
transactionsDf.sort("predError", ascending=False)
Correct! When using DataFrame.sort() and setting ascending=False, the DataFrame will be sorted by the specified column in descending order, putting all missing values last. An alternative, although not listed as an answer here, would be transactionsDf.sort(desc_nulls_last("predError")).
transactionsDf.sort(asc_nulls_last("predError"))
Incorrect. While this is valid syntax, the DataFrame will be sorted on column predError in ascending order and not in descending order, putting missing values last.
transactionsDf.desc_nulls_last("predError")
Wrong, this is invalid syntax. There is no method DataFrame.desc_nulls_last() in the Spark API. There is a Spark function desc_nulls_last() however (link see below).
transactionsDf.orderBy("predError").desc_nulls_last()
No. While transactionsDf.orderBy("predError") is correct syntax (although it sorts the DataFrame by column predError in ascending order) and returns a DataFrame, there is no method DataFrame.desc_nulls_last() in the Spark API. There is a Spark function desc_nulls_last() however (link see below).
transactionsDf.orderBy("predError").asc_nulls_last()
Incorrect. There is no method DataFrame.asc_nulls_last() in the Spark API (see above).
More info: pyspark.sql.functions.desc_nulls_last - PySpark 3.1.2 documentation and pyspark.sql.DataFrame.sort - PySpark 3.1.2 documentation (https://bit.ly/3g1JtbI , https://bit.ly/2R90NCS) Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/32.html ,
https://bit.ly/sparkpracticeexams_import_instructions)

NEW QUESTION 67
Which of the following code blocks creates a new one-column, two-row DataFrame dfDates with column date of type timestamp?

A. 1.dfDates = spark.createDataFrame(["23/01/2022 11:28:12","24/01/2022 10:58:34"], ["date"])
2.dfDates = dfDates.withColumn("date", to_timestamp("dd/MM/yyyy HH:mm:ss", "date"))
B. 1.dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"])
2.dfDates = dfDates.withColumn("date", to_timestamp("date", "dd/MM/yyyy HH:mm:ss"))
C. 1.dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"])
D. 1.dfDates = spark.createDataFrame(["23/01/2022 11:28:12","24/01/2022 10:58:34"], ["date"])
2.dfDates = dfDates.withColumnRenamed("date", to_datetime("date", "yyyy-MM-dd HH:mm:ss"))
E. 1.dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"])
2.dfDates = dfDates.withColumnRenamed("date", to_timestamp("date", "yyyy-MM-dd HH:mm:ss"))

Answer: B

Explanation:
Explanation
This question is tricky. Two things are important to know here:
First, the syntax for createDataFrame: Here you need a list of tuples, like so: [(1,), (2,)]. To define a tuple in Python, if you just have a single item in it, it is important to put a comma after the item so that Python interprets it as a tuple and not just a normal parenthesis.
Second, you should understand the to_timestamp syntax. You can find out more about it in the documentation linked below.
For good measure, let's examine in detail why the incorrect options are wrong:
dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"]) This code snippet does everything the question asks for - except that the data type of the date column is a string and not a timestamp. When no schema is specified, Spark sets the string data type as default.
dfDates = spark.createDataFrame(["23/01/2022 11:28:12","24/01/2022 10:58:34"], ["date"]) dfDates = dfDates.withColumn("date", to_timestamp("dd/MM/yyyy HH:mm:ss", "date")) In the first row of this command, Spark throws the following error: TypeError: Can not infer schema for type:
<class 'str'>. This is because Spark expects to find row information, but instead finds strings. This is why you need to specify the data as tuples. Fortunately, the Spark documentation (linked below) shows a number of examples for creating DataFrames that should help you get on the right track here.
dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"]) dfDates = dfDates.withColumnRenamed("date", to_timestamp("date", "yyyy-MM-dd HH:mm:ss")) The issue with this answer is that the operator withColumnRenamed is used. This operator simply renames a column, but it has no power to modify its actual content. This is why withColumn should be used instead. In addition, the date format yyyy-MM-dd HH:mm:ss does not reflect the format of the actual timestamp: "23/01/2022 11:28:12".
dfDates = spark.createDataFrame(["23/01/2022 11:28:12","24/01/2022 10:58:34"], ["date"]) dfDates = dfDates.withColumnRenamed("date", to_datetime("date", "yyyy-MM-dd HH:mm:ss")) Here, withColumnRenamed is used instead of withColumn (see above). In addition, the rows are not expressed correctly - they should be written as tuples, using parentheses. Finally, even the date format is off here (see above).
More info: pyspark.sql.functions.to_timestamp - PySpark 3.1.2 documentation and pyspark.sql.SparkSession.createDataFrame - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 2

NEW QUESTION 68
Which of the following code blocks writes DataFrame itemsDf to disk at storage location filePath, making sure to substitute any existing data at that location?

A. itemsDf.write.mode("overwrite").path(filePath)
B. itemsDf.write.option("parquet").mode("overwrite").path(filePath)
C. itemsDf.write(filePath, mode="overwrite")
D. itemsDf.write().parquet(filePath, mode="overwrite")
E. itemsDf.write.mode("overwrite").parquet(filePath)

Answer: E

Explanation:
Explanation
itemsDf.write.mode("overwrite").parquet(filePath)
Correct! itemsDf.write returns a pyspark.sql.DataFrameWriter instance whose overwriting behavior can be modified via the mode setting or by passing mode="overwrite" to the parquet() command.
Although the parquet format is not prescribed for solving this question, parquet() is a valid operator to initiate Spark to write the data to disk.
itemsDf.write.mode("overwrite").path(filePath)
No. A pyspark.sql.DataFrameWriter instance does not have a path() method.
itemsDf.write.option("parquet").mode("overwrite").path(filePath)
Incorrect, see above. In addition, a file format cannot be passed via the option() method.
itemsDf.write(filePath, mode="overwrite")
Wrong. Unfortunately, this is too simple. You need to obtain access to a DataFrameWriter for the DataFrame through calling itemsDf.write upon which you can apply further methods to control how Spark data should be written to disk. You cannot, however, pass arguments to itemsDf.write directly.
itemsDf.write().parquet(filePath, mode="overwrite")
False. See above.
More info: pyspark.sql.DataFrameWriter.parquet - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3

NEW QUESTION 69
The code block shown below should return a one-column DataFrame where the column storeId is converted to string type. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__.__3__(__4__))

A. 1. select
2. storeId
3. cast
4. StringType()
B. 1. select
2. col("storeId")
3. as
4. StringType
C. 1. cast
2. "storeId"
3. as
4. StringType()
D. 1. select
2. col("storeId")
3. cast
4. StringType
E. 1. select
2. col("storeId")
3. cast
4. StringType()

Answer: E

Explanation:
Explanation
Correct code block:
transactionsDf.select(col("storeId").cast(StringType()))
Solving this question involves understanding that, when using types from the pyspark.sql.types such as StringType, these types need to be instantiated when using them in Spark, or, in simple words, they need to be followed by parentheses like so: StringType(). You could also use .cast("string") instead, but that option is not given here.
More info: pyspark.sql.Column.cast - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2

NEW QUESTION 70
The code block shown below should return a DataFrame with columns transactionsId, predError, value, and f from DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__)

A. 1. filter
2. "transactionId", "predError", "value", "f"
B. 1. where
2. col("transactionId"), col("predError"), col("value"), col("f")
C. 1. select
2. col(["transactionId", "predError", "value", "f"])
D. 1. select
2. ["transactionId", "predError", "value", "f"]
E. 1. select
2. "transactionId, predError, value, f"

Answer: D

Explanation:
Explanation
Correct code block:
transactionsDf.select(["transactionId", "predError", "value", "f"])
The DataFrame.select returns specific columns from the DataFrame and accepts a list as its only argument.
Thus, this is the correct choice here. The option using col(["transactionId", "predError",
"value", "f"]) is invalid, since inside col(), one can only pass a single column name, not a list. Likewise, all columns being specified in a single string like "transactionId, predError, value, f" is not valid syntax.
filter and where filter rows based on conditions, they do not control which columns to return.
Static notebook | Dynamic notebook: See test 2

NEW QUESTION 71
Which of the following code blocks prints out in how many rows the expression Inc. appears in the string-type column supplier of DataFrame itemsDf?

A. 1.accum=sc.accumulator(0)
2.
3.def check_if_inc_in_supplier(row):
4. if 'Inc.' in row['supplier']:
5. accum.add(1)
6.
7.itemsDf.foreach(check_if_inc_in_supplier)
8.print(accum.value)
B. print(itemsDf.foreach(lambda x: 'Inc.' in x).sum())
C. 1.counter = 0
2.
3.def count(x):
4. if 'Inc.' in x['supplier']:
5. counter = counter + 1
6.
7.itemsDf.foreach(count)
8.print(counter)
D. 1.counter = 0
2.
3.for index, row in itemsDf.iterrows():
4. if 'Inc.' in row['supplier']:
5. counter = counter + 1
6.
7.print(counter)
E. print(itemsDf.foreach(lambda x: 'Inc.' in x))

Answer: A

Explanation:
Explanation
Correct code block:
accum=sc.accumulator(0)
def check_if_inc_in_supplier(row):
if 'Inc.' in row['supplier']:
accum.add(1)
itemsDf.foreach(check_if_inc_in_supplier)
print(accum.value)
To answer this question correctly, you need to know both about the DataFrame.foreach() method and accumulators.
When Spark runs the code, it executes it on the executors. The executors do not have any information about variables outside of their scope. This is whhy simply using a Python variable counter, like in the two examples that start with counter = 0, will not work. You need to tell the executors explicitly that counter is a special shared variable, an Accumulator, which is managed by the driver and can be accessed by all executors for the purpose of adding to it.
If you have used Pandas in the past, you might be familiar with the iterrows() command. Notice that there is no such command in PySpark.
The two examples that start with print do not work, since DataFrame.foreach() does not have a return value.
More info: pyspark.sql.DataFrame.foreach - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

NEW QUESTION 72
The code block displayed below contains an error. The code block should return a DataFrame in which column predErrorAdded contains the results of Python function add_2_if_geq_3 as applied to numeric and nullable column predError in DataFrame transactionsDf. Find the error.
Code block:
1.def add_2_if_geq_3(x):
2. if x is None:
3. return x
4. elif x >= 3:
5. return x+2
6. return x
7.
8.add_2_if_geq_3_udf = udf(add_2_if_geq_3)
9.
10.transactionsDf.withColumnRenamed("predErrorAdded", add_2_if_geq_3_udf(col("predError")))

A. Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.
B. The udf() method does not declare a return type.
C. UDFs are only available through the SQL API, but not in the Python API as shown in the code block.
D. The Python function is unable to handle null values, resulting in the code block crashing on execution.
E. The operator used to adding the column does not add column predErrorAdded to the DataFrame.

Answer: E

Explanation:
Explanation
Correct code block:
def add_2_if_geq_3(x):
if x is None:
return x
elif x >= 3:
return x+2
return x
add_2_if_geq_3_udf = udf(add_2_if_geq_3)
transactionsDf.withColumn("predErrorAdded", add_2_if_geq_3_udf(col("predError"))).show() Instead of withColumnRenamed, you should use the withColumn operator.
The udf() method does not declare a return type.
It is fine that the udf() method does not declare a return type, this is not a required argument. However, the default return type is StringType. This may not be the ideal return type for numeric, nullable data - but the code will run without specified return type nevertheless.
The Python function is unable to handle null values, resulting in the code block crashing on execution.
The Python function is able to handle null values, this is what the statement if x is None does.
UDFs are only available through the SQL API, but not in the Python API as shown in the code block.
No, they are available through the Python API. The code in the code block that concerns UDFs is correct.
Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.
You may choose to use the transactionsDf.predError syntax, but the col("predError") syntax is fine.

A. itemsDf.select(explode("attributes").alias("attributes_exploded")).filter(attributes_exploded.contains("i"))
B. itemsDf.select(explode("attributes")).filter("attributes_exploded".contains("i"))
C. itemsDf.select(col("attributes").explode().alias("attributes_exploded")).filter(col("attributes_exploded").co
D. itemsDf.select(explode("attributes").alias("attributes_exploded")).filter(col("attributes_exploded").contain
E. itemsDf.explode(attributes).alias("attributes_exploded").filter(col("attributes_exploded").contains("i"))

Answer: D

Explanation:
Explanation
Result of correct code block:
+-------------------+
|attributes_exploded|
+-------------------+
| winter|
| cooling|
+-------------------+
To solve this question, you need to know about explode(). This operation helps you to split up arrays into single rows. If you did not have a chance to familiarize yourself with this method yet, find more examples in the documentation (link below).
Note that explode() is a method made available through pyspark.sql.functions - it is not available as a method of a DataFrame or a Column, as written in some of the answer options.
More info: pyspark.sql.functions.explode - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2

NEW QUESTION 74
Which of the following describes the role of tasks in the Spark execution hierarchy?

A. Tasks are the second-smallest element in the execution hierarchy.
B. Stages with narrow dependencies can be grouped into one task.
C. Tasks with wide dependencies can be grouped into one stage.
D. Tasks are the smallest element in the execution hierarchy.
E. Within one task, the slots are the unit of work done for each partition of the data.

Answer: D

Explanation:
Explanation
Stages with narrow dependencies can be grouped into one task.
Wrong, tasks with narrow dependencies can be grouped into one stage.
Tasks with wide dependencies can be grouped into one stage.
Wrong, since a wide transformation causes a shuffle which always marks the boundary of a stage. So, you cannot bundle multiple tasks that have wide dependencies into a stage.
Tasks are the second-smallest element in the execution hierarchy.
No, they are the smallest element in the execution hierarchy.
Within one task, the slots are the unit of work done for each partition of the data.
No, tasks are the unit of work done per partition. Slots help Spark parallelize work. An executor can have multiple slots which enable it to process multiple tasks in parallel.

NEW QUESTION 75
Which of the following describes Spark's Adaptive Query Execution?

A. Adaptive Query Execution features are dynamically switching join strategies and dynamically optimizing skew joins.
B. Adaptive Query Execution applies to all kinds of queries.
C. Adaptive Query Execution is enabled in Spark by default.
D. Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.
E. Adaptive Query Execution reoptimizes queries at execution points.

Answer: A

Explanation:
Explanation
Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.
This is almost correct. All of these features, except for dynamically injecting scan filters, are part of Adaptive Query Execution. Dynamically injecting scan filters for join operations to limit the amount of data to be considered in a query is part of Dynamic Partition Pruning and not of Adaptive Query Execution.
Adaptive Query Execution reoptimizes queries at execution points.
No, Adaptive Query Execution reoptimizes queries at materialization points.
Adaptive Query Execution is enabled in Spark by default.
No, Adaptive Query Execution is disabled in Spark needs to be enabled through the spark.sql.adaptive.enabled property.
Adaptive Query Execution applies to all kinds of queries.
No, Adaptive Query Execution applies only to queries that are not streaming queries and that contain at least one exchange (typically expressed through a join, aggregate, or window operator) or one subquery.
More info: How to Speed up SQL Queries with Adaptive Query Execution, Learning Spark, 2nd Edition, Chapter 12 (https://bit.ly/3tOh8M1)

NEW QUESTION 76
The code block shown below should add a column itemNameBetweenSeparators to DataFrame itemsDf. The column should contain arrays of maximum 4 strings. The arrays should be composed of the values in column itemsDf which are separated at - or whitespace characters. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Sample of DataFrame itemsDf:
1.+------+----------------------------------+-------------------+
2.|itemId|itemName |supplier |
3.+------+----------------------------------+-------------------+
4.|1 |Thick Coat for Walking in the Snow|Sports Company Inc.|
5.|2 |Elegant Outdoors Summer Dress |YetiX |
6.|3 |Outdoors Backpack |Sports Company Inc.|
7.+------+----------------------------------+-------------------+
Code block:
itemsDf.__1__(__2__, __3__(__4__, "[\s\-]", __5__))

A. 1. withColumnRenamed
2. "itemName"
3. split
4. "itemNameBetweenSeparators"
5. 4
B. 1. withColumn
2. "itemNameBetweenSeparators"
3. split
4. "itemName"
5. 4
(Correct)
C. 1. withColumnRenamed
2. "itemNameBetweenSeparators"
3. split
4. "itemName"
5. 4
D. 1. withColumn
2. itemNameBetweenSeparators
3. str_split
4. "itemName"
5. 5
E. 1. withColumn
2. "itemNameBetweenSeparators"
3. split
4. "itemName"
5. 5

Answer: B

Explanation:
Explanation
This question deals with the parameters of Spark's split operator for strings.
To solve this question, you first need to understand the difference between DataFrame.withColumn() and DataFrame.withColumnRenamed(). The correct option here is DataFrame.withColumn() since, according to the question, we want to add a column and not rename an existing column. This leaves you with only 3 answers to consider.
The second gap should be filled with the name of the new column to be added to the DataFrame. One of the remaining answers states the column name as itemNameBetweenSeparators, while the other two state it as "itemNameBetweenSeparators". The correct option here is
"itemNameBetweenSeparators", since the other option would let Python try to interpret itemNameBetweenSeparators as the name of a variable, which we have not defined. This leaves you with 2 answers to consider.
The decision boils down to how to fill gap 5. Either with 4 or with 5. The question asks for arrays of maximum four strings. The code in gap 5 relates to the limit parameter of Spark's split operator (see documentation linked below). The documentation states that "the resulting array's length will not be more than limit", meaning that we should pick the answer option with 4 as the code in the fifth gap here.
On a side note: One answer option includes a function str_split. This function does not exist in pySpark.
More info: pyspark.sql.functions.split - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

NEW QUESTION 77
The code block displayed below contains an error. The code block is intended to perform an outer join of DataFrames transactionsDf and itemsDf on columns productId and itemId, respectively.
Find the error.
Code block:
transactionsDf.join(itemsDf, [itemsDf.itemId, transactionsDf.productId], "outer")

A. The join type needs to be appended to the join() operator, like join().outer() instead of listing it as the last argument inside the join() call.
B. The "outer" argument should be eliminated from the call and join should be replaced by joinOuter.
C. The term [itemsDf.itemId, transactionsDf.productId] should be replaced by itemsDf.itemId == transactionsDf.productId.
D. The "outer" argument should be eliminated, since "outer" is the default join type.
E. The term [itemsDf.itemId, transactionsDf.productId] should be replaced by itemsDf.col("itemId") == transactionsDf.col("productId").

Answer: C

Explanation:
Explanation
Correct code block:
transactionsDf.join(itemsDf, itemsDf.itemId == transactionsDf.productId, "outer") Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/33.html ,
https://bit.ly/sparkpracticeexams_import_instructions)

NEW QUESTION 78
In which order should the code blocks shown below be run in order to assign articlesDf a DataFrame that lists all items in column attributes ordered by the number of times these items occur, from most to least often?
Sample of DataFrame articlesDf:
1.+------+-----------------------------+-------------------+
2.|itemId|attributes |supplier |
3.+------+-----------------------------+-------------------+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+------+-----------------------------+-------------------+

A. 2, 5, 4
B. 4, 5
C. 2, 5, 3
D. 1. articlesDf = articlesDf.groupby("col")
2. articlesDf = articlesDf.select(explode(col("attributes")))
3. articlesDf = articlesDf.orderBy("count").select("col")
4. articlesDf = articlesDf.sort("count",ascending=False).select("col")
5. articlesDf = articlesDf.groupby("col").count()
E. 5, 2
F. 2, 3, 4

Answer: F

NEW QUESTION 79
Which of the following describes characteristics of the Spark driver?

A. If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.
B. The Spark driver processes partitions in an optimized, distributed fashion.
C. In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.
D. The Spark driver's responsibility includes scheduling queries for execution on worker nodes.
E. The Spark driver requests the transformation of operations into DAG computations from the worker nodes.

Answer: C

Explanation:
Explanation
The Spark driver requests the transformation of operations into DAG computations from the worker nodes.
No, the Spark driver transforms operations into DAG computations itself.
If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.
No. There is always a single driver per application, but one or more executors.
The Spark driver processes partitions in an optimized, distributed fashion.
No, this is what executors do.
In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.
Wrong. In a non-interactive Spark application, you need to create the SparkSession object. In an interactive Spark shell, the Spark driver instantiates the object for you.

NEW QUESTION 80
Which of the following code blocks returns a DataFrame with an added column to DataFrame transactionsDf that shows the unix epoch timestamps in column transactionDate as strings in the format month/day/year in column transactionDateFormatted?
Excerpt of DataFrame transactionsDf:

A. transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate"))
B. transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="MM/dd/yyyy"))
C. transactionsDf.withColumnRenamed("transactionDate", "transactionDateFormatted", from_unixtime("transactionDateFormatted", format="MM/dd/yyyy"))
D. transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="dd/MM/yyyy"))
E. transactionsDf.apply(from_unixtime(format="MM/dd/yyyy")).asColumn("transactionDateFormatted")

Answer: B

Explanation:
Explanation
transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="MM/dd/yyyy")) Correct. This code block adds a new column with the name transactionDateFormatted to DataFrame transactionsDf, using Spark's from_unixtime method to transform values in column transactionDate into strings, following the format requested in the question.
transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="dd/MM/yyyy")) No. Although almost correct, this uses the wrong format for the timestamp to date conversion: day/month/year instead of month/day/year.
transactionsDf.withColumnRenamed("transactionDate", "transactionDateFormatted", from_unixtime("transactionDateFormatted", format="MM/dd/yyyy")) Incorrect. This answer uses wrong syntax. The command DataFrame.withColumnRenamed() is for renaming an existing column only has two string parameters, specifying the old and the new name of the column.
transactionsDf.apply(from_unixtime(format="MM/dd/yyyy")).asColumn("transactionDateFormatted") Wrong. Although this answer looks very tempting, it is actually incorrect Spark syntax. In Spark, there is no method DataFrame.apply(). Spark has an apply() method that can be used on grouped data - but this is irrelevant for this question, since we do not deal with grouped data here.
transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate")) No. Although this is valid Spark syntax, the strings in column transactionDateFormatted would look like this:
2020-04-26 15:35:32, the default format specified in Spark for from_unixtime and not what is asked for in the question.
More info: pyspark.sql.functions.from_unixtime - PySpark 3.1.1 documentation and pyspark.sql.DataFrame.withColumnRenamed - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

NEW QUESTION 81
Which of the following code blocks reads in the JSON file stored at filePath as a DataFrame?

A. spark.read.path(filePath)
B. spark.read.path(filePath, source="json")
C. spark.read().path(filePath)
D. spark.read.json(filePath)
E. spark.read().json(filePath)

Answer: D

Explanation:
Explanation
spark.read.json(filePath)
Correct. spark.read accesses Spark's DataFrameReader. Then, Spark identifies the file type to be read as JSON type by passing filePath into the DataFrameReader.json() method.
spark.read.path(filePath)
Incorrect. Spark's DataFrameReader does not have a path method. A universal way to read in files is provided by the DataFrameReader.load() method (link below).
spark.read.path(filePath, source="json")
Wrong. A DataFrameReader.path() method does not exist (see above).
spark.read().json(filePath)
Incorrect. spark.read is a way to access Spark's DataFrameReader. However, the DataFrameReader is not callable, so calling it via spark.read() will fail.
spark.read().path(filePath)
No, Spark's DataFrameReader is not callable (see above).
More info: pyspark.sql.DataFrameReader.json - PySpark 3.1.2 documentation, pyspark.sql.DataFrameReader.load - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3

NEW QUESTION 82
Which of the following DataFrame operators is never classified as a wide transformation?

A. DataFrame.aggregate()
B. DataFrame.select()
C. DataFrame.join()
D. DataFrame.sort()
E. DataFrame.repartition()

Answer: B

Explanation:
Explanation
As a general rule: After having gone through the practice tests you probably have a good feeling for what classifies as a wide and what classifies as a narrow transformation. If you are unsure, feel free to play around in Spark and display the explanation of the Spark execution plan via DataFrame.[operation, for example sort()].explain(). If repartitioning is involved, it would count as a wide transformation.
DataFrame.select()
Correct! A wide transformation includes a shuffle, meaning that an input partition maps to one or more output partitions. This is expensive and causes traffic across the cluster. With the select() operation however, you pass commands to Spark that tell Spark to perform an operation on a specific slice of any partition. For this, Spark does not need to exchange data across partitions, each partition can be worked on independently. Thus, you do not cause a wide transformation.
DataFrame.repartition()
Incorrect. When you repartition a DataFrame, you redefine partition boundaries. Data will flow across your cluster and end up in different partitions after the repartitioning is completed. This is known as a shuffle and, in turn, is classified as a wide transformation.
DataFrame.aggregate()
No. When you aggregate, you may compare and summarize data across partitions. In the process, data are exchanged across the cluster, and newly formed output partitions depend on one or more input partitions. This is a typical characteristic of a shuffle, meaning that the aggregate operation may classify as a wide transformation.
DataFrame.join()
Wrong. Joining multiple DataFrames usually means that large amounts of data are exchanged across the cluster, as new partitions are formed. This is a shuffle and therefore DataFrame.join() counts as a wide transformation.
DataFrame.sort()
False. When sorting, Spark needs to compare many rows across all partitions to each other. This is an expensive operation, since data is exchanged across the cluster and new partitions are formed as data is reordered. This process classifies as a shuffle and, as a result, DataFrame.sort() counts as wide transformation.
More info: Understanding Apache Spark Shuffle | Philipp Brunenberg

NEW QUESTION 83
......

What is the Databricks Associate Developer Apache Spark Exam?

The Databricks Associate Developer Apache Spark Exam is a certification that can be earned by anyone who has successfully completed the Databricks Associate Developer Apache Spark Certification Training. The exam covers all the material that was covered in the training. The exam is designed to test your knowledge of the concepts, skills, and abilities that you learned during the course.

Do you want to become a Data Engineer or a Spark Architect? If so, then the Databricks Associate Developer Apache Spark Exam is a must-pass. The Databricks Associate Developer Apache Spark Exam is designed to help you develop a complete understanding of the technology used by the Databricks platform. You will learn about the basics of Spark, including the Spark programming language, Spark SQL, Spark Streaming, and the Spark ecosystem. Databricks Associate Developer Apache Spark exam dumps are the choice of champions.

The Databricks Associate Developer Apache Spark Exam is a test that aims to assess whether you have the knowledge required to become a certified Apache Spark developer. The Databricks Associate Developer Apache Spark Exam consists of two parts: the first part tests your knowledge of the fundamentals of the Apache Spark framework and the second part tests your ability to apply this knowledge. This post will help you get a head start in preparing for the Databricks Associate Developer Apache Spark Exam. The executors disk division actions documentation frame for the executor syntax variables object return allowed partition for the fit output transformation to induce couple of manager and evaluated expected safely, lazily named nodes broadcast operations for correctly mock driver.

Free Associate-Developer-Apache-Spark Test Questions Real Practice Test Questions: https://www.2pass4sure.com/Databricks-Certification/Associate-Developer-Apache-Spark-actual-exam-braindumps.html

Associate-Developer-Apache-Spark Dumps Updated Feb 16, 2023 WIith 179 Questions: https://drive.google.com/open?id=1Us7MZtP-VBTefOj2SfxLeQZq5ERbeFPa

Associate-Developer-Apache-Spark Study Guide Brilliant Associate-Developer-Apache-Spark Exam Dumps PDF [Q64-Q83]

Databricks Associate-Developer-Apache-Spark Certification Practice Exam

Associate-Developer-Apache-Spark Study Guide Brilliant Associate-Developer-Apache-Spark Exam Dumps PDF [Q64-Q83]

The advantages of taking the Databricks Associate Developer Apache Spark Exam?

What is the Databricks Associate Developer Apache Spark Exam?

Related Articles