Notes

Apache Spark

2 min read

Apache Spark is a cluster computing frameworks, it is not a distributed file system, it is used for distributed computing. It is used for real-time processing and analysis of large amount of data and streams.

Spark supports all kinds of data models and file systems.

To start using spark we first have to initialize a SparkContext in the code.

NOTE: SparkContext is deprecated and only gave access to RDD apis SparkSession is now used and it does everything

Spark Component

The Spark core is the execution engine responsible for scheduling, distributed task dispatching, etc…

Resilient Distributed dataset

This is the fundamental data structure in spark, it is distributed on multiple spark clusters. It is immutable. Operations are done in it in parallel. Transformations and actions are can applied.

Due to the RDD being immutable every transformation returns a new RDD. Also all operations are stored in memory and so is the RDD itself, it is never stored in disk, this is what makes it so fast.

to create a simple RDD use:

sc.Paralellize(iterable like list or range)

DAGs in Apache spark

Apache spark uses DAGs to optimize its operations, this allows the transformation to be done lazily, not instantly when called. A new DAG is always built for a new action, the DAG contains all transformation that should be done now to get the current needed output, all other transformations are not done.

NOTE: transformations are functions that return a new RDD, and actions are all functions that basically prints or give me outputs (a side effect like printing).

Some transformations and actions

NOTE: collect() should only be used if the data is small, since this will collect all the parallelized data to the memory of the driver, it is better to use take(n)

Some actions:

rdd.reduce(f) -> integer
rdd.collect() -> None
rdd.count() -> int
rdd.collectAsMap() -> Dict

Word Count example

lines = sc.textFile("file.txt") # loads as a list of lines
lines = lines.flatMap(lambda line: line.split(" "))
words = lines.map(lambda word: (word, 1))

Graph View