Data EngineeringFunctional Programming Features In ScalaScala language features and patternsKevin LawrenceBlockedUnblockFollowFollowingJul 1I’ve been exploring functional programming with Scala and its eco system for the past few months.
In this post, I’ll highlight some of the features of the language that enable intuitive creation of functional code for distributed systems and data operations.
Photo by apoorv mittal on UnsplashHigher Order FunctionsAs per the official documentation, Functions are first class objects in Scala, which means that they can -Take another function as an argument, or …Return a functionAn example of a function taking another function as an argument is the map() function in Scala's standard collections library.
val examplelist: List[Int] = List(2,9,8,14)examplelist.
map(x=> x * 2) // anonymous function as argumentWhen working with standard Scala collections, it’s also very intuitive to chain operators, especially with the infix notation.
In the small code example below, I’m defining a list of numbers from 1 to 20, filtering on even numbers and then summing them up.
(1 to 20).
toList filter (_%2 == 0) reduce (_ + _)The _ is the wildcard operator – in the case of maps and filters, it refers to the value in the collection.
RecursionThe recommended way to do operations on all the items in a collection is to use the operators map, flatMap, or reduce.
In case those operators don’t meet a use case’s requirements, it’s very useful to write a tail-recursive function to operate on all the items in a collection.
The code example below shows a tail-recursive function definition to compute the factorial of a number.
import scala.
annotation.
tailrec@tailrec// Factorial Function Implementation that uses Tail Recursiondef factorial(in_x: Double, prodsofar: Double = 1.
0): Double = { if (in_x==0) prodsofar else factorial(in_x-1, prodsofar*in_x)}factorial(5)In Scala, a tail-recursive function, as above, can be optimised by the compiler (using the @tailrec annotation above) to occupy just 1 stack frame – so there's no chance of a stackoverflow error even for many levels of recursion.
This is possible out-of-the-box, without any need for frameworks or plugins.
As mentioned above, the recommended way is to use collections operators (such as reduce, etc.
).
As a demo of the simplicity of the collection's APIs, the above factorial function can also be implemented by the 1-liner below -(1 to 5).
toList reduce (_*_)To conceptually understand reduce, check out this great link!(Also do check out the explanations of foldLeft, foldRight, map, flatMap to understand some commonly used data operations!)Photo by Patrick Baum on UnsplashCase ClassesCase classes can be instantiated very easily with no boiler plate code, such as the example below.
case class BusinessTransaction(sourceaccountid: Long, targetaccountid: Long, amount: Long)// create some transactions now to demo case classes// I lend my friendval 1_xaction = BusinessTransaction(112333L, 998882L, 20L)// My friend pays me back val 2_xaction = BusinessTransaction(998882L, 112333L, 20L)Just 1 case class .
line above does the following useful things -Defines the 3 immutable values sourceaccountid, targetaccountid and amountDefines get methods to access the constructor arguments (eg: 1_xaction.
amount)While the ease of use is great, case classes are the recommended way to store immutable data instances in Scala.
For example, in a big data application, each line of a large datafile can be modelled by a case class and stored.
An example of the use of a case class to store data is here.
In the linked example, the function rawPostings models each line of the datafile as an instance of case class Posting.
Eventually, it returns a dataset of type RDD[Posting].
Pattern MatchingIn Scala, objects such as case classes, regular classes, and collections can be decomposed through pattern matching.
You can use pattern matching to -Decompose an object’s type (example below)Get the head of a collection (such as a List or a Seq)The code example below shows how to use pattern matching to decompose a Seq.
val seq1: Seq[Int] = Seq(1,3,4,5,5)seq1 match { case x::y => println(s"The first element in the sequence is ${x}") case Nil => println("The sequence is empty")}The cons operator (::) creates a list made of the head ( x) and the rest of the list (called the tail, y).
Companion ObjectsIn OOP, a static variable is sometimes used in a class to store state or property across multiple instantiated objects.
However, there is no static keyword in Scala.
Instead, what we use are Companion Objects aka Singleton Objects.
A Companion Object is defined using the object keyword and has the exact same name as the class that its accompanying.
The companion objects can define immutable values, which can then be referenced by methods in the class.
There are 2 common patterns to use companion objects in Scala -As a factory methodTo provide functionality that is common to the class (i.
e.
static function in Java)// The 'val specie' straightaway defines an immmutable parameter abstract class Animal(val specie: String) { import Animal.
_ // Common Behaviour to be mixed-in to Canine/Feline classes def getConnectionParameters: String = Animal.
connectionParameter }object Animal { // .
apply() is the factory method def apply(specie: String): Animal = specie match { case "dog" => new Canine(specie) case "cat" => new Feline(specie) } val connectionParameter:String = System.
getProperty("user.
dir") }class Canine(override val specie: String) extends Animal(specie) { override def toString: String = s"Canine of specie ${specie}"}class Feline(override val specie: String) extends Animal(specie) { override def toString: String = s"Feline of specie ${specie}"} // syntactic sugar, where we don't have to say new Animalval doggy = Animal("dog")val kitty = Animal("cat")doggy.
getConnectionParametersOptionsMost application code checks for Null/None types.
Null types are handled a little differently in Scala — the construct used is called an Option.
This is best demonstrated with an example.
val customermap: Map[Int, String] = Map( 11-> "CustomerA", 22->"CustomerB", 33->"CustomerC")customermap.
get(11) // Map's get() returns an Option[String]customermap.
get(11).
get // Option's get returns the Stringcustomermap.
get(999).
get // Will throw a NoSuchElementExceptioncustomermap.
get(999).
getOrElse(0) // Will return a 0 instead of throwing an exceptionIn a language like Python, if None: checks would be quite common throughout the codebase.
In Java, there would be try-catch blocks that would handle thrown exceptions.
Option s allow for focusing on the logic flow with minimal diversions for type or exception checks.
A standard way of using Options in Scala is for your custom functions to return Option[String] (or Int, Long, etc.
).
Let's look at the Map structure's get() function signature -def get(key: A): Option[B]One (intuitive) way to use this is to chain it with the getOrElse() function as shown below -// Map of IDs vs Namesval customermap: Map[Int, String] = Map( 11-> "CustomerA", 22->"CustomerB", 33->"CustomerC")customermap.
get(11).
getOrElse("No customer found for the provided ID")A very useful way of using Options is together with a collection operator like flatMap that directly handles the types for you transparently.
// Map of IDs vs Namesval customermap: Map[Int, String] = Map( 11-> "CustomerA", 22->"CustomerB", 33->"CustomerC")val listofids: List[Int] = List(11,22,33,99)listofids flatMap (id=> customermap.
get(id)) //flatMap magicAnd that’s it for this post from me!.I’m looking into concurrent systems with Akka and the Actor model currently.
In a future post, I’ll share my learnings on that topic (and its relationship to Scala’s approach to functional programming).
Originally published at http://github.
com.
.. More details