A Simple Implementation of Filtering Logic in Go

for _, r := range records { keep := true for _, f := range filters { if !f(r) { keep = false break } } if keep { filteredRecords = append(filteredRecords, r) } } return filteredRecords}// ApplyBulkFilters applies a set of filters to the entire slice of records.// Used when each record filter requires knowledge of the other records, e.g..de-duping.func ApplyBulkFilters(records []string, filters …FilterBulk) []string { for _, f := range filters { records = f(records) } return records}These two functions take the slice of records and a slice of Filter or FilterBulk as their arguments..ApplyFilters ranges through each record in the slice and then passes that record to each filter in the set of filters..If the record passes each filter successfully, it is kept, otherwise it is discarded..ApplyBulkFilters instead ranges over the set of filters and then passes the entire slice of records to each bulk filter..Implementing these two functions with for loops rather than perhaps recursion was chosen because recursion can be expensive in a situation like this: if you iterate recursively through each filter and each record, allocations build up on each subsequent recursion and for large sets of filters or records, this can begin to cause performance issues.At this point, we have a well-defined implementation for our filtering sets, and all we need is a way to apply them in a simple way..To do that, we’ll define one more type:// FilterSet is a function that applies a set of filters and returns the filtered records..type FilterSet func([]string) []stringFilterSet will let us define various functions that can take a slice of records and return a slice of the filtered records..This lets different business logic criteria get divided up based on certain identifiers or criteria..A set of filters could also be used in-line in whatever way makes sense, but for this example we’ll divide up some contrived business logic based on an ID:var filters = map[int]FilterSet{ 1: FilterForAnimals, 2: FilterForIDs, }filters will be our set of business logic requirements, and FilterForAnimals and FilterForIDs will be sets of filters that satisfy the FilterSet type..Let’s define what those two functions do now:// FilterForAnimals applies a set of filters removing any non-animals.func FilterForAnimals(records []string) []string { return ApplyBulkFilters( ApplyFilters(records, FilterMagicalCreatures, FilterStringLength, FilterInts, FilterWords, ), FilterDuplicates, )}// FilterForIDs applies a set of filters removing any non-IDs.func FilterForIDs(records []string) []string { return ApplyBulkFilters( ApplyFilters(records, FilterIDs, ), FilterDuplicates, )}Each one of these functions applies various filters to the slice of records but with different filter requirements..In both cases, ApplyFilters is called first and then the result of that is passed to ApplyBulkFilters.Our individual filters are fairly contrived so we’ll keep the explanation to a minimum:// FilterMagicalCreatures filters out common mythical creatures.func FilterMagicalCreatures(record string) bool { magicalCreatures := []string{ "Unicorn", "Dragon", "Griffin", "Minotaur", } for _, c := range magicalCreatures { if record == c { return false } } return true}// FilterStringLength removes any long records.func FilterStringLength(record string) bool { if len(record) > 75 { return false } return true}// FilterInts removes an integers disguised as strings.func FilterInts(record string) bool { if _, err := strconv.Atoi(record); err == nil { return false } return true}// FilterWords removes any records with spaces.func FilterWords(record string) bool { split := strings.Split(record, " ") if len(split) > 1 { return false } return true}// FilterIDs removes any IDs that don't match contrived criteria.func FilterIDs(record string) bool { split := strings.Split(record, "-") if len(split) != 2 { return false } if _, err := strconv.Atoi(split[0]); err == nil { return false } if _, err := strconv.Atoi(split[1]); err == nil { return false } return true}Each filter does some sort of validation against the record and decides whether it is valid or not.. More details

Leave a Reply