dnlserrano

Monk scale

Tue, 31 Dec 2024 00:00:00 +0000

During early Ancient Greece (400s BC), whilst Hippocrates pioneered medicine and particularly skin disease categorisation, psoriasis (“psora”, for “itch”) and lepra (“lopos”, for “epidermis”) were often conflated. Much later during the Roman empire, another medical encyclopaedist by the name of Aulus Cornelius Celsus would go on to describe the skin condition and conflate it this time with impetigo, but still charaterise it by its tell tale signs to the human eye: “red skin covered with scales.”

Dr. Thomas Fitzpatrick

Some 2000 years later, Thomas Fitzpatrick was born. And with him, medicine would be changed forever. Fascinated by skin as an organ, Dr. Fitzpatrick is hailed as the father of modern dermatology. He would go on to develop PUVA treatment to combat psoriases and other skin disorders. As part of PUVA, he created what is known today as the Fitzpatrick scale of human skin phototypes. It described initially four and later six different skin tones by their level of ability to burn or tan. Put more technically, it is used to estimate the response of your skin type to UV (ultraviolet) light:

Fitzpatrick scale

Dr. Fitzpatrick went on to develop some of the early modern sunscreens and his Fitzpatrick scale would most recently be used to create diverse and inclusive emoji modifiers to depict different human skin tones (instead of the standard universal smiley face yellow), with the five modifiers being a direct mapping of it, with types I and II merged together.

Emoji modifiers based on the Fitzpatrick scale

Different forms of categorising human skin tones have existed and they have been applied as part of different scientific studies. Research about socio-economic impact of skin tone (fairness, opportunity), others around anthropology and diversity across geographic areas. All of the proposed methods seem very much flawed from the get go, though. They all depend on subjective measurements that are bottlenecked by the human ability to distinguish skin tones or colors under a specific environment (e.g., dawn, day light, evening). In that regard, nothing beats the scientific accuracy of something like spectrophotometry.

As machine learning and AI models were growing in adoption, questions around fairness and bias started to arise (2015). Early research focused on the well-established Fitzpatrick scale of six categories to stratify human skin tone, as that was at the time deemed a good enough proxy. Gender Shades (2017) and other studies surfaced consistent issues, particularly on darker skin tones. E.g., women with type VI Fitzpatrick skin tone were incorrectly gendered by AI/ML systems in almost 50% of the cases, “the chances (…) of being correctly gendered [come] close to a coin toss”.

Google’s response to the need for something better was the Monk Skin Tone Scale. In 2019, Dr. Ellis Monk partnered with Google to develop a new scale comprised of 10 different skin tones with the intention of having “an inclusive skin tone scale.”

Monk Skin Tone Scale

Data set construction, data annotation, and, ultimately, the algorithms that flow from these processes in facial recognition software, self-driving cars recogntion of their environments, automatic image classification in photos and videos (e.g., skin tone classification, gender classification, etc.), and more, are all vulnerable to biases when the standard of measurement for skin tone is not as inclusive as it could or should be.

– in The Monk Skin Tone Scale (MST)

Dr. Monk hits it right in the nail, going into detail of how models are trained and the influence of the quality of (ideally diverse) datasets on the final byproduct of such training, i.e., the model:

For example, lack of comprehensive and diverse training data sets (in terms of race and skin tone) is one leading explanation for facial recognition inaccuracies (e.g., spurious matches) and inaccuracies in automated, machine-learning driven image and video classifiers. In short, similar issues plague both pulse oximetry and machine learning and artificial intelligence (e.g., light-sensing and computer vision).

– in The Monk Skin Tone Scale (MST)

Dr. Ellis Monk

We’ve come a long way since the Gender Shades study, and nowadays transparency and anti-bias are key components of reputational cost for companies working with artificial intelligence. And unlike greenwashing opportunities for off-loading carbon to remote locations (and usually not verifiable by end users), AI bias is often times visible first-hand on user’s devices and apps we use everyday. To that extent, model cards–standard practice to “describe a model DNA”, including its performance, details on how it has been trained, what and how it should be used for–have started to include ethical considerations, including bias and geographical information, as well as fairness evaluation results. E.g., precision and recall metrics for face detection for model MediaPipe BlazeFace Short Range are reported across gender, skin tone and subregion.

At Onfido/Entrust, we’re doing something similar via our Building AI without bias whitepaper, and as we continue to ship new models to our customers. As part of our go/no-go decision to releasing a new model, we bake in several of these metrics.

Building AI without bias whitepaper by Onfido/Entrust

We shouldn’t stick to skin tone as part of bias though. As hinted at by the Gender Shades study, the Google model cards and the Onfido/Entrust report above, other examples of biases in AI/ML systems include, but are not limited to: age, gender, geography, culture, race (not to be confused with skin tone), and other non-trivial dimensions relevant to your business, e.g., low-end vs. high-end devices, mobile device operating system and platform used (e.g., iOS, Android, Web browser).

Data, data, data

Data is crucial in the process of training the best, fairer models according to recommended practices of curating balanced datasets. Meta and Google are two of the companies pioneering here and putting their money where their mouth is, by open-sourcing datasets that directly contribute to a more just landscape of AI/ML development. With their release of the MST-E dataset and the Casual Conversations v2 dataset, both of these companies are doing their part to setting a higher bar for machine learning performance when it comes to diverse representation.

An example of a past dataset available in the wild is the UTKFace dataset from 2017, maintained by the University of Tennesse, Knoxville, USA. In it the differences in balancing between different skin tones is evident when comparing the Fitzpatrick scale to the Monk scale.

UTKFace dataset

Using a script to approximate a person’s skin tone via the Monk Scale or Fitzpatrick Scale—by averaging color values to derive RGB representations with the help of the stone Python library—we quickly encounter issues with the Monk Scale, highlighting a lack of representation. For the Fitzpatrick Scale, identifying five faces per level is efficient, with the script completing in under a minute. However, with the Monk Scale, the process slows down dramatically, needing an escape hatch in the code to cap execution time at five minutes. This was tested on a standard M1 Mac with unoptimized, adapted Claude-generated code. And the escape hatch? It skips the need for finding the total of 5 required faces for skin tone 10. Exactly what you’d want to avoid in a real-world AI/ML training scenario.

Fitzpatrick vs. Monk scale samples on the UTKFace dataset

You can also see various other issues with the approach. We did not ignore black and white images, and as such there are some matches for level 9 in the Monk scale that don’t make sense. Some additional mismatches on level 2. On the other hand, we can see the same black and white issue on the Fitzpatrick scale, but happening on levels 1 and 3, instead. Some pre-processing and additional cross-validation would be required to achieve production-grade data segmentation by considering a combination of, e.g., subregion, nationality, ethnicity.

Plotting the bar charts for the ~23K data points in the UTKFace dataset for each of the skin tone scales under analysis yields the following:

Fitzpatrick vs. Monk scale bar chart on the UTKFace dataset

As you can see, Monk Scale is making even more evident the under-representation of darker skin tone individuals in this dataset.

When analsing the Casual Conversations v2 dataset from Meta, one can observe a more bell-like curve typical of a normal distribution of skin tones. In absolute numbers though, we might get the feeling arguably more needs to be done.

Fitzpatrick vs. Monk scale bar chart on the Casual Conversations v2 dataset

In the aforementioned UTKFace dataset, we had ~0.01% of representation for individuals of the darkest skin pigment. In the Casual Conversations v2 dataset that goes to ~0.14%, i.e., not much better. Is this representative of the world population? I’m not sure, but a quick Perplexity search seems to point not. Even if so, we know population-like stratification might not suffice in making algorithms fair and unbiased, since neural nets are very data hungry.

But you shouldn’t think this is the end of it and that companies are giving up. Indeed these datasets only form the basis of discussion and examples of best practices and recommendations. It is up to all of us in the AI/ML space to make the best out of pre-processing and training stages, incorporating these best practices and making sure our proprietary datasets, the ones our specific models feed off of, often times with millions more data points and terabytes (or even petabytes) of data, are well balanced. And we should make sure we have the proper guard-rails in place to make incremental releases with ever-improving levels of fairness and anti-bias.

Races have never existed in nature. They were formulated by people who had opinions and very few facts.

– Dr. Nina Jablonski

In the 70s, Dr. Fitzpatrick was studying the effects of PUVA on different human skin tones, particularly white skin. Its eurocentricity lead to constructive and well-placed criticism. Today, we continue to face data imbalance issues in dermatology research:

Machine learning will likely play a key role in the future of dermatology. Diligence and transparency are needed to prevent new health care disparities in patients with skin of color.

We’ll have to continue to adapt our data science best practices in order to guarantee we rip the benefits of the tidal wave of artificial intelligence, from financial services to transportation, law, energy, scientific research, medicine and many other verticals.

We keep pushing.

When machines could see you

Sun, 20 Oct 2024 00:00:00 +0000

In 2024, Geoffrey Hinton was awarded the Nobel Prize for his contributions to artificial intelligence. A key milestone in his work came in 2012 with AlexNet, a neural network he developed with his students Alex Krizhevsky and Ilya Sutskever, sparking a revolution in AI. But more than a decade earlier, in 2001, Paul Viola and Michael Jones first gave machines the ability to see you in real-time.

Seen in historical context, Viola-Jones is nothing short of a technological sleight of hand. While the individual pieces of its framework—Haar features, cascades, and classifiers—weren’t revolutionary on their own, the brilliance lay in how they were stitched together to outmaneuver the brutal hardware constraints of the early 2000s. It didn’t provide the level of generality of deep neural networks that were to come later, but it did something those networks couldn’t yet dream of: it unlocked real-time face detection, reshaping our interaction with machines. From cameras that automatically framed smiles to surveillance systems quietly scanning the streets, Viola-Jones wasn’t just a step forward—it was the moment machines began truly seeing us.

Faces detected by the Viola-Jones algorithm

A pivotal moment

The Viola-Jones algorithm, as it came to be known, was introduced in the seminal paper “Rapid Object Detection using a Boosted Cascade of Simple Features” at the 2001 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). A mix of academic researchers, industry professionals, and students were eagerly waiting, keenly interested in seeing how this breakthrough could push the boundaries of what computers were able to achieve. Most sitting in that conference room didn’t know at the time that they were witnessing first-hand an inflection point on machine perception. Ten years later, the Viola-Jones paper would be awarded the Longuet-Higgins Prize for most significant contributions to the Computer Vision field in that decade.

The core intuition behind Viola-Jones is that certain facial regions, like the eyes, nose, and mouth, are critical for identifying human faces. In greyscale front-facing images (assuming lighting is balanced), you’ll notice key patterns: the eye-brow region tends to be darker at the top (brows) and lighter below (eyes), while the nose bridge appears brighter compared to the adjacent cheeks.

Haar-like features type 2 and type 3

Viola-Jones takes this intuition and encodes it into Haar-like features–simple contrast-based patterns that can capture these facial structures. Reminding you of convulotional neural networks yet? In fact, the algorithm uses around 6,000 Haar features and scans these across small sub-windows (24x24 pixels) throughout the image. Each Haar feature serves as a weak learner, detecting basic patterns, and when combined into an ensemble by running a variation of AdaBoost, a boosted feature learning algorithm, they form an effective classifier.

Convolutions from later deep learning methods
Both convolutions and Haar-like features detect patterns and edges

To make the detection fast and scalable, Viola-Jones introduces a cascade of classifiers. Early stages quickly eliminate non-face regions, allowing more computational effort to focus only on promising, face-like areas. This cascading structure ensures the algorithm is both efficient and effective, making real-time face detection possible.

Cascading logic used by Viola-Jones to quickly discard non-face regions

Its versatility and efficiency established Viola-Jones as a foundational technology that significantly shaped the landscape of machine vision in everyday devices.

The human brain

The human brain identifies faces primarily through the Fusiform Face Area (FFA), a specialized region in the temporal lobe responsible for processing faces holistically. Rather than breaking a face into individual parts (like just the eyes or mouth), the FFA recognizes the entire face as a unified whole.

Highlight of the FFA region of the human brain

This holistic perception explains the Thatcher Effect, where flipping only the eyes and mouth in an upside-down face makes it look relatively normal—but when viewed upright, the distortion becomes jarringly obvious. This suggests that our brains struggle to process faces correctly when they’re misaligned because we rely on seeing them in their full, integrated form.

The Thatcher effect

To explore this ability, scientists created the Greebles test—a study that exposes participants to strange, face-like objects called Greebles. Initially, participants struggle to differentiate between Greebles, but with training, they develop a more specialized way of recognizing them, indicating that the FFA can adapt to learn new categories of visual expertise, not just human faces.

Greebles used for testing the ability of the FFA region to understand more than faces

In contrast, as revised above, Haar features, used in the Viola-Jones algorithm, take the opposite approach. They analyze faces in a piecewise manner by detecting simple patterns. This method allowed for early real-time face detection on low-power devices, but it lacked the holistic capacity seen in the FFA, making it prone to errors when encountering complex angles or lighting conditions that humans would easily overcome.

This limitation foreshadowed the need for more sophisticated methods, such as Convolutional Neural Networks (CNNs), which mimic the brain’s hierarchical way of learning. CNNs don’t rely on hand-crafted features like Haar patterns but instead learn directly from vast datasets. Early layers of CNNs detect edges and textures, similar to Haar features, but deeper layers build complex, abstract representations of objects—just as the FFA captures the entirety of a face. This transition from manually designed features to data-driven learning revolutionized computer vision, allowing machines to not only detect but also understand faces with near-human precision, paving the way for modern facial recognition systems powered by deep neural networks.

Deep neural networks revolution

In 2012, the SuperVision team, made up of 3 Canadian academic researchers submitted their contender model AlexNet to try and beat the competition on the ImageNet benchmark. Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton completely aced the test and beat the runner up by more than 10%. The formula to it? Deep neural networks trained on GPUs.

AlexNet, the neural network that changed AI forever

NVIDIA GeForce GTX 580 GPU

Another deep neural network-related Nobel laureat was announced in the same year as Geoffrey Hinton. Demis Hassabis, founder of Deepmind–developers of AlphaGo–won the Nobel prize in Chemistry for AlphaFold.

The elephant in the room of this essay is Yann LeCun and his LeNet-1 model, which goes as far back as 1989. It was never applied to face detection with the same level of success as other deep neural networks, mainly due to computational resource constraints at the time. Only in the 2010s did neural networks get applied to faces in real-time scenarios (MTCNN, BlazeFace, etc.). Still, his work and that of Yoshua Bengio were some of the most premonitory in the field, and it served to influence an entire industry.

Today Geoffrey Hinton, Yann LeCun and Yoshua Bengio are considered the godfathers of the AI revolution.

Paul Viola is currently a Distinguished Engineer at Microsoft, after having led Amazon Prime Air robotics and drone technology. Michael Jones is a Senior Principal Research Scientist at Mitsubishi Electric Research Laboratories, researching the explainability of deep neural networks.

Paul Viola and Michael Jones

Lenna and her Haar features

Balinese dancers

Wed, 22 Jun 2022 00:00:00 +0000

When Balinese dancers are performing the Ramayana monkey chant dance, they move their hands and arms percussively.

Secury Hashing Algorithms are a family of cryptographic hash functions published by NIST, a United States laboratory for the promotion of American innovation and industrial competitiveness by means of… you guessed it, establishing standards.

SHA-0 (fka “original SHA”), SHA-1, SHA-2 and SHA-3 are all a part of SHA. All of the algorithms up until SHA-3 (excluding), were designed by the NSA and all of them use the Merkle–Damgård construction method for building collision-resistant hashes (1979). SHA-3 is different, though. Read on.

The Merkle–Damgård construction used in SHA-{0,1,2} builds on one-way compression functions. It takes an initialization vector (IV), each block of the input a multiple of a fixed number (e.g., 512) and an (MD-compliant) padding. Sequentially, it will take the last result with the next block to create another result (by means of some function f). Once the final block (plus necessary padding, if any) is reached, finalisation is performed to output the final hash.

Phase outs of SHA-{0,1} started happening only long after various attacks on these SHA variants came to light, especially flaws related to collision resistance – known M1 and M2, H(M1)=H(M2), or pre-image resistance – given known hash h, hard to find H(M1)=h.

In 2007, NIST announced an open competition for the creation of a better hash function, the natural successor to SHA-{0,1} and competitor to SHA-2 (many versions of it accepted as secure still today). Five years and 64 submissions later, 5 finalists were found:

BLAKE (Aumasson et al.)
Grøstl (Knudsen et al.)
JH (Hongjun Wu)
Skein (Schneier et al.)
Keccak (Keccak team, Daemen et al.)

In 2012, it was annouced that the winner was Keccak. Keccak revealed “large security margin after significant cryptanalytic effort” and “much better throughput/area performance in hardware [than competitors]”.

And hence, Keccak became SHA-3.

SHA-3, uses a sponge construction at its core. It is split into two phases: absorbing and squeezing. The absorbing phase runs for the whole padded blocks of input, interleaving applications of the function f. Squeezing happens afterwards where bits of the resulting state are again interleaved with applications of the function f to generate an arbitrary number of output blocks.

Circling back to the title, the name “Keccak” is a variant spelling of “Kecak”, the aforementioned type of Balinese dance. The story goes that in Kecak dancers move their hands fast and it looks like they are chopping and mixing something, and this reminded one of the co-authors of the algorithm about an hash function.

The Keccak team is Michaël Peeters, Guido Bertoni, Joan Daemen, Ronny Van Keer, Gilles Van Assche and Seth Hoffert.

New York city subway

Sun, 19 Jun 2022 00:00:00 +0000

14.23.34.42.50.59.66.72

Do you know what this sequence is?

About the New York city subway and computer science data structures — a blog post on real life city navigation and abstractions for efficient search.

If you live in New York city maybe you guessed it. The sequence represents a trip on the New York city subway (or so I’m told). Stops 14, 23, etc. up until 72.

The New York city subway is a skip list: it works just like the simple efficient dynamic randomised search structure created by William Pugh (1989). When you hop into the NYC subway you may use the express line to jump over stations you know you’re not interested in stopping at.

Going from 14 to 72 you may jump through 23, 50, 59, 66 by taking the express line. Now imagine doing that for linked lists for efficient searching. Instead of traversing the whole list, just jump through some elements on the list, like a pebble ricocheting over water.

The numbered boxes (in yellow) at the bottom represent the ordered data sequence. Searching proceeds downwards from the sparsest subsequence at the top (express lines) until consecutive elements bracketing the search element are found.

Pugh proved average O(log n) complexity on both insertion and search with skipped elements being picked probabilistically, by means of randomising which nodes get promoted to an upper level, i.e., which nodes are “a part of the [multiple] express lines”.

Compared with other efficient search structures (e.g., treaps, red-black trees, b-trees), the randomized balancing of skip lists has been argued to be easier to implement than the deterministic balancing schemes used in the aforementioned data structures.

Treap

Red-black tree

B-tree

Skip lists also prove especially useful in parallel computing, where updates to different parts of the list may be done independently, hence being a great fit for designing highly scalable lock-free efficient search structures.

Lucene, Redis and Google’s LevelDB all use skip lists in their implementation.

Credit for the NYC subway analogy goes to the brilliant Erik Demaine, a professor of computer science at MIT.

Okapi BM25

Fri, 10 Jun 2022 00:00:00 +0000

Just recently learned about Okapi BM25 as a better information retrieval function for search engines (after the widely used “vanilla” TF/IDF) and it is hands down the best comp sci name I’ve come across. “Okapi BM25”, damn. Some history.

Karen Spärck Jones was a pioneering British computer scientist responsible for the concept of inverse document frequency (1972); Elasticsearch still uses TF/IDF as part of its full-text scoring algorithm as of today. it’s cheap to run and tremendously useful.

TF/IDF stands for “term frequency/inverse document frequency”. The main ideas behind it are:

A document is more relevant if the term we’re looking for appears often in that document (e.g., “quick” has a frequency of 2 in “the quick brown fox jumps over the quick dog”) but only a frequency of 1 in “the quick brown fox” (hence less relevant)
A document is more relevant if the term it is matching is not as common (e.g., documents with term “quick” will likely be more relevant than documents with more general term “the”); Another better example would be, if you have a blog about birds the word “sparrow” will be more relevant than “bird”, since it is likely many documents will mention “bird” (maybe all of them), but only one or two will mention [house] “sparrow” (cute Passer domesticus species)

The (simplified) formula as explained in Elasticsearch: The Definitive Guide is as follows:

scoring(q, d) =
∑ (
    tf(t in d)
    idf(t)²
    t.getBoost()
    norm(t,d)
) (t in q)

Essentially, we score results of a query over a set of documents by taking into account the term frequency, inverse document frequency, boosting and field-length norm.

By boosting, we can attribute more relevance to certain terms.

Field-length norm indicates relevance of term in the context of that document – the bigger the document, the less relevant it is (e.g., if “quick” appears in text “quick: a novel” it is more relevant than it appearing in bigger text “the quick brown fox jumps over the lazy dog”).

Enter Okapi BM25, the TF/IDF scoring default in Elasticsearch – whilst vanilla TF/IDF favours term frequency within a document and penalizes term prevalence across documents, Okapi BM 25 takes it further and weighs in average document size and term frequency saturation.

Average document size is important as a way to normalize field-length norm. “quick” is only more relevant in “quick: a novel” if that’s the average size for our set of documents.

Term frequency saturation is used to slow down (even set an upper bound) on term frequency relevance. “quick” is relevant if it appears twice in a document. it is more relevant if it appears 3 times. but appearing 4, 5+ times is not much more relevant between them. It’s saturated.

The name of the actual ranking function is BM25 (for “best matching 25”). The fuller name, Okapi BM25, includes the name of the first system to use it, which was the Okapi information retrieval system, implemented at London’s City University in the 1980s and 1990s.

Data Intensive maps

Wed, 04 Nov 2020 23:50:00 +0000

In Designing Data Intensive Applications, a great book by Martin Kleppmann, each chapter starts out with a fun map. It’s a distinctive and fun way to break the usual boring tone of entering a new monotonically incrementing chapter.

The book reads like a mix of a distributed systems bible and a (quite large) pocket book for the majority of your state-of-the-art solutions for designing applications for massive scalability. It lays out general concepts and then goes in deep on different specific technologies. I see myself returning to it in the future when I face some issue that I hope I’m smart enough to pattern-match with the kinds of problems at hand in the examples in the book.

When I was reading through the pages of this great book about some of the major challenges of (and solutions to) designing highly scalable and reliable systems, I was always interested in zooming in the map ahead in my Kindle. Each map has quirky bits related to the chapter you’re about to dive into. Some that stuck were “Mount Consensus” in Chapter 5, “Harsh winds of reality” in Chapter 8; “Forest of logs” in Chapter 11. The list goes on. 😂

Anyway, I went digging and Martin Kleppmann has actually wrote about how the maps came about. He explains how the Tolkien-esque style design was eventually (consistently 😬) illustrated by Shabbir Diwan, Edie Freedman and Ron Bilodeau. Its final form was inspired by ideas from readers wanting to have auxiliary flowcharts or diagrams explaining the themes in question. From there, he started to think of mind maps, geographic maps, family trees and the “wonderful map of Iceland, with Mount Hekla spewing fire”. The end result is a fun visual collection of distributed systems concepts and technologies.

You can easily find the individual maps online (Martin Kleppmann and O’Reilly were kind enough to have them up for free in high resolution). The JPEG in the cover of this blog post can be accessed (4.52 MB) here. There was even a Kickstarter campaign for printing a poster of it!

If you want to access a very unoptimised version of the same map that’s clickable to zoom on each chapter, you can do so (~15 MB) here.

Good readings! 🤓

Tensorflow Serving in Elixir

Sat, 28 Mar 2020 14:00:00 +0000

I’ve been learning more about Tensorflow Serving lately. If you are where I was at around a month ago and you’re not super familar with it, TF Serving is a “flexible, high-performance serving system for machine learning models, designed for production environments.” Follow a set of conventions (over configuration), and you can get a Tensorflow model up and running faster than ever before, powered by highly-optimised C++ code that runs blazingly fast. It is the de facto way of isolating your TF models into a specific microservice capable of serving multiple versions of your trained algorithms.

Provided you have a TF model, TF Serving allows you to expose either a REST or gRPC interface to interact with it. Given all the bespoken advantages of gRPC, I wanted to give calling TF Serving via gRPC from Elixir a try.

Serving a Tensorflow model

Since I’m more interested in Computer Vision problems, I looked for something easy to start with. Tensorflow Hub seemed like a good starting point, but I ended up going for a pre-trained handwritten digits model I found on GitHub in HDF5 (.h5), trained on the well-known MNIST dataset.

In order to serve a model via Tensorflow Serving though, we need to use the SavedModel format, “a complete TensorFlow program, including weights and computation.” Converting a Keras-saved HDF5 model to the SavedModel format is as simple as loading it and then saving it in the desired format. Once that’s done, we’re ready to run the model in TF Serving. In Python land:

import tensorflow as tf

model = tf.keras.models.load_model('cnn.h5')
tf.saved_model.save(model, 'digits/1/')

Then just run it with:

docker run -p 8501:8501 -p 8500:8500 \
  --mount type=bind,source=/path/to/digits,target=/models/digits \
  -e MODEL_NAME=digits -t tensorflow/serving &

In this case, I’m exposing both the REST and the gRPC interfaces.

If you go to http://localhost:8501/v1/models/digits/versions/1/metadata, you should see something like the following:

{
  "model_spec": {
    "name": "digits",
    "signature_name": "",
    "version": "1"
  },
  "metadata": {
    "signature_def": {
      "signature_def": {
        "__saved_model_init_op": {
          "inputs": {},
          "outputs": {
            "__saved_model_init_op": {
              "dtype": "DT_INVALID",
              "tensor_shape": {
                "dim": [],
                "unknown_rank": true
              },
              "name": "NoOp"
            }
          },
          "method_name": ""
        },
        "serving_default": {
          "inputs": {
            "conv2d_1_input": {
              "dtype": "DT_FLOAT",
              "tensor_shape": {
                "dim": [{
                    "size": "-1",
                    "name": ""
                  },
                  {
                    "size": "28",
                    "name": ""
                  },
                  {
                    "size": "28",
                    "name": ""
                  },
                  {
                    "size": "1",
                    "name": ""
                  }
                ],
                "unknown_rank": false
              },
              "name": "serving_default_conv2d_1_input:0"
            }
          },
          "outputs": {
            "dense_2": {
              "dtype": "DT_FLOAT",
              "tensor_shape": {
                "dim": [{
                    "size": "-1",
                    "name": ""
                  },
                  {
                    "size": "10",
                    "name": ""
                  }
                ],
                "unknown_rank": false
              },
              "name": "StatefulPartitionedCall:0"
            }
          },
          "method_name": "tensorflow/serving/predict"
        }
      }
    }
  }
}

This should match what you see when inspecting the model via saved_model_cli:

saved_model_cli show --dir digits/1 --tag_set serve --signature_def serving_default

The given SavedModel SignatureDef contains the following input(s):
  inputs['conv2d_1_input'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 28, 28, 1)
      name: serving_default_conv2d_1_input:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['dense_2'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 10)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict

The output describes a model that takes a 28x28x1 matrix of floats as its input tensor tagged conv2d_1_input, and it yields an output tensor of 10 floats tagged dense_2.

You might have concluded by now the model takes 28x28 black-and-white images, hence the extra 1-channel dimension for black or white (0 and 1, respectively); and it returns different confidence values for each digit from 0 to 9, hence the 10 floats output.

Tensorflow Serving from Elixir

I wanted to experiment with gRPC and the Tensorflow Serving gRPC interface specifically. It was a great coincidence that shortly before I started with this experiment, the great Andrea Leopardi published a blog post about “sharing Protobuf schemas across services.”

For the Elixir implementation of gRPC, I went with elixir-grpc/grpc by another great one from the Elixir community, Tony Han. Since he has done work on elixir-grpc/grpc, associated with the feedback I had from the aforementioned post about protobuf-elixir, choosing this project for generating Elixir code from Protobufs didn’t seem like a bad idea. And it wasn’t! More on that next.

Tensorflow and TF Serving provide great out-of-the-box APIs for Python based on the different language-agnostic Protobufs. One such case of that are the prediction APIs based on prediction_service.proto. After I experimented with it in Python, given the lower barrier for prototyping, I was confident it should be simple enough to replicate a similar interaction in Elixir (provided the gods of gRPC and HTTP/2 had worked in my favour - Tony Han, creator of grpc-elixir/grpc and Loïc Hoguin, creator of cowboy did 🙏).

I generated the Elixir code given the necessary Protobufs (under protos/):

protoc --elixir_out=plugins=grpc:./lib/messages \
  protos/tensorflow/core/framework/* \
  protos/tensorflow_serving/apis/* \
  protos/google/protobuf/* \
  -I protos

In the end, the messages/ directory contained the generated Elixir code, with nice structs I could leverage to call TF Serving from Elixir. 😊

Media manipulation in Elixir

Another challenge when doing Computer Vision with Elixir is the support for media manipulation. Resizing images, extracing frames from videos, etc. can either be done by shelling out to applications such as ImageMagick or FFmpeg, or by making use of (dirty) NIFs, Ports, port drivers or “equivalent alternatives” (out of scope of this blog post). TL;DR is I ended up using a mixture of both, because I was being hacky.

For resizing my images to 28x28 1-channel black-and-white thumbnails, I ended up using ImageMagick’s convert. ✨ Who would have said there were so many dithering algorithms out there?

{_, 0} = System.cmd(
  "convert",
  ~w(#{input_file} -resize 28x28^ -dither FloydSteinberg -set background white
  -flatten -colors 2 -gravity center -extent 28x28 #{output_file})
)

To translate the 1-channel pixels to arrays of 0, 1 values, I used Arjan Scherpenisse’s pixels library (which I got to know in his awesome talk about Nerves). I’m sure I could have been smarter here (piping the resulting image somehow to the running Elixir process and avoiding having to use this NIF), but again… I was being scrappy. And pixels is awesome, by the way. It integrates with both lodepng and ujpeg. Thanks Arjan!

{:ok, %{data: pixels_binary}} = Pixels.read_file(output_file)
pixels = :erlang.binary_to_list(pixels_binary)

input =
  pixels
  |> Enum.chunk_every(4)
  |> Enum.map(fn [r, g, b, _] ->
    case (r + g + b) / 3 > 128 do
      true -> 0
      _ -> 1
    end
  end)

Putting it all together

Having all of these bits and pieces in place, we’re left with sending the actual input to TF Serving and making sense of what it returns back. At this point we have the input (or kind of) and we have the Elixir structs to talk gRPC with TF Serving.

Building the request entails defining a PredictRequest containing inputs and a model specification (model_spec). As previously mentioned, the name of the input is in this case "conv2d_1_input" and it is of type float (:DT_FLOAT). The input is a 28x28x1 tensor with arbitrary number of batches (we use a batch of 1 image, but you could evaluate more in one single network call). The value itself, is passed as a vector of size 784 (28 * 28), named input in the code that follows:

request =
  Tensorflow.Serving.PredictRequest.new(
    inputs: %{
      "conv2d_1_input" => Tensorflow.TensorProto.new(
        dtype: :DT_FLOAT,
        tensor_shape: Tensorflow.TensorShapeProto.new(
          dim: [
            Tensorflow.TensorShapeProto.Dim.new(size: 1),
            Tensorflow.TensorShapeProto.Dim.new(size: 28),
            Tensorflow.TensorShapeProto.Dim.new(size: 28),
            Tensorflow.TensorShapeProto.Dim.new(size: 1),
          ],
          unknown_rank: false
        ),
        float_val: input
      ),
    },
    model_spec: Tensorflow.Serving.ModelSpec.new(name: "digits")
  )

We then make the request and return back the guessed digit, for which the ML algorithm yielded the highest score:

{:ok, channel} = GRPC.Stub.connect("#{tf_serving_host()}:8500")
{:ok, %Tensorflow.Serving.PredictResponse{outputs: outputs}} =
  Tensorflow.Serving.PredictionService.Stub.predict(channel, request)

digit =
  outputs["dense_2"].float_val
  |> Enum.with_index()
  |> Enum.max_by(fn {val, _} -> val end)
  |> elem(1)

Try it out below!

Browser not compatible.

What’s happening here is we’re providing a UI where the user can draw a digit, we then convert the content in the Canvas element to a Base64-encoded image that is sent to the backend. There, we resize it and apply it the need transformations before we send it to the TF Serving model. I then Dockerized it and docker-composed it. 👨‍🍳

Find the whole code over at my GitHub.

Wrapping up

And that’s it. Elixir calling via gRPC a Machine Learning model deployed using Tensorflow Serving. It’s possible, today!

Sure, there are some rough edges. The support for gRPC isn’t 100% mature in elixir-grpc/grpc (I had to point to master branches for some of its dependencies in order to get the latest fixes for existing bugs). The “media manipulation ecosystem” in Elixir could also be better, but you can already go a long way (particularly with dirty NIFs). And if we push a bit, we can make it even better! ❤️

Happy Tensorflow Serving… in Elixir! 🤖

Behaviours and
Dynamic dispatch

Sat, 21 Dec 2019 15:43:22 +0000

This is going to be a quick one.

Recently I had the opportunity to take part in the refactoring of some Elixir code.

It looked something like this. Picture more interesting (and complex) logic in place of the classic, old and honestly a bit boring beverages example:

defmodule Beverages do
  def prepare(beverage_type) do
    with :ok <- roll_up_sleeves(),
         {:ok, quantity} <- random_quantity(),
         {:ok, preparation} <- prepare_it(beverage_type, quantity),
         {:ok, _beverage} <- serve_it(preparation) do
      {:ok, "happy days!"}
    else
      error -> {:error, error}
    end
  end

  defp random_quantity, do: {:ok, Enum.random(30..40)}

  defp roll_up_sleeves do
    "rolling up sleeves to start working" |> IO.puts()
  end

  defp prepare_it(:coffee, quantity) do
    "getting #{quantity}gr of beautiful Peruvian coffee beans" |> IO.puts()
    {:ok, "prepped with the finest coffee ☕️"}
  end

  defp prepare_it(:tea, quantity) do
    "getting #{quantity}gr of mixed Indian Assam leaves, brisk Ceylon and bright Kenyan teas"
    |> IO.puts()

    {:ok, "prepped with the finest teas 🍵"}
  end

  defp serve_it(preparation) do
    served = "ready to drink beverage " <> preparation
    served |> IO.puts()
    {:ok, served}
  end
end

We wanted to move away from pattern-matching on atoms (first function argument) as a way of forking to different implementations in the same module. Single responsibility principle anyone?

Our goal was to split the logic in various different modules, given we would now need to add another beverage type: the lemonade. The logic inside each was also more complex than in this toy example.

This split would make it easier to reason about each one of the beverage types and the way they differ from each other, while hopefully still keeping the comon logic in a central place that privileges dryness. Furthermore, having each of them as separate modules made it simpler to test them (each one in isolation) and also to extend them (both now given the new business requirements, and in the future in case yet another beverage comes along).

Besides all this, we were also not following a specific way of properly documenting the fact that each of the specific beverage implementations needed to respect a well-defined interface. Smells like ~~teen spirit~~ Elixir behaviours.

After doing some quick research over the Elixir hexdocs and official pages, we eventually hit gold when we came across the recommended way of doing “dynamic dispatching” in Elixir.

So we changed it to this:

defmodule NewBeverages do
  @type quantity :: integer()
  @type preparation :: binary()
  @callback prepare_it(quantity) :: {:ok, preparation}

  def prepare(beverage_type) do
    beverage_impl = beverage_for(beverage_type)

    with :ok <- roll_up_sleeves(),
         {:ok, quantity} <- random_quantity(),
         {:ok, preparation} <- beverage_impl.prepare_it(quantity),
         {:ok, _beverage} <- serve_it(preparation) do
      {:ok, "happy days!"}
    else
      error -> {:error, error}
    end
  end

  defp beverage_for(:coffee), do: NewBeverages.Coffee
  defp beverage_for(:tea), do: NewBeverages.Tea
  defp beverage_for(:lemonade), do: NewBeverages.Lemonade

  defp random_quantity, do: {:ok, Enum.random(30..40)}

  defp roll_up_sleeves do
    "rolling up sleeves to start working" |> IO.puts()
  end

  defp serve_it(preparation) do
    served = "ready to drink beverage " <> preparation
    served |> IO.puts()
    {:ok, served}
  end
end

defmodule NewBeverages.Coffee do
  @behaviour NewBeverages

  def prepare_it(quantity) do
    "getting #{quantity}gr of beautiful Peruvian coffee beans" |> IO.puts()
    {:ok, "prepped with the finest coffee ☕️"}
  end
end

defmodule NewBeverages.Tea do
  @behaviour NewBeverages

  def prepare_it(quantity) do
    "getting #{quantity}gr of mixed Indian Assam leaves, brisk Ceylon and bright Kenyan teas"
    |> IO.puts()

    {:ok, "prepped with the finest teas 🍵"}
  end
end

defmodule NewBeverages.Lemonade do
  @behaviour NewBeverages

  def prepare_it(quantity) do
    "getting #{quantity}ml of hearty lemon juice" |> IO.puts()
    {:ok, "prepped with the finest lemon 🍋"}
  end
end

We making lemonade, boy.

> NewBeverages.prepare(:lemonade)
rolling up sleeves to start working
getting 32ml of hearty lemon juice
ready to drink beverage prepped with the finest lemon 🍋
{:ok, "happy days!"}

You can easily extend lemonade to do crazy things other beverages don’t need to do as part of preparing your beverage (prepare_it/1) without polluting the “orchestrator module”. Buy one of those nice looking metal squeezers? Looking at you Rodrigo.

You can take this further, and do something like the Template method pattern:

def prepare_beverage(beverage_type) do
  beverage_impl = beverage_for(beverage_type)

  with :ok <- roll_up_sleeves(),
       {:ok, quantity} <- random_quantity(),
       {:ok, preparation} <- beverage_impl.prepare_it(quantity),
       {:ok, beverage_with_side} <- beverage_impl.add_side(preparation),
       {:ok, _beverage} <- serve_it(beverage_with_side) do
    {:ok, "happy days!"}
  else
    error -> {:error, error}
  end
end

In this case, the template defines two very general actions: randomising the quantity used to prepare your beverage (random_quantity/0) and serving the drink (serve_it/1). Actually preparing it (prepare_it/1): beans for coffee, leaves for tea, etc., and getting a special side for each of these (add_side/1): chocolate brownie for coffee, butter cookie for tea, etc. is different for each, so each specific implementation should do its own thing.

Functional purists may be crying right now, so I’ll stop. But you get the gist of it. This new design seems to, for now, have put us in a much better place. Leveraging language features such as Protocols and Behaviours has truly helped us maintain and improve our code for the long run.

Code available here.

Note: Thanks to Dino Costa for proposing the usage of a “factory” function (e.g., beverage_for/1). Also, thanks to him, Daniel Caixinha, and Pedro Madeira for validating this design.

Merry Xmas! 🎄🎅

Learning Rust

Mon, 11 Nov 2019 08:44:00 +0000

I’ve been learning Rust lately. I have also got back to playing futsal with some friends. Weirdly enough, these two worlds intersected a few weeks ago. This is a summary of my experience learning Rust. Starting with reading about it here and there, eventually getting the book, doing a Hackathon project at Onfido using it and then spinning up my own personal project as a way to further experiment with the language.

At this point, Rust is a pretty well-known programming language. I remember one of the first blog posts that really caught my attention to the language appeared expectedly (given Rust started at Mozilla) in the Mozilla Hacks blog. In this blog post, Lin Clark introduced Project Quantum. Rust was said to be “a language that was free of […] data races”, and Servo (the browser engine developed in Rust at Mozilla) “made full use of […] fine-grained parallelism”.

Ever since that “first” blog post, lots of articles have been popping up more and more on popular aggregators like Hacker News and Lobsters. And everytime, I would get that FOMO feeling. The tipping point came about when learning about more and more developers in the Elixir community which were looking at using Rust as a way to squeeze the most performance out of their resources. This was true for Bleacher Report, Discord and more recently SimpleBet.

Starting out

To start out, I got the official book for “The Rust Programming Language” and started reading it at the end of my summer vacations. I stopped for a while, and started thinking about ways I could actually practice what the book was preaching. My initial idea was to implement the Skip List data structure in Rust. It’s one of the most fun data structures around so I was excited about reading the paper that originally introduced it and about the possiblity of implementing it in Rust, obviously!

So I set out to do so. I installed rustup, started playing with cargo to generate my “Hello, world!” project and moved on to a bit of online tutorial reading about idiomatically writing singly linked lists in Rust. My co-worker Tom Forbes, who had also been reading about Rust, warned me it might not be an easy task for a first project, given this sort of data structure implementation usually comes with lots of reference handling and non-trivial ownership transfers as a consequence of Rust’s ownership rules and borrow checking.

Around the time this was happening, we had a Hackathon at Onfido. My colleague Daniel Caixinha suggested we tried to rewrite one of our machine learning services in Rust, as a way to experiment with the language. He got me and Tom excited, and soon we were a team of 8 trying to actually re-write two microservices instead of just one. Having that much people on board was great. We got to feel different pain points for people more used to one or more of VM-backed, garbage collected and dynamically typed languages. I definitely felt it too. In the end, we completed the re-write of one of the projects, which allowed us to experiment with Rocket, Serde and the Rust bindings for Tensorflow. It was a great feeling, to have a working version of something after only 2 days of hacking, when the majority of people had not yet programmed a single line of Rust.

After the Hackathon, I got back to the singly linked lists. I was having to dwelve into Box, lifetimes and lots of… I had been warned… ownership transfers. While it made sense, given the language’s constraints optimise for safety (while trying to not compromise on speed), it didn’t feel right.

But I found hope.

One of my personal Rust heroes is Andrew Gallant. He’s the maintainer of a number of cool projects like ripgrep, xsv and rust-lang/regex. These are projects known for their performance. As good candidates for role-model projects, I went looking around the code. It looked so much cleaner than the stuff I was having to do to make a singly linked list work. 😭 Was I looking at idiomatic Rust code? Error handling with the Result type, structs with well-defined fields and mostly primitive or user-defined types, almost never needing to explicitly set lifetimes (thanks to the lifetime elision rules).

Having fun

Shortly after this, my co-worker Tom kept working on his personal project, and he blogged about the sense of fun one should thrive for when learning a new programming language.

While skip lists are fun (I highly recommend you reading the paper about them, and particularly understanding the trade-offs when comparing to other data structures), it didn’t feel like the right project to continue pursuing in light of the conflicting goals I had: learning the language, not getting sucked into premature optimisation details, and having fun!

One way I was having fun outside of programming, was by playing five-a-side football with a group of friends. We usually play at Estádio Universitário de Lisboa and the way I book the football or futsal pitch is by filling in an online form that sends an e-mail with my booking details. Everytime, wherever I am, I need to fill in the same details over and over again. Other than the day, everything is exactly the same: my name, fiscal number, phone number, e-mail, booking hours, preferred pitch, etc. One way to automate this would be to get a template of the absolutely necessary information for a booking, and have a way to submit that info without needing to go about writing it all down in the form ourselves.

I settled on inspecting the POST request. I now had a list of query string parameters where the information for the booking lived. I had the endpoint, and some headers and fields I wasn’t sure about. But that was enough to get going. I put my Rust belt on. Created some structs. Springled some serde magic on top of them for easy serialisation to various different outputs (e.g., JSON, query string, YAML, etc.).

At this point, I was ready to hard-code some values to fill in the structs, but I didn’t have a way to perform an actual request yet. And so, ureq came to the resque. When I started using it, async-await hadn’t landed in stable Rust yet, and the ureq library provided an intuitive sync approach to HTTP request/response handling. It was also more than enough for what I was trying to achieve. Or so I thought…

Because std::net::ToSocketAddrs used by ureq might return IPv4 or IPv6 addresses, and because the website we try to contact is not fully IPv6 ready, I needed to hack a way to set a preferred IP version in order to prioritise IPv4 on demand. You can find that here.

When doing the first couple of live requests (I was using a sinatra Ruby mock server in the beginning), I couldn’t really get a successful e-mail back, although I was seeing 200 responses from the server. Apparently, the form just kept returning 200 even when it didn’t actually accept the request. After tinkering a bit with it, I understood this was actually a Drupal form that relied on two “special” fields: form_id and form_build_id, which were fetched from the DOM of the retrieved form web page. This helps prevent basic replay attacks as well as CSRF. So now I needed to scrape the retrieved web page.

For scraping, I resorted again to a well-supported library in Rust. scraper did the trick here. It has a nice API for selecting DOM nodes and searching in them. This was all I needed to fetch the form_id and form_build_id from the DOM. Cool. Now I could make an actual request. After a few submissions with invalid input (e.g., phone numbers can’t have spaces in them), I got it working as expected.

I brushed it up with a nice looking CLI with two distinct forms of input. You can either use a list of option arguments for each field to send as part of the request, or use a YAML configuration file with the necessary form data. For YAML deserialisation, serde was used again, via serde_yaml. CLI functionality was achieved using the well-known clap Rust library, which even allows you to use a YAML file to describe your CLI options.

I called it futbladaj.

There are still lots of improvements to make. If I think I’ll keep having fun doing those, I might get back to this project. Otherwise, I’ll just move on to something else. It was fun and it served its purpose well. Rust is nice. It is fun. It’s also rigid but with that rigidity comes power. Power to not fail miserably in production. Power to perform. If not in Rust, at least in futsal!

Mutation testing in Elixir

Mon, 22 Jul 2019 12:19:00 +0000

I first heard about mutation testing when attending a Ruby meetup here in Lisbon. At the time Tadas presentation focused on mutant, the Ruby library for mutation testing by mbj.

It sounded fun, but not very practical. I think that was because I didn’t understand the full scope, goals and philosophy, so to speak, of mutation testing. This post will help explore mutation testing and why Elixir is a good candidate to give it a shot. I will then introduce a library I’ve been working on exavier which aims to provide a full-blown mutation testing framework for Elixir.

It’s still very early days, it may be haunted by major design flaws, stupid ideas and this blog post also serves a bit as a call for help. If you’re out there reading and you think mutation testing might be your thing and you want to help, hit me up @dnlserrano or open a PR.

What is mutation testing?

Let’s take a step back and discuss what exactly all of this is. Mutation testing is a software testing technique to ensure quality in your tests whereby one runs the same set of unit tests multiple times, making different small changes (mutations) to the source code under test each time. The intuition behind it is easy: if the same green tests you had continue to be green after you’ve applied the mutations on the original source code, then something must be off.

Consider the following example of a simple even?/1 function that checks if a given number is even:

defmodule HelloWorld do
  def even?(x), do: rem(x, 2) == 0
end

We could think of adding this simple test:

defmodule HelloWorldTest do
  use ExUnit.Case, async: true

  @subject HelloWorld

  test "when even" do
    assert @subject.even?(2) == true
  end
end

With this, we’re testing a basic case of an even number. Good, right?

Imagine we now mutate the original code to the following:

defmodule HelloWorld do
  def even?(x), do: rem(x, 2) >= 0 # == replaced by >=
end

Our test will still be green given rem(2, 2) yields 0, which is of course still greater than or equal to (>=) 0.

This is what we refer to as a “live mutant”. Code is changed, but the test still passes. For each test you run, which will contain a given mutation, you register if it passed (generated a live mutant) or failed (mutant is successfully killed). This will be important for actually using mutation testing in practice, which I will address later on in this post.

So, we changed our source code, and the test still passes. This says something about our tests. In this case in particular, it tells us we’re not considering a scenario in which, the remainder of 2 being greater than 0, the return of the function should be false rather than true. If you think about what it means for a second, turns out we forgot to test the “odd cases”:

defmodule HelloWorldTest do
  use ExUnit.Case, async: true

  @subject HelloWorld

  test "when even" do
    assert @subject.even?(2) == true
  end

  test "when odd" do
    assert @subject.even?(3) == false
  end
end

Now one of the tests would still pass, but we’d have a new one that would fail. Summing it up, we have:

1 live mutant
- test “when even”, still passed after mutation
1 killed mutant
- test “when odd”, failed after mutation

It means we had a killed mutant rate of 1/2, which translates to 50% “mutation coverage.”

Using mutation testing

Using mutation testing “in the wild” can translate into running it as part of your CI pipeline on every merge to master. Each time you do that, your unit tests will run. If they pass, you can run a mutation suite and verify that your mutation coverage is above a specified threshold (e.g., 70%). Other options may ideally be passed to the mutation testing framework you’re using so you can further tweak your mutation test suite. Such configurations might involve excluding certain files from mutation testing, only running it for specified folders, modifying the default threshold value, etc.¹

exavier

I’ve implemented the ideas of mutation testing into an Elixir library I called exavier. Its goal is to provide a full-blown mutation testing framework (think ex_unit for mutation testing with much worse performance and code quality at this point in time). You can run a set of mutation tests by doing:

mix exavier.test

The output would be something like the following:

......................

(...)

16) test when infinity (Elixir.HelloWorldTest)
  - if(y == :special) do
  -   :yes
  - else
  -   :no
  - end

  + if(true) do
  +   :yes
  + else
  +   :no
  + end

/Users/dnlserrano/Repos/exavier/test/hello_world_test.exs:10


22 tests, 6 failed (mutants killed), 16 passed (mutants survived)
27.27% mutation coverage

It reports each mutant that survived, alongside the mutation(s) made and shows an aggregate result of your mutation coverage.

Mutators

Mutators specify ways in which we can mutate the code. Currently we have 5 proof-of-concept mutators available in exavier. It should be easy to extend the library with other mutators or even have the possibility (soon) to run your own custom mutators that live only inside your specific code-base, if that’s more your thing.

Each mutator works by manipulating the AST of Elixir code, and that’s part of what makes Elixir such a great fit for trying to develop such a testing framework in this language. Having so many powerful tools around code and the structure of that code in the language, it seemed like an amazing opportunity that shouldn’t be missed. That and the fact it was fun, of course.

The library still needs a lot of work, particularly around adding more mutators and configurations. The major points I’d like to see tackled in the short-time are available in the “To be done” section of the project’s README.

I’d love to have some feedback about how you’re using exavier and what we can, hopefully together as a community, do to make it better.

Happy mutations! 🖖

¹ A very well-known issue in mutation testing is that of Equivalent Mutations, which is out of the scope of this blog post.