Today I will dive into a recent experience I had when playing with erlando – an Erlang library that adds a set of syntax extensions to the language. In particular, it adds a more-less Haskell’s do-notation equivalent in Erlang.

With the purpose of re-iterating the message from Don’t make all defaults Dogmas, I’m here to show another computer science abstraction that goes beyond a particular community or programming language ecosystem: monads. Despite its negative popularity (a lot of the time undeserved), monads in some degree and/or scope can heavily improve your code’s maintanability and readability, assuming everyone involved and any future hires will know how to deal with them.

A great sign that an abstraction is not exclusive to a particular implementation or ecosystem is if it can be represented without any type of programming jargon; a mere drawing does the job just fine – sometimes even simple enough that people outside of our programming or mathematical bubbles can understand. And I do believe this is the case for monads; you can find hints of them in many places. Hopefully at the end of this post, you will agree that its idea is useful regardless of any specific implementation, and however many problems the implementations introduce.

Case Menace

Our story begins with me unsatisfied with particular parts of a project of mine. The project is Lyceum – an MMO I’ve been developing with close friends. In there, we use a PGSQL library for Erlang, epgsql, to interact with our database instance. A lot of the time we want to do multiple queries into the database and compose our payload and dispatch it to the client; sending it only if everything did not fail. An initial naive approach would be to just:

  character_map(Pid, MapName, Credentials, Connection) ->
      Result = case database:retrieve_character(Credentials, Connection) of
		   {ok, Character} ->
		       case database:retrieve_map(MapName, Connection) of
			   {ok, Map} -> {ok, {Character, Map}};
			   {error, Reason} -> {error, Reason}
		       end;
		   {error, Reason} -> {error, Reason}
	       end,
      Pid ! Result.

Assuming our client application provided some credentials and we got a connection to our server database, above we are getting the character for that particular user and, if everything went ok, we do another read in the database to retrieve the character’s map information, in order to render it on the client later. If the first query fails, we just short-circuit the entire thing and go back to the client with an error explaining why we failed.

After doing that type of code enough times (or if you are used to that type of situation), a pattern starts to emerge. Many tutorials about monads show images or diagrams to illustrate short-circuit behavior. And the intuition is simple for that particular monad: we proceed as we keep hitting ok, and if we find error the entire thing stops and we just bubble that up to whoever called the entire process.

Attack of The Functions

When I realized this, I felt sad. At the time, I was not aware of any better way to handle this problem than to make auxiliary functions to help me alleviate the burden – at least at first glance. First up, I’ve made process_postgres_result:

  process_postgres_result({ok, FullColumns, Values}, select, Fun) ->
      Fun(FullColumns, Values);
  process_postgres_result({ok, Count}, update, Fun) ->
      Fun(Count);
  process_postgres_result({ok, Count}, delete, Fun) ->
      Fun(Count);
  process_postgres_result({ok, Count}, insert, Fun) ->
      Fun(Count);
  process_postgres_result({ok, _, _, _}, insert, _) ->
      {error, "Unexpected use of Insert on Server side"};
  process_postgres_result({error, Error}, Tag, _) ->
      io:format("Tag: ~p\nError: ~p\n", [Error, Tag]),
      {error, "Unexpected error (operation or PSQL) on Server side"}.

The goal of this function was to better handle internal uses of epgsql. The library changes its output depending on the kind of query you are executing, e.g., selects, updates, deletes, inserts, etc. So the idea would be that you call the library with your query and immediately call this process result function informing which type of query you did and what type of treatment you want to apply with its result (represented by Fun in the snippet above). Here’s an example of using this function:

  check_user(Username, Password, Connection) ->
      Query =
	  "SELECT * FROM player.record WHERE username = $1::VARCHAR(32) "
	  "AND password = $2::TEXT",
      Result = epgsql:equery(Connection, Query, [Username, Password]),
      Fun = fun(FullColumns, Values) ->
		    case database:columns_and_rows(FullColumns, Values) of
			[] -> {error, "Could not find User"};
			[UserData | _] -> {ok, maps:get(e_mail, UserData)}
		    end
	    end,
      database:process_postgres_result(Result, select, Fun).

In order to check if a user attempting to login has valid credentials, we need to performa a SQL SELECT. We first make the SQL query with holes for data. Then, we call our library providing the data to fill the placeholders. Next is the treatment function: what should happen to the data if we happen to succeed with the query? Finally, we call process_postgres_result accordingly. Notice here that the possible results of our lambda Fun are familiar: they are the same as we saw in the previous section. And, as we make more of those queries, we plan to follow a pattern to have reasonable expectations when calling those functions in the main program, i.e., we shall expect that all of them will respect ok with the correct value or error with the reason for failure. There may be a few exceptions, but the rule should be clear.

Revenge of The Monad

Our function process_postgres_result allows us to more conveniently get our values; those that remind us of Haskell’s Either or F#’s Result. Now it comes the question: how are we suppose to combine them? Initially, my naive previous self thought we could improve things a bit by starting to use psql_bind:

  psql_bind(monadicValue, []) ->
      monadicValue;
  psql_bind(ok, _) ->
      ok;
  psql_bind({ok, Result}, [Fun | Tail]) ->
      psql_bind(Fun(Result), Tail);
  psql_bind({error, _} = Error, _) ->
      Error;
  psql_bind(_, _) ->
      {error, "Wrong monadic value in the chain"}.

The purpose of this function is to allow us to pass a list of functions to keep processing values that are being called “monadic”. Hence, psql_bind would unwrap those values for us and pipe it to the next available function or stop immediately if an error occurred. Given that we plan to use this with our previous function, process_postgres_result, these two are suppose to have some chemistry together. Sadly, I was not satisfied with the end result. Here’s one example of using it:

  character_map(Pid, MapName, Credentials, Connection) ->
      Result =
	  database:psql_bind(
	    database:retrieve_character(Credentials, Connection),
	    [fun(Character) ->
		     database:psql_bind(
		       database:retrieve_map(MapName, Connection),
		       [fun(Map) -> {ok, {Character, Map}} end])
	     end]),
      Pid ! Result.

Underwhelming, isn’t it? No matter how hard we try, the thing still looks convoluted and hardly readable. And let me tell you, it gets way worse as we progress:

  mess(..., Connection) ->
      database:psql_bind(
	database:process_postgres_result(Dimensions, select, FunDimensions),
	[fun(ListDimensionsMap) ->
		 case ListDimensionsMap of
		     [Map] ->
			 Width = maps:get(width, Map),
			 Height = maps:get(height, Map),
			 {ok, {Width, Height}};
		     _ ->
			 io:format("[ERROR] Something to wrong when getting map dimensions!\n"),
			 exit(1)
		 end
	 end,
	 fun({Width, Height}) ->
		 database:psql_bind(
		   database:process_postgres_result(Tiles, select, FunTiles),
		   [fun(TilesV) ->
			    database:psql_bind(
			      database:process_postgres_result(Objects, select, FunObjects),
			      [fun(ObjectsV) ->
				       Quantity = Width * Height,
				       if (length(TilesV) == Quantity) and (length(ObjectsV) == Quantity) ->
					       {ok,
						#{tiles => TilesV,
						  objects => ObjectsV,
						  width => Width,
						  height => Height}};
					  (length(TilesV) == 0) or (length(ObjectsV) == 0) ->
					       {error, "Map can't be instantiated!"};
					  true -> {error, "Mismatch between dimensions, tiles and objects!"}
				       end
			       end])
		    end])
	 end]).

I will not attempt to explain what happening in the code above, but just skimming it feels terrible! The nesting makes it way worse and even more complicated. Then, it begs the question: what is the cause of this? Is it us trying to use the wrong abstraction? Are monads that evil? Are we just jamming it into our program to feel some empty pride about ourselves because we are using a fancy thing that most nerds don’t know about?

I don’t think so, at all, actually. The reason we got into this situation is not because the abstraction is not expressing what we want, but rather that the host language, Erlang in this case, makes it terse for us to express the idea that fits our problem’s description. We are quite literally fighting its syntax and there are consequences.

Now, if it is the case that this is an unavoidable problem and the end of the road, we shall consider dropping the entire idea and going back to the drawing board. It is not because our first idea didn’t work out that there is no better solution to this problem. It is part of intellectual humility to recognize we made the wrong choice; regardless if we like the idea and find it cool most of the time. If it does not fit, it doesn’t. The arrogant decision to keep pursuing the idea knowing for a fact it can’t be done in a way that it is worth it can have huge and devastating consequences for any business – it may even be the main cause of its own destruction.

This, however, is not a fact for us in this particular use-case.

A New Hope

When sharing about Lyceum in Hacker News, I did complain about this problem. A fellow Erlang developer or enthusiast came to save me. He mentioned in the comment that you can actually make new syntax in Erlang, listing erlando as one example. Specifically, the library solves this problem that I was having; I want to have nicer syntax to express PGSQL queries in a monadic way. After some digging, I’ve found a fork of the original library as a package in hex, something that I can use in Erlang.

And when looking at the README of erlando, this is the first thing I read:

  do([monad ||
      A <- foo(),
      B <- bar(A, dog),
      ok]).

Are you telling me that I can not only make a monad, but also that there is a dedicated syntax erlando provides me to nicely chain operations together in a sequence? Sounds too good to be true. And let me share with you the good news: it is true.

The Bind Strikes Back

Erlando provides 3 particular syntax extensions for Erlang, one of which is a Haskell-like do-notation. Further, it provides some common monads that you usually want to have around:

• error_m (Haskell’s Either or F#’s Result)
• identity_m (Haskell’s Identity)
• list_m (Haskell’s List)
• maybe_m (Haskell’s Maybe, F#’s Option, OCaml’s Option, Rust’s Option)

The idea is that you chain an operation in the same fashion we’ve been desiring it for so long:

  if_safe_div_zero(X, Y, Fun) ->
      do([maybe_m ||
	  Result <- case Y == 0 of
			true  -> fail("Cannot divide by zero");
			false -> return(X / Y)
		    end,
	  return(Fun(Result))]).

One may say that there is no need for all of this just to check a simple division by zero. A case would suffice. I agree, but we can’t diminish the potential of this new added syntax:

  write_file(Path, Data, Modes) ->
      Modes1 = [binary, write | (Modes -- [binary, write])],
      do([error_m ||
	     Bin <- make_binary(Data),
	     Hdl <- file:open(Path, Modes1),
	     Result <- return(do([error_m ||
				  file:write(Hdl, Bin),
				  file:sync(Hdl)])),
	     file:close(Hdl),
	     Result]).

We are making a series of IO operations and if any of them fail we just finish our party – exactly the behavior that we want for our PGSQL operations. The final piece of the puzzle is to understand how can we get this power for our custom problem. How to make it interact with epgsql? Can erlando’s do-notation be combined with it somehow? How to get there?

Return of the Do

The answer to this quest is the ability to make a custom monad. Fortunately, this is something supported by erlando. Hence, behold postgres_m! Our custom monad can now give another flavor to our registry check function:

  check_user(Username, Password, Connection) ->
    Query =
	"SELECT * FROM player.record WHERE username = $1::VARCHAR(32) "
	"AND password = $2::TEXT",
    do([postgres_m ||
	   UnprocessedUser <- {epgsql:equery(Connection, Query, [Username, Password]), select},
	   case database_utils:columns_and_rows(UnprocessedUser) of
	       [] -> fail("Could not find User");
	       [UserData | _] -> return(maps:get(e_mail, UserData))
	   end]).

The gains on this function are not that incredible; but at least it looks nicer in my opinion. The flow of data can be more easily understood and the nesting of operations is under control. On the contrary, our previous character_map and mess functions got great to immeasurable gains:

  character_map(Pid, MapName, Credentials, Connection) ->
      Result = do([error_m ||
		      Character <- database:retrieve_character(Credentials, Connection),
		      Map <- database:retrieve_map(MapName, Connection),
		      return({Character, Map})]),
      Pid ! Result.

  mess(..., Connection) ->
      do([postgres_m ||
	     UnprocessedMap <- {Dimensions, select},
	     {ok, {Width, Height}} = check_dimensions(UnprocessedMap),
	     UnprocessedTiles <- {Tiles, select},
	     ProcessedTiles = lists:map(fun transform_tile/1, database_utils:columns_and_rows(UnprocessedTiles)),
	     UnprocessedObjects <- {Objects, select},
	     ProcessedObjects = lists:map(fun transform_object/1, database_utils:columns_and_rows(UnprocessedObjects)),
	     Quantity = Width * Height,
	     if (length(ProcessedTiles) == Quantity) and (length(ProcessedObjects) == Quantity) ->
		     return(#{tiles => ProcessedTiles,
			      objects => ProcessedObjects,
			      width => Width,
			      height => Height});
		(length(ProcessedTiles) == 0) or (length(ProcessedObjects) == 0) ->
		     fail("Map can't be instantiated!");
		true -> fail("Mismatch between dimensions, tiles and objects!")
	     end]).

The Trade-Off Awakens

Curious readers may go to the source code and undercover the secrets behind these transformations. Let me tell you upfront that non-ideal code was necessary to make this happen given Erlang’s constraints and how erlando works.

The implementation of the postgres_m monad is rather unsafe and totally exploits the fact that this is a parser extension and that this is a dynamically typed language. The final result is a monad that can blow up and definitely should trigger most Haskellers out there. From a Haskell perspective, we can’t say that the implementation of monads in Erlang using erlando is feature-equivalent with Haskell’s – a judgement that I share.

Achieving perfection, however, was never my intention. My goal was to demonstrate how an abstraction can fullfil requirements even if some annoyance appears and make its use inconvenient. The understanding of the problem and the abstraction (monads in this case) was the key factor for me to pursue with this solution, regardless of how ugly it was. Fortunately, in this case, there was a way to get around the barrier of use (erlando saved our day) and now using the chosen abstraction does not culminate in a heavy trade-off on readability.

Even more attentive Erlang enthusiasts may point out that the original version of erlando supported monad transformers – something that is absent from Lyceum at the moment of writing this post. Usually, this is one of the main complaints when using monads: their composition via transformers is for sure non-ideal, to say the least. Solutions to this problem ended up producing RIO and effect systems.

The counter to this concern is twofold: this is not Haskell and we don’t have to use transformers. Because we are using Erlang, the idea that we are “locked in a monad and need to go back and forth using lifts” is a non-issue, something that, for better or for worse, is due to the way we are achiving monads in the code (parser extension + dynamic typing). Hence, this allows us to use do-notation only where it makes a huge different, keeping everything else untouched. Identifying where to use it is something that practice can refine, alongside establishing some common conventions, e.g., you should always use it when doing a sequence of PGSQL queries in Lyceum, etc. Even if someday we ended up adding transformers to our backend code, the same rules apply: Erlang is not Haskell and we gotta choose wisely when to use it.

Conclusions

A common mistake people commit in our industry is to conflate an abstraction or idea with its implementation. They notice that a particular implementation is problematic and generalize it to not only other alternative implementations (usually without the proper research) but then the craziness goes all the way up to the idea itself. Wrong or underperforming implementations tints the entire abstraction under a negative light. That is the recipe for a long-lasting trauma, that routes itself on a kingdom of sand, raised by a bad experience with a vendor/language/hardware.

The ability to separate abstractions and their gains/loses (by themselves) from their implementation counterparts is getting extinct. monads are way bigger than what Haskell offers it – we just saw it being done in Erlang, and further you can see all the way from hints to full behavior of monads in various degrees in other languages, e.g., Erlang, Clojure, F#, OCaml, Rust just to name a few. After thinking for a while, one starts to get signs of monads being a generalization of CPS; something that opens your mind and completely debunks the idea that it is tied to Haskell and it should be treated as a Haskell-exclusive thing. There is a difference between a community having a heavier stance on an abstraction and talking more about it, and it being owned by that community. Just like it does not matter how much the PGSQL/SQLite/Oracle communities talk about DBMSs, none of them will ever own the idea of the Relational Model, Haskellers can have decades of monadic conversations and it won’t make the abstraction theirs.

This journey not only improved my Erlang code, but it solidified the notion that an idea may be the appropriate one but you may be limited by the available technologies your ecosystem provides, and thus you may have to surrender the better idea just because of that limiting factor. The trade-offs may be too heavy to bear, and it is necessary to properly let go of it knowing what is being left on the table. You are choosing to not persue it for various reasons (cost reasons, efficiency reasons, staff reasons, etc) being aware of the consequences of doing so. Lyceum’s case was the one in which the abstraction that solves the problem was reinvigorated because of a library diminishing one of the trade-offs; namely readability.