If you must build a Single Page App, it would be nice to avoid some of the work generally required in a separate codebase. Personally, I look for these attributes to justify the overhead:

• Easy to build on - if it takes slightly longer to bootstrap, that’s fine.
• Hard to break - interfaces are shared and nearly 1-to-1*
• Remove Redundancy - (more code is more maintenance)

The below is an extended description of the process I talked about at Purescript LA. (slides here)

The 2 goals for minimum viability in my mind are:

1. provide correct interfaces for domain-specific queries to build on.
2. ensure handling all responses from the server.

A Big Hammer

Codegen can be a bit dangerous to wield, but I’ll argue here for certain cases where it makes sense and the benefit outweighs the cost.

Anything you are serializing across a network should be considered for codegen. If you’re using haskell for your client code, by all means, share your types! For the rest of us using Purescript, we have the fantastic work of purescript-bridge. If you happen to be using Servant you could further utilize servant-purescript. However, my experience was with a Yesod API. Either way, you’ll depend on the generic json deriving of Aeson (haskell) and argonaut’s generic codecs which are directly compatible with aeson.

Forms are a subset of your model

The low-hanging fruit of this approach is form interactions. Your forms will likely be different than your database representations, so this is a nice place to set up types to describe the frontend input. Say one of our pages is submitting an email of someone who should be invited to the platform. The database representation might include a token, inviter’s id, date invited, etc, but you only need an email and a personal note attached.

data CuratorForm = CuratorForm 
  { email :: EmailAddress
  , message :: Text }

This is what the haskell backend expects as a POST so if our form is directly using this type in purescript we can be certain we won’t screw up and send the wrong things.

I’ll gloss over the form implementation here because it’s pretty similar to what’s covered in “Purescript by Example” and covered in detail a bit futher here

initialState = { form: curatorForm }

curatorForm = CuratorForm { email: "", message: Nothing }

render state = do
  -- ...
  div_ $ Form.renderForm state.form NewCurator do
    void $ Form.textField "message" "A Note" _message Form.optional
    Form.textField "email"   "Email"  _email   (Form.nonBlank <=< Form.emailValidator)

Something worth noting here is the 3rd arguments to textField. These lens’ are generated from purescript-bridge (along with Prisms for CreateResponse) and they’re used to access and set values in the form where state.form is giving us the CuratorForm which is also generated from our haskell types. The only thing here that’s actually locally defined is the NewCurator input event. That’s all that should be needed to hook this form into your state.

  -- the CuratorForm is the 'form' field of our state.
  response <- post (apiUrl <> "/admin/curators") (encodeJson state.form)
  let Tuple typ msg = handleCreateResponse "admin.curator" response
  return $ flashMessage typ msg

We’re able to move a lot of code out or generalize it away here because create’s are pretty simple. There’s another type we’re generating called CreateResponse. The handleCreateResponse function will take care of retrieving the status and a message. (the first argument string is a dumb little i18n translation “index”).

--  Success, Warning, Failure are for determining the message color..
handleCreateResponse :: TranslationIndex -> Json -> Tuple Message String
handleCreateResponse idx res =
  case decodeJson res.response of
    Left e -> Tuple Failure e
    Right cr -> 
      let msg = Msg.t idx cr -- lookup the message based on the response type
      in
        case cr of
          CreateSuccess _ -> Tuple Success msg
          NotUnique -> Tuple Warning msg
          CreateFailure t -> Tuple Failure msg

Here’s what it looks like on the haskell side:

data CreateResponse
  = CreateSuccess Int64
  | CreateFailure Text
  | NotUnique
  deriving (Generic, Typeable, Show)

This can change based on the backend needs and you’ll be forced to handle those cases in the frontend.

Your Haskell API

The codegen will have to fit somewhere in your build-chain; I put mine in the application settings fetch

getAppSettings :: IO AppSettings
getAppSettings = do
  CodeGen.main 
  loadEnv
  loadYamlSettings [configSettingsYml] [] useEnv

This will:

• generate the code in “frontend/src” (Will show where this is specified later)
• populate ENV from a .env file
• read the app settings from a yaml and return it

brent rambo thumbs up

Gen

The process of defining “bridges” from haskell to purescript is relatively straightforward. One “gotcha” is in that “.purs” files are generated in the same module name as where they’re defined in your haskell. Perhaps that’s what you expected, but I’d have preferred they get generated into the same file. I’m using a forked version to provide newtype unwrapping and lens for each record field. (PR here)

This next bit could all be found in the example or readme of purescript-bridge, but I’ll briefly explain the context for each of these code blocks.

In your CodeGen module you’ll need to specify which types are going to be generated in purescript.

-- Your list of types to provide
myTypes :: [SumType 'Haskell]
myTypes = [
    mkSumType (Proxy :: Proxy CuratorForm)
  , mkSumType (Proxy :: Proxy EventForm)
  , mkSumType (Proxy :: Proxy CreateResponse)
  -- ...
  ]

There will be types you’ll need to be specific about; one’s without direct purescript primitives. For example, the Int64 of a primary id database column will need to fall to an Int type in purescript:

import Language.PureScript.Bridge.PSTypes (psInt)

-- delegate to a primitive
int64Bridge :: BridgePart
int64Bridge = typeName ^== "Int64" >> return psInt

This is saying: “Purescript already knows how to handle this but it’s called something else”

The other case might involve delegating a type you’ve defined in purescript. This is not great because you can break your dependent code if you change these. This isn’t bad based on how most of the world does frontend development, it’s actually exactly how you’d make promises with a javascript app to decode consistently. I think we can do better though.

psDateTime :: TypeInfo 'PureScript
psDateTime = TypeInfo {
    _typePackage = ""
  , _typeModule = "Types"
  , _typeName = "DateStamp"
  , _typeParameters = []
  }

utcTimeBridge :: BridgePart
utcTimeBridge = typeName ^== "UTCTime" >> return psDateTime

The above example is saying: you’ll find a type named DateStamp in the purescript module Types and it doesn’t require an external package. Use this when generating fields with type UTCTime.

This is about it for the haskell side and you’ll write out the types to a file like so:

-- defaultBridge, buildBridge, etc are provided by 
-- Language.PureScript.Bridge

main :: IO ()
main = writePSTypes "frontend/src/" (buildBridge mainBridge) myTypes
  where
    mainBridge = defaultBridge <|> int64Bridge <|> utcTimeBridge

Gen-Crafted types for your enjoyment

Below are some examples of the output!

-- src/App/Form.purs
derive instance genericCuratorForm :: Generic CuratorForm
derive instance newtypeCuratorForm :: Newtype CuratorForm _

_email = _Newtype <<< prop (SProxy :: SProxy "email")
_message = _Newtype <<< prop (SProxy :: SProxy "message")

-- src/App/Crud.purs
data CreateResponse =
    CreateSuccess Int
  | CreateFailure String
  | FailedUniquenessConstraint 
derive instance genericCreateResponse :: Generic CreateResponse

_CreateSuccess :: Prism' CreateResponse Int
_CreateSuccess = prism' CreateSuccess f
  where
    f (CreateSuccess a) = Just $ a
    f _ = Nothing

_CreateFailure :: Prism' CreateResponse String
_CreateFailure = prism' CreateFailure f
  where
    f (CreateFailure a) = Just $ a
    f _ = Nothing

_FailedUniquenessConstraint :: Prism' CreateResponse Unit
_FailedUniquenessConstraint = prism' (\_ -> FailedUniquenessConstraint) f
  where
    f FailedUniquenessConstraint = Just unit
    f _ = Nothing

I’ve pasted the entirety of both files I’m using to this gist and note the naming is slightly different because in real use you’ll have to deal with the travesty of haskell record namespaces.

Next Steps

This is fine and dandy, but it could be much better. Purescript has the tools to generate forms from data types and ideally that’s the direction I’d like to move this work.

What follows is a further look at a portion of this post by @parsonsmatt. WForm, as explained there, is a nice abstraction which is inspired partly by Yesod forms. I found it useful and dropped the implementation in a repo to work on when I have time (please help).

Right now it’s specific to Halogen and bootstrap css, but I don’t see any reason it needs to be. In essence it comes down to 2 type aliases and 1 data type. Along with some helper methods for creating fields, you can get a long way with these basics.

There are also further details around the following example in a related post about codegen

type WForm v f a =
  ReaderT
    (Tuple a (FormAction f a))
    (Writer (Array (HTML v (f Unit))))
    a

type FormAction f a = FormInput a -> Unit -> f Unit
data FormInput a
  = Submit
  | Edit (a -> a)

This provides a nice, general interface for forms!

Using this abstraction, we can define an Input for a component

data Input a
  = NewCurator (Form.FormInput CuratorForm) a

One thing I like about this is the ease of edit actions using the lens’ provided by codegen and the FormInput’s Edit constructor

For example, you might handle a form edit in your eval loop similar to this:

eval (NewCurator ev next) = handleNewCurator ev $> next
  where
    handleNewCurator (Form.Edit f) = do
      H.modify (_form %~ f)
    -- ...

We can apply the edit action directly on the form via our lens’. The errors are handled internally to the WForm writer monad.

-- we need access to the form from the top level
-- the form itself already has lens' defined (src/App/Form.purs)
_form :: Lens' State CuratorForm
_form = lens _.form _ { form = _}

initialState :: State
initialState = 
  { form: CuratorForm { invitee: "", message: Nothing } }

Between WForm and our generated lens’ we can compose a form quite cleanly:

-- example of a generated field accessor for out form
_message = _Newtype <<< prop (SProxy :: SProxy "message")

render :: State -> H.ComponentHTML Input
render state =
  div_ $ Form.renderForm state.form NewCurator do
    void $ Form.textField "email" "Email" _invitee (Form.nonBlank <=< Form.emailValidator)
    Form.textFieldOpt "message" "A Note" _message  Form.optional
    --                name,      label,  lens,     validations

Some context (and all the flaws) of this example can be found here

If you think this could be useful and could provide documentation, examples, or code go to github

Project Bootstrapping!

I’m announcing a Reflex FRP bootstrapping template!

You can use it now if you have hi installed.

hi some_project --repository git://github.com/tippenein/haskeleton-reflex.git
cd some_project
make

What it gives you

• stack for dependencies
• small base of common widgets and helpers (which I’ll continue to add to)
• a baseline Model/View/Update logic.

The intention is to be able to start right in by adding your Action’s and Model attributes and have them already plugged into the simple reflex event fold.

Other options

The reflex-platform is the official way supported by reflex for development, so this project is for people who want a reference on a working Stack config or people who are simply too stubborn to move to nix.

Peter Norvig has an old post about writing a spell checker which I’ve always loved the succinctness of. I was on a plane for a few hours and wanted to see how this translated from python to haskell. This is the tale of the journey which I did not intend to take.

Keeping the same structure

First step was going through and recreating the main functionality which is mainly lists and sets, so, no big deal.

Splitting

We split the raw text into words

words :: Text -> [Text]
words = T.split (not . Char.isAsciiLower) . T.toLower

This code is the intuitive answer to the problem above, however it’s very slow. We’ll look at performance later in this post.

Counter / Bag

In python, the Counter is implemented as a multiset / “bag”. We’ll create our own with a Map from Text (word) to Int (count)

type Counter = Map Text Int

toCounter :: [Text] -> Hist
toCounter = Map.fromListWith (+) . fmap (,1)

words <- toCounter . words <$> readFile "big.txt"

There is also a package called multiset but I didn’t know this because the wifi on the plane didn’t work.

Probability and Correction

In order to guess the right way of correcting we need to have probabilities based on the corpus’ word counts.

prob :: Counter -> Text -> Int
prob counter word = occurences `div` totalWords
  where
    occurences = fromMaybe 0 $ Map.lookup t counter
    totalWords = Map.size ms

correction :: Counter -> Text -> Text
correction counter word = maximumBy (\a b -> p a `compare` p b) $ candidates counter word
  where p = prob counter

candidates :: Counter -> Text -> Set Text
candidates counter word = detect
  [ known counter $ Set.singleton t
  , known counter (edits1 word)
  , known counter (edits2 word)
  , Set.fromList [t]
  ]

detect :: [Set Text] -> Set Text
detect = fromMaybe Set.empty . head . filter (not . Set.null)

known :: Counter -> Set Text -> Set Text
known counter = Set.filter (\w -> Map.member w counter)

Edits / Permutations

I initially squished all the logic into single list-comprehensions, but you’ll see I’ve split the heavier functions out.

edits1 :: Text -> [Text]
edits1 w = nub' $ mconcat [transposes', deletes', replaces', inserts]
  where
    alphabet    = fmap T.singleton ['a'..'z']
    splits      = zip (T.inits w) (T.tails w)
    deletes'    = deletes splits
    transposes' = transposes splits
    replaces'   = replaces splits
    inserts     = [l <> c <> r | (l,r) <- splits, c <- alphabet]

The splits gets its own type for cleanliness:

type Splits = [(Text, Text)]

Instead of if R or if len(R)<1 and such like we have in python, I used a guard to skip over splits with contents fitting a certain criteria (e.g (l,r) where r is not empty)

unSplit :: (Monad f, Alternative f) => (Text, Text) -> f (Text,Text)
unSplit = unSplitWith (/= "")

unSplitWith :: (Monad f, Alternative f) => (Text -> Bool) -> (Text, Text) -> f (Text,Text)
unSplitWith f (l, r) = guard (f r) >> pure (l, r)

-- | swap the 1st and 2nd letters across our list of splits ("derp" -> "edrp")
transposes :: Splits -> [Text]
transposes splits =
  [l <> swap' r | x <- splits, (l,r) <- unSplitWith (\a -> T.length a > 1) x]
  where
    swap' w = T.intercalate "" [two, one', rest]
      where
        two  = T.take 1 $ T.drop 1 w
        one'  = T.take 1 w
        rest = T.tail $ T.tail w

-- | remove a letter across all splits "derp" -> ["drp","dep","der"]
deletes :: Splits -> [Text]
deletes splits =
  [l <> T.tail r | x <- splits, (l,r) <- unSplit x]

-- | try replacing a letter with one from the alphabet in each spot. This one is very large
replaces :: Splits -> [Text]
replaces splits = [l <> c <> T.tail r | x <- splits, (l,r) <- unSplit x, c <- alphabet]

I think this comes out reasonably concise.

edits2 :: Text -> [Text]
edits2 w = nub' [ e2 | e1 <- edits1 w, e2 <- edits1 e1 ]

-- Prelude's nub is prrrrrretty bad, so we use this instead.
nub' :: [Text] -> [Text]
nub' = Set.toList . Set.fromList

Performance

The performance of the implementation I came to is… really bad. The time taken to guess even short words was ~4 seconds. This was unacceptable considering the python version is nearly instant.

After asking around on irc and slack, two main problems were pointed out.

• The words function was extremely inefficient (thanks to @mwutton for pointing this out)
• The Map and Set in containers package are not optimized for this sort of bagging. (thanks to @yaron)

In order to speed up the words implementation, we just shove the logic into Data.Text’s implementation (which is nasty). This buys us ~1 second off the ridiculous 4 seconds.. So, I went further.

Since I wasn’t using any order-specific functions on Sets or Maps I just replaced the containers dependency with unordered-containers and changed the import statements to use them. Bam! This nearly halved the time! But it’s still real bad at 1 second.

I used the profiteur tool to visualize the performance issues a bit while going through this process, which just basically confirmed that Set/Map operations and words were awful, like we already knew.

It seems as though python’s Counter shouldn’t be all that different than ours (an unordered hash set) but the haskell version lags behind. I kept the code as intuitive as I knew how and it wasn’t quite enough for this type of problem.

Lessons

1. Always use unordered-containers unless you for some reason need to keep the ordering of your data structures.
2. Sometimes pre-processing is worth the effort. You can try as hard as you want to optimize the function, but at some point you have to call it a loss.

I’d welcome any comments about how this could be improved further. The result was not encouraging but despite this, I did learn some things along the way.

The full code is here

A literate haskell file to follow along is here

A few things have changed with Servant’s client deriving between 0.4 and 0.7. I’ll document here what I did to upgrade.

Follow the types

First thing you’ll notice after building your project with the 0.7 version of servant client is some failing functions. Let’s follow the types and do what they tell us.

The first error is that BaseUrl takes an extra “path” argument. It’s just a string; ghc tells us this.

makeBaseUrl = do
  h <- Config.domain <$> Config.remoteConfig
  p <- Config.port <$> Config.remoteConfig
  -- BaseUrl has 1 extra argument
  -- used to be: pure $ BaseUrl Http h p
  pure $ BaseUrl Http h p ""

Now we use ExceptT instead of EitherT. It’s better, and ghc tells us that’s what it’s expecting.

type Action a = ExceptT ServantError IO a

This type alias has existed as ClientM and Handler (see this and this) but I prefer to write this explicitly and forego all the changes that might happen in servant.

The next compiler error says something about an expected Manager argument to one of our bound client functions.

Http manager

There is a new dependency on http-client. Luckily I had structured my code to use this generic run function, so I only had to make manager updates here.

run action = do
  baseUrl <- makeBaseUrl
  manager <- newManager defaultManagerSettings
  result <- runExceptT $ action manager baseUrl
  case result of
    Left message -> error (show message)
    Right x -> pure x

Essentially we only needed to pass 2 new arguments to the action (the derived functions from client) and run them inside runExceptT.

The last compiler error you’ll get is an extra argument to client which is just a remnant you can delete. It was previously the BaseUrl argument, but we’ve already moved that to the run function.

listDocuments' :<|> createDocument' =
  -- we no longer pass the url into the `client` deriving function.
  client documentAPI

That’s pretty much it. I didn’t really have to look anything up. The compiler didn’t directly tell me how to fix it, but it did tell me which parts of my code had strange arguments, which were then easy to correct.

Matter of fact, the commit where I recently did this is here if that’s your sort of thing.

Hard Choices

Everytime you make a design decision about a system, you have to (or should) consider the possible maintenance implications. Personally, I think typed languages make this much more manageable. We’ll use Haskell and Elm to illustrate this case via a project called hasken.

This started as a simple CLI communicating with an API, but as is so common, we now want a frontend to display the resources on the web.

What we want

We want a few reasonable things:

1. Our entities should be synchronized between the frontend and backend.
2. Changes won’t break any existing functionality.
3. No way of leaving out edge cases

Synchronizing changes between Api and Frontend

When we have a change or addition to the representation of data, we’ll have to change both back- and frontend. There are a few possible solutions, but I’ve chosen to use typed-wire which provides us with an intermediate language to define the common entities in.

type Document{
  id : Int;
  title : String;
  content : String;
  tags : List<String>;
}

When we’ve defined our entity, we can generate the code we need to produce and consume this data.

twirec -e Document -i . --elm-out src/Gen --elm-version 0.17

At the moment of this writing, typed wire can also generate a Spock api and json encodings, however, hasken already has a solid servant api defined, so we’ll be discussing how to use the generated elm bindings to have type-safe communication between them.

Note: The Servant side of things will be discussed in a separate post, but the same design principles and ensuing safety will be the same as we’ll see with Elm soon.

Maintainability

Instead of writing a whole lot of tests (which also need maintenance) to ensure the proper manipulation of data between components, let’s encode this information in the types. This way, we’ll know how to fix something before we even write specs.

PSA: To be clear, I’m not saying “don’t write tests”; I’m saying “We can get away with far fewer!”

When you make changes to an Api, you should feel comfortable updating the frontend to match those changes. Types should help not hinder in this regard. For more context I like this sum-up from Well-Typed.

If you give your program precise types and follow systematic design principles, your program almost writes itself.

As an example of using a compiler to guide feature implementation, let’s look at actions in Hasken’s interface:

type Action
  = FetchDocuments
  | ErrorOccurred String
  | DocumentsFetched (List Document)

This encodes every possible state our app can be in. This is huge! The compiler will force us to deal with every case.

Our model holds the data which we’re trying to represent.

model : Model
model =
  { documents = Right []
  , queryString = ""
  }

At this point you might be thinking.. Isn’t this just redux? Yes! Reactive Programming is just a way of thinking about data flows. That fact isn’t as important as the way we can develop programs in Elm though.

An Example: Adding Search

The ease of developing this interface shows with an example of adding a feature. Let’s add search.

First we add an Action union-type Search String which just means there is an action which supplies a string and represents “searching”. That’s all we know at the moment, but we want our app to represent this in some way. Now we look to the compiler.

It tells us that our update function doesn’t handle the Search action case.. So we must write it!

Our current update looks something like this:

update : Action -> Model -> (Model, Cmd Action)
update action model =
  case action of
    FetchDocuments ->
      model ! [fetchDocuments]
    ErrorOccurred errorMessage ->
      { model | documents = Left("Oops! An error occurred: " ++ errorMessage) } ! []
    DocumentsFetched documents ->
      { model | documents = Right(documents) } ! []

-- Syntax tip: `model ! [fetchDocuments]` is syntactic sugar for `(model, fetchDocuments)`
-- Type tip: `Right` and `Left` are encoding success and failure. This is called the `Either` type.

This method returns an updated model with some possible Cmd Action. If we’re going to fix the compilation error, we need to pattern match Search term and tell it how to react.

Search term -> model ! [searchDocs term]

Now we define the function to fetch documents:

getDocs : Maybe (List (String, String)) -> Cmd Action
getDocs mquery_params =
    let url = Http.url docUrl (fromMaybe [] q)
        docUrl = baseUrl ++ "/documents"
    in
      Http.get (Json.list jdecDocument) url
        |> Task.mapError toString
        |> Task.perform ErrorOccurred DocumentsFetched

1. our chaining of tasks |> ... |> ... deals with encoding the result of the http requests into our action types (ErrorOccured & DocumentsFetched)
2. jdecDocument is the json decoder that was defined in our typed-wire-generated module; we expect a list of those.
3. query params might be empty, so we deal with that case via a Maybe and our fromMaybe defaults to [] if none are passed.

The view

The next compiler error will tell us that we don’t have an interface component to trigger the Search action. We’ll be able to succinctly express this in our interface via the view function.

view : Model -> Html Action
view model =
  div []
    [
      statusMessage model.documents
    , searchBox model Search  --- a function to generate markdown for a search box
    , documentList model.documents
    ]

From a high level, this is a very useful abstraction. We can modularize our view “partials” as we see fit and it helps us think about the best way to organize our components.

Now it will compile! We’ve successfully added a feature that we know works.

What next?

In the follow up we’ll talk about deploying all this in a sane way, but for now we can plan out the build steps.

I’ve just used a Makefile for now and it looks like:

all:
    gen
    build
    js

gen:
	twirec -e Document -i . --elm-out src/Gen --elm-version 0.17

build:
	elm package install

js:
	elm make src/Main.elm --output=elm.js

After running this, we’ll have the json encoding module (in src/Gen/) along with a compiled elm.js which we can simply attach to an html node in our index:

<script src="./elm.js" type="text/javascript"></script>
<script>
    var node = document.getElementById('content');
    var app = Elm.Main.embed(node);
</script>

Since these steps won’t work on your machine unless you gather all the dependencies (typed-wire, ghc, elm, etc.), in the next step we’ll put this all in a docker container and show how to deploy it using whichever docker service you’d like.

Starter project

I remember when I started programming in python. I looked for anything I could write as a CLI or automate in some way, so in the spirit of that, I decided to write a bit about doing the same using haskell libraries.

I replaced monosnap functionality (sharing screenshots) with this script awhile back. It’s generically useful and here is the whole of it if you’re only interested in the source.

Also, there’s a deeper dive into Wreq here if you find this cursory intro lacking

Bam! Some code

The important imports:

import Data.Monoid ((<>)) -- just for glueing together Text's
import Network.Wreq       -- the request library
import Control.Lens       -- setting and getting params/headers/etc
import Data.Aeson.Lens    -- same

Since the main thing this script does is upload a photo anonymously to imgur, we’ll start with that function.

The Request

uploadAndReturnUrl :: IO String
uploadAndReturnUrl = do
  imagePath <- parseArgs
  cid <- clientId
  let authHeader = defaults & header "Authorization" .~ ["Client-ID" <> " " <> cid]
  let payload = [ partText "type" "file"
                , partFile "image" imagePath
                ]
                                  
  res <- postWith authHeader "https://api.imgur.com/3/image.json" payload
  let guid = res ^. responseBody . key "data" . key "id" . _String
  return $ "http://imgur.com/" ++ T.unpack guid

Let’s talk about what might be confusing here. The declaration of authHeader; what is that?

-- in this context, simply a merge 
(&) :: a -> (a -> b) -> b

-- build the header to combine with the defaults :: Options
header :: HeaderName -> ([ByteString] -> f [ByteString]) -> Options -> Options

If you wanted to, you could think of this as an equivalent to the ruby code:

defaults = {:params => [], :headers => []}
defaults[:headers].merge({ :Authorization => "Client-ID #{cid}" })

Fortunately, our code using Lens’ is much more fail-safe. (you could read more here or one of the plethora of other lens tutorials online)

The actual post is pretty self explanatory if you know what authHeader and payload are, but here’s the type sig anyway:

postWith :: Postable a => Options -> String -> a -> IO (Response ByteString)

The Postable typeclass refers to our [Part] which we constructed in the payload declaration, nbd.

The Response

res ^. responseBody . key "data" . key "id" . _String

We use ^. to pull out the image id returned from imgur. Unfortunately, lens’ compose left to right, unlike normal functions in haskell so this is responseBody.data.id

The equivalent ruby code might be something like:

response.body[:data][:id]

Again, ours is a bit safer.

Wrap Up

There’s one other piece which grabs the most recent screenshot.

getRecentPath :: IO FilePath
getRecentPath = do
  home <- getHomeDirectory
  d <- return (joinPath [home, "Desktop"])
  files <- globDir1 (compile "Screen Shot*") d
  case headMaybe (reverse (sort files)) of 
    Nothing -> error "no recent screenshots"
    Just a -> return a

This could be improved by returning an Either String FilePath instead of raising an error, but if we stopped and optimized every chance we had in haskell we’d never finish writing what we started.

At the end of all this, we can screen cap and pop over to a terminal:

imgup --screenshot | pbcopy

Now we have an imgur link in our copy buffer which (for me) completely replaces monosnap’s functionality.

In a talk by Avi Bryant about practical systems for Analytics or Aggregation he drew comparisons to Monoids and Abelian Groups. I thought it would be helpful to work through what he was explaining with haskell, so here we go.

The first thing discussed is a simple summing with a monoid. We need implementations for:

• mempty: somewhere to start; something that won’t affect the outcome of the sum.
• mappend: a way of combining 2 instances together, (we’ll use the infix notation or <>)
• mconcat: a way of combining a list of instances together down into a single instance (a fold)

The point of this abstraction is to easily allow a fold (reduce) of any data type into itself. It doesn’t have to be commutative, but these monoid instances happen to be. (Abelian groups however, have to be commutative)

Part 1 - A simple Sum

data Collection = Collection { theSum :: Integer }
instance Monoid Collection where
  mempty = Collection 0
  Collection sum1 `mappend` Collection sum2 = Collection (sum1 + sum2)
  mconcat = foldl' mconcat mempty

That isn’t an astounding monoid instance by any means, but it allows us to do simple summing:

mconcat [Collection 1, Collection 10, Collection 20]
-- > Collection {theSum = 31}

-- <> is just an operator equivalent to mappend
Collection 1 <> Collection 2
-- > Collection { theSum = 3 }

Part 2 - A simple Averaging aggregation

The next step is to average the sum as we go.

data Collection = Collection 
  { theSum :: Integer
  , avg :: Maybe Double }

What we’d expect with this is Collection 1 1 <> Collection 10 10 to equal Collection {sum = 11, avg = 5.5 }

Later on we might want to subtract a Collection from this aggregation or weight the averages. These scenarios can be handled by storing a count of Collections seen so far. Ideally we would have an implicit count instead of a repetitive explicit argument, but for now let’s just add count to the data type and write a quickcheck property for counting:

-- btw, the data type now looks like:
-- { theSum :: Integer, count :: Integer, avg :: Maybe Double }
it "counts consistently" $ property $ \n ->
  count (mconcat $ replicate n justOnes) `shouldEqual` n

QuickCheck will tell us one error that we probably didn’t consider: What if the count is negative? We shouldn’t allow that, but for now we’re just going to take the absolute value of what quickcheck inputs and test all the positive values of count.

count (mconcat $ replicate (abs n) justOnes) `shouldBe` abs n

(I’m fighting the urge to tangent on QuickCheck, but I promise to tackle it in a different post)

Now for the instance definition of our new averaging sum-thingy! I’ll skip over the more trivial parts, but the example repo will be available for a complete look here

-- The only thing changing here is the mappend definition
instance Monoid Collection where
  mempty = Collection 0 0 Nothing
  -- The cases where average is Nothing are omitted
  (Collection s1 c1 (Just avg1)) `mappend` (Collection s2 c2 (Just avg2)) =
    Collection (s1 + s2) (c1 + c2) average
    where
      average = Just $
        ((avg1 * fromInteger c1) + (avg2 * fromInteger c2))
          / (fromInteger c1 + fromInteger c2)
  mconcat = foldl' mconcat mempty

You might say “That’s just some grade-school weighted average stuff.. childs play!”, and you’d be right!

Part 3 - The beauty of this abstraction

Ok, the weighted average is fine.. but what if we also wanted a top 10 aggregation. It turns out we still only need to change a few things to get this aggregation to work.

We might as well drive the implementation with specs since the intention is well defined.

-- I usually write the intended data type first then use that interface to build specs.
-- so here's our new fields for the CollectionTops data type
-- { aSum :: Integer, tops :: [Integer] }

topsCollection1 = CollectionTops 1 (reverse [1..10])

it "collects the top10" $
  topsCollection1 <> mempty `shouldBe` topsCollection1

it "collects the top10" $ do
  let cs = topsCollection1 <> CollectionTops 3000 negativeInfinities
  head (tops cs) `shouldBe` 3000
  (1 `elem` tops cs) `shouldBe` False -- it knocks 1 out of the list

negativeInfinities = replicate 10 (-1)
revSort = sortBy (flip compare)

instance Monoid CollectionTops where
  mempty = CollectionTops 0 negativeInfinities
  CollectionTops s1 tops1 `mappend` CollectionTops s2 tops2 =
    CollectionTops {
      aSum = s1 + s2
    , tops = take 10 $ revSort ([s1] ++ [s2] ++ tops1 ++ tops2) }
  mconcat = foldl' mappend mempty

Part 4 - Sum up (pun intended)

In his talk, Avi obviously goes further than this, however, this post is already becoming too long. Other uses hinted at include skewness, kurtosis, histograms, and unique visitors. The above implementations are intended to give an intuition about commutative monoids as a tool for aggregation.

To sum up…

• we need 3 functions for a monoid instance: mempty, mappend, and mconcat.
• If you can define mappend, the other 2 should be easy (I’ve noticed that mconcat is mostly just used to provide a faster implementation of the standard foldr mappend mempty).
• The monoid laws ensure associativity, but we have to be explicit about our commutative intentions.

I hope this calms your nerves about monoids and perhaps provides a bit of intuition about what is or could be a monoid.

A simple useful case for Template Haskell

{-# LANGUAGE TemplateHaskell #-}

import Language.Haskell.TH

declareAnimals :: [String] -> Q [Dec]
declareAnimals animals =
  return [DataD constraints name vars cons derives]
    where
      constraints = []
      name        = mkName "Animal"
      vars        = []
      cons        = map (\a -> NormalC (mkName a) fields) animals
      fields      = []
      derives     = [''Show, ''Eq, ''Ord, ''Enum, ''Read]

animals = ["Aardvark", "Bobcat", "Quokka"]
$(declareAnimals animals)

-- data Signal = Aardvark | Bobcat | Quokka

If you only cared about getting an enum type, this is where you can jump off.

Digging into the types

If we look further at the types involved here, we can learn more about what this is doing.

At a high level, we’re simply building a haskell AST and introducing it into our code-base. You can think of this as meta-programming but if you ask me it’s more like playing with legos.

We construct a data declaration with DataD

DataD :: Cxt -> Name -> [TyVarBndr] -> [Con] -> [Name] -> Dec

or if we spread this out and document each piece.

DataD :: Cxt          -- constraints (eg. Num a => a -> a)
      -> Name         -- the type name
      -> [TyVarBndr]  -- type variable bindings
      -> [Con]        -- constructors
      -> [Name]       -- deriving types (eg. Eq, Ord, etc)
      -> Dec

simple Name constructor

mkName :: String -> Name

build a normal Constructor

NormalC :: Name -> [StrictType] -> Con

The derives need to have the double-tick in order to be read as a syntax builder rather than an actual deriving statement. The same goes for type constraints and such.

You can look at all this documentation on the Language.Haskell.TH page

P▲rT2

In the previous post I showed how to use parsec to parse data in a format like this:

"1%400:3.2 6%some_description|100:1"

Why not regex?

I certainly could’ve used a regex pattern like \d\%(\w*\|)?(\d+):(\d+\.?\d?)

…but, there are some scenarios where this falls apart quite quickly:

• if we learn about other formats of data that can be included
• if we have other parsing tasks that need similar matchers?
• if we need to morph the data in some way before matching
• if the list of possible separators are very large. (\d\%|\$$|\&|...)

An example to prove I’m not making this up

I had never encountered the acronym FFR until I started working in financial software. It stands for Fixed Format Response, but that’s not really important. The important part is that the FFR we’re dealing with has ~100 different signals which indicate a specific type of data.

So, we’ll create a data type deriving Enum to describe how we expect to split the data up.

data Signal
  = AD02 | AD11 | AH11 | AM01 |
    AO01 | AR01 | AS01 | AT11 | BR01 |
    -- ... more of these removed for reading clarity
    UA11 | UF11 | VH01 | VS01 | WS01 |
    YI01 | ZC01
  deriving (Show, Enum, Ord, Eq, Read)

allSignals :: [String]
allSignals = map show [AD02 ..]

Note: The syntax for allSignals is just enumerating all the constructors. (The space is significant [YourFirstEnum ..])

-- notice we're reusing this from the previous parser
anythingUntil p = manyTill anyToken p

anySignal :: Parser (Signal, String)
anySignal = do
  signal <- signalParser
  content <- anythingUntil (endOfLineOrInput <|> signalLookahead)
  return (toSignal signal, content)

signalLookahead = lookAhead signalParser *> return ()

signalParser :: Parser String
signalParser = choice $ fmap try $ string <$> allSignals

We’re going to use the anySignal parser to pull out many pieces of content from a string, but the interesting part is the signalParser itself. choice and <|> are the same, but we need to choose between all the signals so we pass a list of Parsers. If it helps, it looks a bit like this if you were to expand it:

choice [(try $ string "AD02"), (try $ string "AD11"), ...]

Another thing to note is the signalLookahead. We need to avoid eating up the next signal and just use it to signal the end of input.

Once again, there’s a freeze of the jupyter notebook if you’d like to see it in the full context (here)

Further processing

There are many more things we can do with our data in this format, but the first thing I would do is consume the data into some Map like this:

type SignalMap = Map.Map Signal String

From here we’d want to inspect what each signal has inside of it, so we can take from this Map and further parse the string content.

Credit

Thanks a bunch to both of these resources (which are both far better and more comprehensive than this):

• https://github.com/JakeWheat/intro_to_parsing
• http://unbui.lt/#!/post/haskell-parsec-basics/