me and my wife are using the same credit card, which is linked to my bank account. meaning every month i close the debt, and my wife transferres me some part of that debt.

which means when we have visited some cafe, and payed from a credit card, i will log it like that:

2025/03/08 Cafe Pupp
    expenses:Food:Eating Out              508.00 CZK @@ 224.72 SEK
    liabilities:shared:nordea:gold

what i really want is to log say 35% into my wife’s liability

2025/03/08 Cafe Pupp
    expenses:Food:Eating Out              508.00 CZK @@ 224.72 SEK
    liabilities:shared:nordea:gold
    expenses:Food:Eating Out             -177.80 CZK @@ 78,65 SEK
    liabilities:wife                      78.65 SEK

i don’t want to manually calculate that every time, so naturally i’ve reached out to automated postings.

that’s where that problem starts. hedger doesn’t support references in auto-posting rules

but ledger allows me to do this:

; transactions tagged with shared:: 35% will be shared with wife
; a - The posting’s amount; the balance of an account, without considering children.
; b - The cost of a posting; the cost of an account, without its children.
= expenses and %shared
    $account                                                           (a * -tag("shared") * 0.01)
    liabilities:wife                                                   (b *  tag("shared") * 0.01)
    equity:conversion:%(commodity(a))-%(commodity(b)):%(commodity(b))  (b * -tag("shared") * 0.01)
    equity:conversion:%(commodity(a))-%(commodity(b)):%(commodity(a))  (a *  tag("shared") * 0.01)

which does exactly what i want automatically if i tag transaction with new ‘shared’ tag like so:

2025/03/08 Cafe Pupp
    ; shared:: 35%
    expenses:Food:Eating Out              508.00 CZK @@ 224.72 SEK
    liabilities:shared:nordea:gold

which is perfect.

also now with ledger i can finally properly track investment lots

it is hard to describe the feeling i had learning vim. but it is definately the same feeling i have now when i ride on a fixed gear bike.

to be doing a mundain activity with foreign tools captures a lot of attention. and it is surprisingly pleasent to pay nearly all the attention to the present moment.

constantly remembering which mode i am in now and all the shortkeys, or asessing traffic and people around because i can not stop fast enough.

i think that is what buddist monks mean when they talk about “living in the present moment”. there is no past or a future, only me and a that traffic light ahead that i should either race now or slow down and wait.

but that feeling flees. i do not have it with vim at all now, after many years. and i have it less and less on my bike every day.

i would like to keep it for a long as i can.

a few days ago this post popped up in my rss feed. it is about a small service Duarte built for himself to keep track of news in denmark, where he lives, but does not speak danish.

i thought it would be cool to have something similar for sweden, where i live, but do not speak swedish. so i built it. with a little sprinkles of ai.

instead of translating articles, my service first groups them by meaning and only translates headlines for each group.

grouping is done by using openai’s embeddings api and dbscan algorithm. it is not perfect, but good enough for my use case.

here it is: sverige-news.fly.dev

i also like rust and have become comfortable with it last year.

and since new year is a good reason to rewrite your entire website to a new stack,

this website is now powered by a single binary built with rust.

source

one thing I’ve discovered browsing through other people’s default apps - there are mostly no apps i haven’t heard about.

which either means I spend too much time in the Internet, or that there are not so many good apps after all.

first, being a software engineer is a mix between being an architect and being a writer. most people can imagine what both architects’ and writers’ do. so the comparison like that is a good start.

architect part:

• considering business requirements

  a building has a purpose: it’s a mall, bridge, apartement house, hotel, etc. architects convert buisiness requirements into something that could solve a buisiness problem. that’s what programmers do too. maybe we are building a social network, text editor, or photo editing tool - it’s all about solving a problem.

• applying technical expertise

  building something is always constrained by a real world. and those constraint affect a lot how you build a solution for a proble. for architects it’s a physical limitations that constraint a building shape or materials they can use. programmers are limited by internet connectivity, processor power, memory size, etc.

• a style

  the artifact inevitably captures personality. same way you can sometimes tell when a building was created, or who is the author, you could tell similar things by reading programmer’s code.

• organization structure

  large projects involve a lot of people. high level architects who prepare blueprints. low level workers who put briks together. managers all around. specialists who take care of electricity, painting or putting the right amount of trees around.

  software organizations are built in the same way. different people own different part of a software system. some have different specializations, some do not write actuall code at all anymore.

writer part:

• self-fulfilling prophecy

  here is the most interesting part for me.

  since software exists somewhere between real world and your computer, you can not build it by putting physical things together. what you do instead, you write down a very detailed description of what you want to build and how to do that. and then it just happens, becoming some kind of a self-fulfilling prophecy.

  imagine if you had a magic book where you could write down a very detailed description of your dream house. after you’ve written it down, you have the house built.

• using a language

  writing means using a language. computers do not understand english yet, but similar aspects matter. is it easy to read? how well is it structured? and so on.

• many ways to write it

  you can communicate a similar idea by writing a poem, or a very strict scientific text. it can be short, or long. you could rephrase the same thought over and over again. you can translate it to a different language, because not all people speak the same language.

  all those things are almost literaly the same in the programming world.

this metaphor could also be used to explain version control:

> how to organize multiple people writing the same book at the same time?

or why software is never completed

> like if environment around your building is constantly changing, so you have to reinforce / adjust / extend parts to adjust

1. it must fit inside my hat

   right now i have a little too much of hairs to fit inside the hat. it kind of feels like i am stuffing the hat with hairs a little bit - that i’d like to avoid

2. when i take my hat off, the hairs should already be good

   i don’t want to think about how i look just after i take my hat off. i’d like it to be exactly as i looked when i left my home

normally, this limits me quite a bit in terms of the hair length. i just never realised i prefer shorter haircuts because the place i grew up in gets cold.

accumulating useless information like that only stresses me and is not useful at all. i usually rememeber what i actually liked and will use that from now on.

still, some journaling i consider important. like keeping a ledger of my financial history - that is useful.

• without a dedicated server software that i have to run and maintain
• works without javascript
• doesn’t share data with 3rd parties

and last week this post poped up in my rss feed. it’s written in russian, so i’ll summarize it briefly.

the author noticed that disqus (a cloud comments engine) that he uses sends a lot of garbage js requests and decided to fix that. the fix is neat: use github pull requests to create website comments.

i really like the idea, so now my website has comments that work in a silimar (yet a little beter) way! here is how:

1. i have a github action to create a new comment. it’s really simple: given a author name, url and comment body, create a new .md file with the coment body like this:

---
pathname: "/posts/blog/cluster/"
author-name: "Nikita"
timestamp: "1664822102"
---

comment body

2. a sveltekit form to trigger action via http:

<form method="POST" use:enhance>
 <!-- author name input -->
 <label for="author_name"> Your name: </label>
 <input name="author_name" type="text" required />

 <!-- challange text -->
 <label for="solution">{data.challange} = </label>
 <input name="solution" type="text" required />

 <!-- comment body -->
 <textarea rows="3" name="body" type="text" required />

 <!-- url pathname to link comment to the page -->
 <input name="pathname" value={$page.url.pathname} type="text" hidden />

 <!-- input challange for validation -->
 <input name="challange" value={data.challange} type="text" hidden />

 <input type="submit" value="Comment" />
</form>

3. some server-side typescript to trigger the action:

import { solve } from "$lib/challange";
import { invalid } from "@sveltejs/kit";
import type { Actions } from "./$types";
import { env } from "$env/dynamic/private";

const GITHUB_TOKEN = env.GITHUB_TOKEN;

const trigger = (inputs: any) =>
 fetch(
   "https://api.github.com/repos/ngalaiko/galaiko.rocks/actions/workflows/create-comment.yaml/dispatches",
   {
     method: "POST",
     headers: {
       Accept: "application/vnd.github+json",
       Authorization: `Bearer ${GITHUB_TOKEN}`,
     },
     body: JSON.stringify({
       ref: "master",
       inputs,
     }),
   }
 );

export const actions: Actions = {
 default: async ({ request }) => {
   const data = await request.formData();
   const body = data.get("body");
   const author_name = data.get("author_name");
   const pathname = data.get("pathname");
   const solution = data.get("solution");
   const challange = data.get("challange");

   if (body === "")
     return invalid(400, { message: "Message can not be empty" });
   if (author_name === "")
     return invalid(400, { message: "Please, fill in name" });

   if (solution === "") {
     return invalid(400, { message: "Challange solution is empty" });
   } else if (solution !== solve(challange as string)) {
     return invalid(400, { message: "Wrong solution" });
   } else {
     const res = await trigger({ body, author_name, pathname });
     if (res.status !== 204) {
       return invalid(500, { message: await res.text() });
     } else {
       return {
         success: true,
         message:
           "Thanks! Your comment will appear after moderation. Check in later!",
       };
     }
   }
 },
};

and that’s it! thanks to sveltekit, the implementation is both nice, js-free and i don’t need to think about hosting

What is IndieWeb

IndieWeb how it describes itself, is:

a people-focused alternative to the “corporate web”

I also don’t like the “corporate web” and try not to participate in it. However, right now IndieWeb for me is mostly something fun to do on the weekends.

On a more technical level, it’s a combination of protocols like WebMentions, Microsub and some others that together enable a distributed social interaction on the Web.

What I would like to achieve

• be discoverable on IndieWeb
• be able to interact with others on IndieWeb
• not use any 3rd party IndieWeb services like webmention.io
• pay zero moneys for my setup
• have all of my data locally stored in a form of static files
• my website to be full functioning without client side javascript

Setup

I used to power this website with Hugo - a static website generator. But it doesn’t quite fit IndieWeb requirements for me.

Most people, as I understand, use centralized platforms like webmention.io or micro.blog mixed with different CMS plugins to be on IndieWeb. For me - it’s no fun. I want to do it all myself - thus static website won’t do it. I have to have a server to at least receive webmentions.

Some time ago SvelteKit got me interested. SvelteKit is a framework to build webapps using Svelte. What is especially cool about it for my use-case is that it is:

• prerendered by default
• comes with adapters for different deploy targets out of the box
• allows to seamlessly write serverless functions

With that said, my tech choice is: SvelteKit deployed to Cloudflare Pages

SvelteKit

I won’t dive into details here, you can check out the code. Pretty much it’s SvelteKit with mdsvex. While configuring it, there two articles helped me a lot:

• Let’s learn SvelteKit by building a static Markdown blog from scratch
• MDSveX and Svelte Kit

Microformats

First step in IndieWeb is to setup microformats. It’s a data format built on top of xml that provides a standard way of communicating between IndieWeb sites, meaning:

You add some attributes to your website’s html:

<a class="h-card" href="/" rel="me">Jimmy</a>

Microformats parsers take html and convert it to standardised JSON:

{
  "items": [
    {
      "properties": {
        "name": ["Jimmy"],
        "url": ["http://example.com/"]
      },
      "type": ["h-card"]
    }
  ],
  "rel-urls": {
    "http://example.com": {
      "rels": ["me"],
      "text": "Jimmy"
    }
  },
  "rels": {
    "me": ["http://example.com/"]
  }
}

The JSON is used to power comments functionality on your website (for example).

You can read more here and here.

Webmentions

Second step is webmentions. Webmentions is a very simple protocol to let other’s know that you’ve mentioned their page. A mention could be anything form a ‘like’ to ‘comment’ or even ‘rsvp’.

Since the protocol is so simple, I am implementing it myself. Both because it’s more fun and to satisfy my requirements for the project.

There are two parts when it comes to webmentions: receiving and sending.

Receiving

Since I don’t want to pay for the hosting, here is a plan:

1. When someone send me a webmention, the request is be processed by a serverless handler hosted on cloudflare workers and then stored in their key value storage
2. Later, an export script is triggered by myself manually or via GitHub Actions cronjob to download all of the new webmentions into my repository
3. Then, a processing script is triggered in a similar manner to process new webmentions
4. Viola, now all the raw webmentions data is available in my repository at all times, and I can use it to render static pages

What I like about this plan is that:

• it doesn’t really depend much on the hosting solution since SvelteKit is quite flexible with that
• I expect the amount of data and read/write operations be close to 0, so free tier on cloudflare and github is more than enough
• all of the data is stored inside my repository which means I can render static pages

Of course the non real timeness of the solution might be considered a downside.

Sending

After receiving part was done I conviniently had written code that can be reused to send webmentions too. Since this website is fully prerendered and distributed as a set of static files, I can analyze those files and extract my webmentions from it.

Once webmentions are extracted, it’s trivial to implement the discovery and make script ping the endpoints.

Test

The footer of this article contains a list of received webmentions, and I’ve already sent one!

Update

Support for webmentions was successfully removed, since I do not use it :)

SD card

• Download operation system.

  I am using HypriotOS v1.10.0 It is based on Raspbian Buster Lite, has a preinstalled docker and some other minor things to simplify cluster setup:

    $ uname -a
      Linux node-red 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux
    $ docker version
      Client:
       Version:           18.06.3-ce
       API version:       1.38
       Go version:        go1.10.3
       Git commit:        d7080c1
       Built:             Wed Feb 20 02:42:54 2019
       OS/Arch:           linux/arm
       Experimental:      false
  
      Server:
       Engine:
       Version:          18.06.3-ce
       API version:      1.38 (minimum version 1.12)
       Go version:       go1.10.3
       Git commit:       d7080c1
       Built:            Wed Feb 20 02:38:25 2019
       OS/Arch:          linux/arm
       Experimental:     false

• Flash it to SD cards using flash

  Don’t forget to define a hostname. It will save you some time later.

    $ flash -n node-red ./Downloads/hypriotos-rpi-v1.10.0.img

• Provision all nodes you have like this and boot them up.

• Install Kubernetes on all nodes.

  Note the version here: 1.13.5. Because of changes in 1.14.0, Kubernetes requires enabled pids cgroup. But kernels before 4.19.46-v7+ do not support it. If you are reading this after some time, it should be fixed, and the latest version of Kubernetes might work just fine.

    $ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
        echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
        sudo apt-get update -q && \
        sudo apt-get install -qy kubeadm=1.13.5 kubectl=1.13.5 kubelet=1.13.5

• Initialize Kubernetes on the master node:

    $ sudo kubeadm init

  Notice sudo here. For some reason, if you do it as root, it can fail with an error (couldn’t find what the error was :( )

  If all goes fine, you will see an output similar to:

    ...
    Your Kubernetes master has initialized successfully!
    To start using your cluster, you need to run the following as a regular user:
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
      You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    You can now join any number of machines by running the following on each node
    as root:
      kubeadm join --token TOKEN 192.168.1.100:6443 --discovery-token-ca-cert-hash HASH

• Do what the output says:

    $ sudo cp /etc/kubernetes/admin.conf $HOME/
    $ sudo chown $(id -u):$(id -g) $HOME/admin.conf
    $ export KUBECONFIG=$HOME/admin.conf

  Now you should be able to list all nodes. There is a single NotReady master node at this point:

    $ kubectl get nodes
    NAME         STATUS     ROLES    AGE   VERSION
    node-red     NotReady   master   17m   v1.13.5

• Add other nodes to the cluster.

    $ sudo kubeadm join --token TOKEN 192.168.1.100:6443 --discovery-token-ca-cert-hash HASH

  Notice sudo again.

• Now, you should be able to see all nodes from the master node:

    $ kubectl get nodes
    NAME         STATUS     ROLES    AGE    VERSION
    node-black   NotReady   <none>   106m   v1.13.5
    node-blue    NotReady   <none>   105m   v1.13.5
    node-green   NotReady   <none>   105m   v1.13.5
    node-red     NotReady   master   110m   v1.13.5

• Deploy container network from the master node.

    $ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=1.13.5"

• After some time, all nodes should be ready.

    $ kubectl get nodes
    NAME         STATUS     ROLES    AGE    VERSION
    node-black   Ready     <none>   106m   v1.13.5
    node-blue    Ready     <none>   105m   v1.13.5
    node-green   Ready     <none>   105m   v1.13.5
    node-red     Ready     master   110m   v1.13.5

Links

• https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/
• https://kubecloud.io/setting-up-a-kubernetes-1-11-raspberry-pi-cluster-using-kubeadm-952bbda329c8
• https://github.com/hypriot/flash
• https://github.com/teamserverless/k8s-on-raspbian/issues/16

I will describe what optimizations I used to solve the problem from the most to the least obvious.

Solution was measured on MacBook Pro 2018 with 2,2 GHz Intel Core i7 CPU and 32 GB RAM.

Generic solution

The most simple way to solve this is to iterate over the file, save words to a map with number of occurences, then iterate over the map and print top 10 most common words.

func countWords(fileData []byte) {
    words := map[string]uint{}

    wordBuf := make([]byte, 64)
    wordPos := 0
    for _, c := range fileData {
            switch {
            case c >= 'A' && c <= 'Z':
                    c += 32
                    fallthrough
            case c >= 'a' && c <= 'z':
                    if wordPos == maxLen {
                            continue
                    }
                    wordBuf[wordPos] = c
                    wordPos++
            default:
                    if wordPos == 0 {
                        continue
                    }

                    words[string(wordBuf[:wordPos])]++
                    wordPos = 0
            }
            words[string(wordBuf[:wordPos])]++
    }
}

This will make a map with number of occurences of each English word in the file.

It takes ~1min to make this map.

After that keys needs to be sorted:

freq := make([]*string, 2<<25)
for word, count := range words {
    wCopy := word
    freq[count] = &wCopy
}

done := 0
for i := uint64(len(freq) - 1); i > 0 && done < 10; i-- {
    if freq[i] == nil {
        continue
    }
    fmt.Printf("%s: %d\n", *freq[i], i)
    done++
}

This is a terrible way of sorting anything in real life, but it works fast, and that’s exactly what we need in our case. It took ~113ms

Reading file concurrently

The next step is to assume that it could go faster if file is processed in batches concurrently. That is true, but it will also creates data races. To solve it, I will use this implementation of a map with atomic write.

Another way to go from here is to save results from each thread to it’s own map and then merge maps.

func fromFile(filepath string, batchSize int64, tk *count.Stream) error {
        file, err := os.Open(filepath)
        if err != nil {
                return fmt.Errorf("failed to read `%s`: %s", filepath, err)
        }
        defer file.Close()

        info, err := file.Stat()
        if err != nil {
                return fmt.Errorf("failed to stat `%s`: %s", filepath, err)
        }

        all := info.Size() / batchSize
        wg := &errgroup.Group{}
        for i := int64(0); i < all; i++ {
                i := i
                // NOTE: read concurrently and process in batch
                wg.Go(func() error {

                        buff := make([]byte, batchSize)

                        off, err := file.ReadAt(buff, batchSize*i)
                        switch err {
                        case nil:
                        case io.EOF:
                                return nil
                        default:
                                return err
                        }

                        processBatch(buff[:off], tk)

                        return nil
                })
        }

        return wg.Wait()
}

I was using 2<<19-1 as a batchSize. This is based on test runs and nothins else.

tk here contains an instance of github.com/cornelk/hashmap

Law of Large numbers

The last optimization, and the biggest one I’ve made is not completely about programming.

There is this theorem in probablility theory that says:

the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed

That means that if you take a large file with English text and count top 10 words in there, the top will always look almost the same.

Next step here is to google most common words in English and use this list to decrease number of words we count.

By doing that, amount of writes to the map will dramaticaly decrease, what will allow to spend less time waiting for write access to the map.

After implementing this, processing time is ~25s

You can find full code on Github

type Block interface {
    Chain(Block) (string, error)
}

I want to make sure that input Block is always valid, and the output string is never empty if the error is nil. How can I do that? There are a couple of options.

The first option is to rely on every implementation to validate it, but it means a boilerplate code that is easy to miss.

Another option is to create another layer and move all of the logic there. Something like:

type Chain interface {
    Add(Block) error
}

type Block interface {
    Hash() string
}

It’s better, but I don’t want to create another entity.

I can also create an implementation that wraps all other implementations and does all the checks:

type check struct {
    i block
}

func Check(i Block) *check {
    return &check{
        i: i,
    }
}

func (c *check) Chain(b Block) (string, error) {
    hash, err := c.i.Chain(b)

    if hash == "" && err == nil {
        fmt.Println("err is nil, and hash is empty, do something here.")
    }

    return hash, err
}

There is still the same problem: it’s easy to forget to wrap a custom implementation into the check and skip validation.

Likely, I found a way to enforce it during the build time. To do that, Block interface and the check implementation should be slightly changed:

type block interface {
    Chain(Block) (string, error)
}

type Block interface {
    block

    p()
}

func Check(i block) Block {
    return &check{
        i: i,
    }
}

type check struct {
    i block
}

func (c *check) Chain(b Block) (string, error) {
    hash, err := c.i.Chain(b)

    if hash == "" && err == nil {
        fmt.Println("err is nil, and hash is empty, do something here.")
    }

    return hash, err
}

func (c *check) p() {}

Now, it’s impossible to implement Block interface, because there is no way to implement a private method.

The only way to do that is to implement all public methods and call Check function.

You can find full example code on GitHub

The idea of a peer to peer something is to create a system that doesn’t require any centralized server to operate. In the case of a messenger, two users should be able to communicate directly between each other until at least one of their instances is running.

That’s why the most important part of such systems is a discovery. If peers can’t find each other, it’s useless.

The first problem is to build a messenger app, that can discover the same apps in the network and securely communicate with each other.

The second problem is to allow regular people (non-programmers) to use it. As I can see, there are two ways here. First is to build a mobile app and discover peers via Bluetooth or WiFi, plus have some bridge that allows joining two local networks together via the internet.

I chose another way. Instead, there is a command line app for usage within the same network. And a dispatcher app which purpose is to manage a cloud with dockerized peers and create a new instance for every user that needs it.

1. User goes to a dispatcher webpage and clicks Login
2. Dispatcher creates a new peer in a docker container and redirects the user to it

I don’t think that this approach is super scalable and makes sense in the real world; it requires too many resources for a single user. But I have chosen it because it is more fun to implement for me as it needs more infrastructure work.

Peer

Communication

Peers use gRPC to communicate with each other. It has a few benefits:

1. There is a well defined .proto API schema. So it shouldn’t be a problem to build clients using other programming languages.
2. Stream support. Peers can exchange messages over a single TCP connection (thanks to HTTP/2). There is also ongoing work on supporting UDP as a transport. Initially, I wanted to use UDP based communication with QUIC but decided to go with gRPC because of other features.
3. Small things like TLS, custom resolvers, compression, schema versioning, binary marshaling out of the box.

Overall communication scenario between two peers looks like this:

1. Check if there is an open stream connection to the peer
   • If it exists, send a message using the connection
2. Exchange public keys with a peer using an insecure connection
3. Open a secure stream connection to the peer and send a message.

Discovery

In general, discovery is used to find other peers in the network. Discovery message contains name, id, ports, address and a list of known peers. Public key not included in the discovery as it makes discovery message too big to be transmitted via UDP multicast.

It also allows the peer to find its address by listening for its own announcement message.

UDP multicast discovery allows discovery within the same network. Perfect for standalone runs without docker.

Peers also can register itself in Consul catalog and fetch information about other peers from there. Perfect for dockerized runs inside docker swarm as it also can be used by Traefik to automatically create routes for new peers and monitor their health.

Dispatcher

The goal of this service is to start a new peer by connecting to a docker swarm API, and redirect a user to his peer instance.

Thanks to consul peers registration, it can fetch vacant peers directly from consul and use the information to have a small buffer of unused peers. It is needed as a peer takes a few seconds to start and generate a certificate.

The main logic for the service is relatively simple:

1. Check if the user has an assigned peer (it is stored it his cookies)
   • If it is there, redirect to the peer.
2. Pop an unused peer from consul
3. Store its credentials in the user’s session
4. Redirect the user to the peer.

Bridge

This is not implemented part so far, but the idea is to create another app that has two parts: discovery and proxy.

Two bridges in different networks, for example, a cloud network from the scheme above and a local network, should connect and exchange information about known peers.

When a message is sent from peer in the one network to the peer in another, bridge acts as a gateway and proxies the connection between two peers.

Final architecture:

You can find the code and maybe participate at github.

Links:

• Messenger
• GitHub
• Traefik
• Consul

I will try to explain what is a race condition, why does it happen and how to avoid it.

Wikipedia:

A race condition or race hazard is the behavior of an electronics, software, or another system where the system’s substantive behavior is dependent on the sequence or timing of other uncontrollable events. It becomes a bug when one or more of the possible behaviors is undesirable.

Let’s say we have a list of links to Wikipedia pages that we want to download.

I wrote a simple tool as an example that uses the interface to do that:

type Downloader interface {
    Download(...*url.URL) (map[*url.URL][]byte, error)
}

The most straightforward implementation would be to iterate over a list of urls, download each of them and store in the resulting map:

type Downloader struct {
    client *http.Client
}

func (d *Downloader) Download(urls ...*url.URL) (map[*url.URL][]byte, error) {
    result := make(map[*url.URL][]byte, len(urls))
    for _, u := range urls {
        data, err := d.download(u)
        if err != nil {
            return nil, fmt.Errorf("error downloading %s: %s", u, err)
        }
        result[u] = data
    }
    return result, nil
}

func (d *Downloader) download(u *url.URL) ([]byte, error) {
    resp, err := d.client.Get(u.String())
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    return ioutil.ReadAll(resp.Body)
}

The code works, but not so fast, because the total time to download links is a sum of times to download for each link. It’s really inefficient when the number of links is big. For me, it took over 16s to download 100 links. To improve it, let’s change the code, so we download all links at the same time:

func (d *Downloader) Download(urls ...*url.URL) (map[*url.URL][]byte, error) {
    result := make(map[*url.URL][]byte, len(urls))

    wg := &sync.WaitGroup{}
    for _, u := range urls {
        wg.Add(1)
        go func(u *url.URL) {
            data, _ := d.download(u)
            result[u] = data
            wg.Done()
        }(u)
    }
    wg.Wait()

    return result, nil
}

To do that, we have wrapped our download function to run in its goroutine. Wait group here is used to wait until all of the urls are downloaded, so the total time equals the time to download the heavies page. On my test run, it is 679.53154ms for 100 pages. Down from sixteen seconds to less than a second - an impressive result!

But here is a problem:

==================
WARNING: DATA RACE
Write at 0x00c0007baf60 by goroutine 257:
  runtime.mapassign_fast64()
      /usr/local/go/src/runtime/map_fast64.go:92 +0x0
  github.com/ngalayko/talks/concurrency/examples/download/downloader/race.(*Downloader).Download.func1()
      /Users/nikitagalaiko/golang/src/github.com/ngalayko/talks/concurrency/examples/download/downloader/race/downloader.go:37 +0x8e

Previous write at 0x00c0007baf60 by goroutine 262:
  runtime.mapassign_fast64()
      /usr/local/go/src/runtime/map_fast64.go:92 +0x0
  github.com/ngalayko/talks/concurrency/examples/download/downloader/race.(*Downloader).Download.func1()
      /Users/nikitagalaiko/golang/src/github.com/ngalayko/talks/concurrency/examples/download/downloader/race/downloader.go:37 +0x8e

Goroutine 257 (running) created at:
  github.com/ngalayko/talks/concurrency/examples/download/downloader/race.(*Downloader).Download()
      /Users/nikitagalaiko/golang/src/github.com/ngalayko/talks/concurrency/examples/download/downloader/race/downloader.go:35 +0x123
  main.run()
      /Users/nikitagalaiko/golang/src/github.com/ngalayko/talks/concurrency/examples/download/cmd/download/main.go:71 +0x8e
  main.main()
      /Users/nikitagalaiko/golang/src/github.com/ngalayko/talks/concurrency/examples/download/cmd/download/main.go:56 +0x3b4

Goroutine 262 (finished) created at:
  github.com/ngalayko/talks/concurrency/examples/download/downloader/race.(*Downloader).Download()
      /Users/nikitagalaiko/golang/src/github.com/ngalayko/talks/concurrency/examples/download/downloader/race/downloader.go:35 +0x123
  main.run()
      /Users/nikitagalaiko/golang/src/github.com/ngalayko/talks/concurrency/examples/download/cmd/download/main.go:71 +0x8e
  main.main()
      /Users/nikitagalaiko/golang/src/github.com/ngalayko/talks/concurrency/examples/download/cmd/download/main.go:56 +0x3b4
==================

I’ve built the code with -race flag to enable golang race detector and got a warning about the data race. Of cource, a warning is not an error, and we can ignore it, the result was correct: we’ve downloaded a page for each link.

But why race detector does not agree?

First, let’s see what lines he doesn’t like:

    result[u] = data

A map is a data structure that contains keys and values. All keys are unique, and each key has an associated value with it. For the sake of the example let’s say that every key is associated with a particular address in the memory, and the address is permanent.

So when we are updating the key’s value, we are updating data at the same address in the memory.

In our example, we have unique urls, we download each of them and place the content to the individual memory address associated with a map key.

But what should happen when we write different data into the same key? If we follow the abstractions of a map that I’ve described above and goroutines - light-weight threads that are running concurrently at the same time - then the final value for the map is not determined.

Because what happens is that we are telling go to write some data at the same memory address multiple times at the same time. This can cause all kinds of problems - from the unexpected result for a programmer to data corruption. And it’s not a responsibility of a language to decide what does a programmer wants to see as a result. And a result in the situation depends on uncontrollable events.

That’s why go gives us a warning. It can’t guarantee the result.

It doesn’t apply only to maps. The same can happen when you share a pointer between two goroutines. Or you are writing to the same file from two different programmes, or maybe you are saving data from two instances of your applications to the same database. These all are different kinds of the same problem.

Data races happen only when you have multiple threads/goroutines/processes that can access the same data at the same time.

Likely, that problem was around for a while, and there is a solution to it.

You need to make sure that only one process has access to the same piece of memory at a time.

Though, there is a difference between write and read access. Reading the same data from multiple threads is safe. The problems start only when you have a thread that writes.

Golang has a few instruments to deal with data races like mutexes and channels.

Mutexes are used to solve exactly data race problems, here how to use them:

func (d *Downloader) Download(urls ...*url.URL) (map[*url.URL][]byte, error) {
    result := make(map[*url.URL][]byte, len(urls))
    wg := &sync.WaitGroup{}

    guard := &sync.Mutex{}

    for _, u := range urls {
        wg.Add(1)
        go func(u *url.URL) {
            data, _ := d.download(u)

            guard.Lock()
            result[u] = data
            guard.Unlock()

            wg.Done()
        }(u)
    }
    wg.Wait()
    return result, nil
}

It has two methods, Lock and Unlock. Everything between them can be accessed only by one goroutine at a time. Others will wait for the mutex to unlock before they can access it.

Channels are a bit more tricky, and can be also used to controll goroutines:

type done struct {
    u    *url.URL
    data []byte
    err  error
}

func (d *Downloader) Download(urls ...*url.URL) (map[*url.URL][]byte, error) {
    result := make(map[*url.URL][]byte, len(urls))

    doneChan := make(chan *done)

    wg := &sync.WaitGroup{}
    for _, u := range urls {
        wg.Add(1)
        go func(u *url.URL) {
            data, err := d.download(u)
            doneChan <- &done{
                u:    u,
                data: data,
                err:  err,
            }
            wg.Done()
        }(u)
    }

    go func() {
        wg.Wait()
        close(doneChan)
    }()

    for done := range doneChan {
        if done.err != nil {
            return nil, fmt.Errorf("error downloading %s: %s", done.u, done.err)
        }
        result[done.u] = done.data
    }

    return result, nil
}

Data from the multiple goroutines sent to the single goroutine via a channel and that goroutine stores the data in the map.

Links:

• example

1. First 6 or 8 digits is a birthrate with or without a century.
2. Last four digits are random secret digits.
3. The whole number satisfies the Luhn algorithm.
4. Birthdate and secret digits can be divided with - or +.

For example 19900101-0017

Here is the initial code of the package:

package personnummer

import (
    "math"
    "reflect"
    "regexp"
    "strconv"
    "time"
)

// luhn will test if the given string is a valid luhn string.
func luhn(str string) int {
    sum := 0

    for i, r := range str {
        c := string(r)
        v, _ := strconv.Atoi(c)
        v *= 2 - (i % 2)
        if v > 9 {
            v -= 9
        }
        sum += v
    }

    return int(math.Ceil(float64(sum)/10)*10 - float64(sum))
}

// testDate will test if date is valid or not.
func testDate(century string, year string, month string, day string) bool {
    t, err := time.Parse("01/02/2006", month+"/"+day+"/"+century+year)

    if err != nil {
        return false
    }

    y, _ := strconv.Atoi(century + year)
    m, _ := strconv.Atoi(month)
    d, _ := strconv.Atoi(day)

    if y > time.Now().Year() {
        return false
    }

    return !(t.Year() != y || int(t.Month()) != m || t.Day() != d)
}

// getCoOrdinationDay will return co-ordination day.
func getCoOrdinationDay(day string) string {
    d, _ := strconv.Atoi(day)
    d -= 60
    day = strconv.Itoa(d)

    if d < 10 {
        day = "0" + day
    }

    return day
}

// Valid will validate Swedish social security numbers.
func Valid(str interface{}) bool {
    if reflect.TypeOf(str).Kind() != reflect.Int && reflect.TypeOf(str).Kind() != reflect.String {
        return false
    }

    pr := ""

    if reflect.TypeOf(str).Kind() == reflect.Int {
        pr = strconv.Itoa(str.(int))
    } else {
        pr = str.(string)
    }

    re, _ := regexp.Compile(`^(\d{2}){0,1}(\d{2})(\d{2})(\d{2})([\-|\+]{0,1})?(\d{3})(\d{0,1})$`)
    match := re.FindStringSubmatch(pr)

    if len(match) == 0 {
        return false
    }

    century := match[1]
    year := match[2]
    month := match[3]
    day := match[4]
    num := match[6]
    check := match[7]

    if len(century) == 0 {
        yearNow := time.Now().Year()
        years := [...]int{yearNow, yearNow - 100, yearNow - 150}

        for _, yi := range years {
            ys := strconv.Itoa(yi)

            if Valid(ys[:2] + pr) {
                return true
            }
        }

        return false
    }

    if len(year) == 4 {
        year = year[2:]
    }

    c, _ := strconv.Atoi(check)

    valid := luhn(year+month+day+num) == c && len(check) != 0

    if valid && testDate(century, year, month, day) {
        return valid
    }

    day = getCoOrdinationDay(day)

    return valid && testDate(century, year, month, day)
}

Let’s add a benchmark:

func BenchmarkValid(b *testing.B) {
    for i := 0; i < b.N; i++ {
        Valid("19900101-0017")
    }
}

Results:

$ go test -bench=BenchmarkValid$ -benchmem
goos: darwin
goarch: amd64
pkg: github.com/ngalayko/go
BenchmarkValid-4  100000  18498 ns/op  54576 B/op  118 allocs/op
PASS

That’s a lot of resources to validate a string using four simple rules.

There are several problems in the code. I am going to remove it one by one:

Regexp package

The most popular way to optimize functions in go is to move the regexp compilation out from a function:

var (
    re = regexp.MustCompile(`^(\d{2}){0,1}(\d{2})(\d{2})(\d{2})([\-|\+]{0,1})?(\d{3})(\d{0,1})$`)
)

// Valid will validate Swedish social security numbers.
func Valid(str interface{}) bool {
    //...

    match := re.FindStringSubmatch(pr)

    //...
}

Results are not surprising:

$ go test -bench=BenchmarkValid$ -benchmem
goos: darwin
goarch: amd64
pkg: github.com/ngalayko/go
BenchmarkValid-4  1000000  1301 ns/op  309 B/op  13 allocs/op
PASS

Reflect package

Reflection is generally slow and is not made to be used in such cases. Here it is used to check if the input type in int or string. Let’s assume we actually need a function that accepts an interface, and not just a string.

To check the type of an interface you don’t need a reflect package. You can always switch on an interface type:

// Valid will validate Swedish social security numbers.
func Valid(i interface{}) bool {
    switch v := i.(type) {
    case int, int32, int64, uint, uint32, uint64:
        return ValidString(fmt.Sprint(v))
    case string:
        return ValidString(v)
    default:
        return false
    }
}

// ValidString will validate Swedish social security numbers.
func ValidString(s string) bool {
    // ...
}

Results are pretty much the same. I think it’s because of some compiler optimizations where switch v.(type) is syntax sugar for reflect.TypeOf(v), plus undercover all objects know what types they are even when you use them as an interface{}.

$ go test -bench=BenchmarkValid$ -benchmem
goos: darwin
goarch: amd64
pkg: github.com/ngalayko/go
BenchmarkValid-4  1000000  1305 ns/op  309 B/op  13 allocs/op
PASS

Regexp package again

Why do we even need a regexp package here? To validate that string contains only digits and - or - regexp is an overkill.

First, we clean out everything except digits from a string and check it’s length because we know what that length should be (6+4 or 8+4).

To check if a character is a digit, it’s enough to make sure that it’s greater than '0' and less than '9', because all digits have sequential codes in the (ASCII table)[http://www.asciitable.com/].

func ValidString(s string) bool {
    cleanNumber := ""
    for _, c := range s {
        if c == '+' { // `+` is allowed, but we don't need it.
            continue
        }
        if c == '-' { // `-` is allowed, but we don't need it.
            continue
        }

        if c > '9' {
            return false
        }
        if c < '0' {
            return false
        }

        cleanNumber += string(c)
    }
    //...
}

And once we have a string that has only digits, it’s easy to get all parts of it:

    //...

    var (
        century string
        year    string
        month   string
        day     string
        num     string
        check   string
    )

    switch len(cleanNumber) {
    case 10:
        year = string(cleanNumber[:2])
        month = string(cleanNumber[2:4])
        day = string(cleanNumber[4:6])
        num = string(cleanNumber[6:9])
        check = string(cleanNumber[9:])
    case 12:
        century = string(cleanNumber[:2])
        year = string(cleanNumber[2:4])
        month = string(cleanNumber[4:6])
        day = string(cleanNumber[6:8])
        num = string(cleanNumber[8:11])
        check = string(cleanNumber[11:])
    default:
        return false
    }

    //...

Results are a bit better:

$ go test -bench=BenchmarkValid$ -benchmem
goos: darwin
goarch: amd64
pkg: github.com/ngalayko/go
BenchmarkValid-4  1000000  1302 ns/op  208 B/op  34 allocs/op
PASS

Time package

Standard time package is great, but what we need here is just simple validation of a date:

var monthDays = map[int]int{
    1:  31,
    3:  31,
    4:  30,
    5:  31,
    6:  30,
    7:  31,
    8:  31,
    9:  30,
    10: 31,
    11: 30,
    12: 31,
}

// testDate will test if date is valid or not.
func testDate(century string, year string, month string, day string) bool {
    y, err := strconv.Atoi(century + year)
    if err != nil {
        return false
    }
    m, err := strconv.Atoi(month)
    if err != nil {
        return false
    }
    d, err := strconv.Atoi(day)
    if err != nil {
        return false
    }

    if m != 2 {
        days, ok := monthDays[m]
        if !ok {
            return false
        }
        return d <= days
    }

    leapYear := y%4 == 0 && y%100 != 0 || y%400 == 0

    if leapYear {
        return d <= 29
    }
    return d <= 28
}

Benchmark:

$ go test -bench=BenchmarkValid$ -benchmem
goos: darwin
goarch: amd64
pkg: github.com/ngalayko/go
BenchmarkValid-4   2000000  901 ns/op  192 B/op  33 allocs/op
PASS

Strings

Notice that there are still 33 allocations per function call. Where do they come from? It’s because we use string type all the time, string is the same thing as a []byte, but immutable. So every time we call cleanNumber += string(c), allocation happens. It’s impossible to change the string, so new the string is allocated, and both strings are copies there.

Let’s remove string usage:

package personnummer

import "fmt"

const (
    lengthWithoutCentury = 10
    lengthWithCentury    = 12
)

// ValidateStrings validate Swedish social security numbers.
func ValidString(in string) bool {
    cleanNumber := make([]byte, 0, len(in))
    for _, c := range in {
        if c == '+' {
            continue
        }
        if c == '-' {
            continue
        }

        if c > '9' {
            return false
        }
        if c < '0' {
            return false
        }

        cleanNumber = append(cleanNumber, byte(c))
    }

    switch len(cleanNumber) {
    case lengthWithCentury:
        if !luhn(cleanNumber[2:]) {
            return false
        }

        dateBytes := append(cleanNumber[:6], getCoOrdinationDay(cleanNumber[6:8])...)
        return validateTime(dateBytes)
    case lengthWithoutCentury:
        if !luhn(cleanNumber) {
            return false
        }

        dateBytes := append(cleanNumber[:4], getCoOrdinationDay(cleanNumber[4:6])...)
        return validateTime(dateBytes)
    default:
        return false
    }
}

var monthDays = map[int]int{
    1:  31,
    3:  31,
    4:  30,
    5:  31,
    6:  30,
    7:  31,
    8:  31,
    9:  30,
    10: 31,
    11: 30,
    12: 31,
}

// input time without centry.
func validateTime(time []byte) bool {
    length := len(time)

    date := charsToDigit(time[length-2 : length])
    month := charsToDigit(time[length-4 : length-2])

    if month != 2 {
        days, ok := monthDays[month]
        if !ok {
            return false
        }
        return date <= days
    }

    year := charsToDigit(time[:length-4])

    leapYear := year%4 == 0 && year%100 != 0 || year%400 == 0

    if leapYear {
        return date <= 29
    }
    return date <= 28
}

// Valid will validate Swedish social security numbers.
func Valid(i interface{}) bool {
    switch v := i.(type) {
    case int, int32, int64, uint, uint32, uint64:
        return ValidString(fmt.Sprint(v))
    case string:
        return ValidString(v)
    default:
        return false
    }
}

var rule3 = [...]int{0, 2, 4, 6, 8, 1, 3, 5, 7, 9}

// luhn will test if the given string is a valid luhn string.
func luhn(s []byte) bool {
    odd := len(s) & 1

    var sum int

    for i, c := range s {
        if i&1 == odd {
            sum += rule3[c-'0']
        } else {
            sum += int(c - '0')
        }
    }

    return sum%10 == 0
}

// getCoOrdinationDay will return co-ordination day.
func getCoOrdinationDay(day []byte) []byte {
    d := charsToDigit(day)
    if d < 60 {
        return day
    }

    d -= 60

    if d < 10 {
        return []byte{'0', byte(d) + '0'}
    }

    return []byte{
        byte(d)/10 + '0',
        byte(d)%10 + '0',
    }
}

// charsToDigit converts char bytes to a digit
// example: ['1', '1'] => 11
func charsToDigit(chars []byte) int {
    l := len(chars)
    r := 0
    for i, c := range chars {
        p := int((c - '0'))
        for j := 0; j < l-i-1; j++ {
            p *= 10
        }
        r += p
    }
    return r
}

Final result:

$ go test -bench=BenchmarkValid$ -benchmem
goos: darwin
goarch: amd64
pkg: github.com/ngalayko/go
BenchmarkValid-4  20000000  94.0 ns/op  16 B/op  1 allocs/op
PASS

If you have an idea how to improve it more, please share.

I am renting virtual servers for a long time now to host different personal projects, including this website. I tried all of them: AWS, Google Cloud, DigitalOcean , Azure, and I was entirely satisfied with them until two weeks ago. Two weeks ago I came across the article where the guy described how easy is to set up a RaspberryPI cluster powered by Docker Swarm. This idea seemed exciting to me, so I ordered all the equipment and spent some time to set it up and move most of the services from DigitalOcean servers to my living room. During that time I faced a lot of unexpected difficulties and read most of the articles where people described how to set up the Docker Swarm cluster, but none of them completely covered everything I wanted to do, so here is another one.

Hardware Used:

• 4 x microSDHC 32 Pro Class 10
• 4 x Layer case for Raspberry Pi
• 4 x microUSB cable
• 5 x Gigabit Ethernet patch cable
• 1 x USB Charger for 6 ports
• 4 x Raspberry PI 3 Model B+
• 1 x 8-Port Gigabit Switch

Technologies used:

• HypriotOS - operating system for Raspberry Pi with preinstalled docker
• Docker Swarm - container orcestrator
• Minio - s3 compatible object storage
• rexray/s3fs - docker volume driver for s3 system
• s3fs (v1.8.4) - itself
• Prometheus / Grafana - for monitoring
• Traefik - reverse proxy and load balancer

Software

ARM

First of all, you should be aware that ARM architecture that Raspberry PI is based on is a pain in the ass. Almost none of Docker images that you find are going to work as you would expect, so that be ready to rewrite docker files and recompile binaries. What really helps here is the --resolve-image never flag for deployment. Because sometimes even if the image uses ARM compatible architecture, docker think that it’s amd64. You can find an example of a simple deployment script here Here is the list of images I found good as a base:

• apcheamitru/arm32v7-alpine - for alpine based images. Use this one when possible, it’s much smaller.
• arm32v7/debian:7.11-slim - for Debian based images and when you are can’t/lazy to use alpine.

Docker Swarm

Initially, I planned to use Kubernates for the cluster orchestrator, but change that for Docker Swarm, because it turns out much more straightforward to set up. However, it has some disadvantages according to DevOps experts on the Internet, one that I found is that if you want to have shared storage for your volumes across the cluster, you are going to waste a week to find a solution that works on ARM and it will just cause more problems, because it’s super slow.

Traefik

For routing inside of the cluster, I use Traefik - a modern reverse proxy. What I liked about it has an integration with the docker API, so you don’t need to describe all of your routes in a configuration file, you just put all containers in the same network and label them in the same compose file where you describe the container.

Prometheus, Grafana

The first thing you want to do after you have a working cluster is monitoring, so you know how much resources are in use and what’s the best way to distribute them across the cluster.

As a source of data for the dashboards I use:

• node-exporter This provides the necessary node information like overall usage of RAM / SPU / SSD , disk read/write rate, network activity, etc.
• arm-exporter I use this to have information about Pi’s temperature. It has a strict correlation with CPU usage, but you want to make sure it’s never higher than 50C.
• cadvisor It provides information from containers point of view. How much of the allocated resources are in use by a service. For example, I have a limit of 256MB of RAM for Prometheus, so it goes down every day or so because OOM killer restarts it once the RAM usage crosses the limit. Thanks to this exporter I am aware of this and can increase the limit or do nothing (I do nothing)

Minio

After I configured the basic cluster, I faced a serious problem. Every time I update the cluster, and Docker Swarm decides to move a service to another node, your persistent data is lost, because by default Swarm supports only local volumes, it means that volume is mounter to the current node and you have to bother yourself syncing it between multiple nodes if you need to.

To have a shared volume, I have chosen Minio cluster and rexray as an s3fs volume driver. s3fs is a file system from Amazon that allows you to mount a bucket from s3 to the disk and store data there. Rexray allows you to use s3fs to create volumes. However, since I wanted to go full self-hosted, Amazon is not an option. That’s why I use minio since it has fully compatible API with Amazon’s s3. To configure all of that I had to build the Minio image because the one in their docker registry is one year old and didn’t work because it wasn’t fully UNIX compatible (mkdir didn’t work). To make it work, you need an s3fs version higher than 1.82, and latest fuse version that you need to compile yourself. Also , you need a rexray plugin built for the ARM platform. You can find all of the images in my docker hub. However, I should warn you that when you manage to set this up, you find out that write speed is extremely slow and the solution for persistence is to bind a service to a node using placement constraints.

Migration

I used to deploy all services using docker compose. You can read about it here. The first step was to rewrite all compose files to version 3, so I can reuse them for Swarm deployment. The second step is to set up a cluster and expose it to the Internet. To avoid downtime, I created a test domain that was a base domain for all cluster services while I was testing it. When cluster was ready for “production”, I changed DNS settings and pointer the real domain to cluster IP, so after DNS cache was updated, all users were migrated to the new cluster without noticing (let’s pretend I have more than 20 daily visitors and they care about this blog).

Next plans are to set up DNS over HTTPS and VPN over HTTPS on the same cluster.

Links

• github repo

Problem

You are given an M by N matrix consisting of booleans that represents a board. Each True boolean represents a wall. Each False boolean represents a tile you can walk on.

Given this matrix, a start coordinate, and an end coordinate, return the minimum number of steps required to reach the end coordinate from the start. If there is no possible path, then return null. You can move up, left, down, and right. You cannot move through walls. You cannot wrap around the edges of the board.

For example, given the following board:

[
  [f, f, f, f],
  [t, t, f, t],
  [f, f, f, f],
  [f, f, f, f],
]

and start = (3, 0) (bottom left) and end = (0, 0) (top left), the minimum number of steps required to reach the end is 7, since we would need to go through, (1, 2) because there is a wall everywhere else on the second row.

Solution

It is a new type of problems I faced. I remember, I solved some during university, but it was pretty hard to come up with the solution right away.

I googled basic types of maze solving algorithms, and it looks like Lee algorithm will be a pretty good choice in most of the shortest path problems since at the end of the day a number of different paths in a maze makes a tree.

The idea is deadly simple:

1. go to start cell, mark it 0.
2. mark all neighbors as +1. It is a distance to the starting cell
3. make the same for each of the neighbors

By running this algorithm for each cell, we will get the number of steps it takes to get to any other point from the start. Of course, we should ignore walls and previously marked cells on each iteration.

This is a basic solution and can be optimized for a given problem. For example, we can stop our recursive calls once we meet finish cell.

Code

// the maze is a matrix that represents a maze.
// all cells have value 0, and all walls have value 1.
// start and finish are arrays of 2 elements, [i,j] of the cells.
func solution(maze [][]int, start []int, finish []int) int {
        // mark start cell is -1
        maze[start[0]][start[1]] = -1

        // mark all cells starting from start recursively
        mark(maze, start, 0)

        // return value of a finish cell
        return maze[finish[0]][finish[1]]
}

// pos is a structure to hold cell coordinates,
// because []int can't ba used as a map key ¯\_(ツ)_/¯
type pos struct {
        i int
        j int
}

// mark marks all neighbors of a given cell with n+1
func mark(maze [][]int, point []int, n int) {
        i, j := point[0], point[1]

        neighbors := map[pos]bool{
                pos{i + 1, j}: false,
                pos{i - 1, j}: false,
                pos{i, j - 1}: false,
                pos{i, j + 1}: false,
        }
        for p := range neighbors {
                neighbors[p] = markP(maze, p.i, p.j, n+1)
        }

        for p, ok := range neighbors {
                if ok {
                        mark(maze, []int{p.i, p.j}, n+1)
                }
        }
}

// markP used to mark maze[i][j] with given n if exists and not marked.
// returns true if it was marked, otherwise false.
func markP(maze [][]int, i, j, n int) bool {
        if i >= len(maze) || j >= len(maze[0]) {
                return false
        }
        if i < 0 || j < 0 {
                return false
        }
        if maze[i][j] != 0 {
                return false
        }

        maze[i][j] = n
        return true
}

Links

• github

PiHole

It started when someone shared a link to the PiHole on Twitter. It is a self-hosted DNS service that is designed for RaspberryPi for blocking advertisements and trackers on the DNS level.

Turns out, about 20% of queries I make are blocked, and it doesn’t hurt daily usage at all. Instead, now I have nice stats of a website I visit and handy tools to block/whitelist some of them.

You can install it directly on your router, on every device that you use. For MacBook and AppleTV I changed DNS settings to use my custom IP address, for iPhone, I found this app that can change DNS for all requests, even when you use cellular. If you want to try it, the address is the same as the IP of this website 167.99.219.223

Matomo

After enabling PiHole, Google Analytics stopped working, so I found a self-hosted alternative and installed it on this website.

Matomo is pretty good. It has all essential analytics features and respects user privacy. For example, I configured it anonymize last digits of user IP. For example, 167.99.219.223 will be changed to 167.99.219.000 before saving.

If you don’t want it, you can configure a browser to send “Do not track me” requests to websites.

VPN

And, of course, VPN. I hope everyone knows what it is.

This time I use Libreswan and xl2tpd setup.

If you are going to China, try streisand. VPN detecting and blocking technology there is the next level, something simple will be banned within a couple of days for sure.

Links

• PiHole
• docker-compose file for PiHole
• domains to block
• iOS app to change dns
• Matomo
• docker-compose file for Matomo
• docker-compose file for VPN
• streisand vpn

Since most of the times things I want to host are useless and I often change my mind, there are several requirements for it:

1. Easy to add/remove new components.

   Let’s say I created a website, and a couple of days later I added a DNS server on the same host, and then I went to Russia, so I also need to host VPN. Later I realize I don’t really need DNS server, so I want to remove it.

   Doing these things, I want to do only them. I don’t really want to fix website deployment while deploying VPN server, and accidentally remove VPN when stopping DNS server.

2. No vendor lock.

   I want to remove, or start a new server with the same configuration on different hosting provider any time I want. Digital Ocean, AWS, Google Cloud, my old computer - whatever.

3. Automate as much as possible.

   Deployment, https certificates for subdomains, restarting failed instances, etc. I don’t want to care about this at all.

In the beginning, I was setting up Nginx on the host and Let’s Encrypt certificates update. So to add or remove something new I had to change nginx configuration for a hole server, what could lead to crashing of all components because they are exposed to the world via same Nginx. Also, I need new subdomain certs, because Let’s encrypt didn’t have wildcards back then.

My first attempt to automate it was this. Ansible roles for each component (usually dockerized) and Makefile to execute them. The main problem was that I had my own CI server running (for some reason). That’s why if it crashes, you should go and set it up back manually, what is always a pain in the ass and takes some time. So a after couple of crashes, it got boring, I stopped care about it and deleted host.

A couple of weeks ago I started this blog, so I tried to find a way to manage a host server one more time.

Here it is.

There are 2 components:

1. Automated nginx-docker-with-lets-encrypt compose file

   What it does is generating nginx configuration based on other containers in the same docker network and taking care of https for them. You can read more in the repository readme file, but basically, it contains three parts: nginx, nginx configuration generator and a certificates generator using Let’s Encrypt.

   That allows us to add new components easily - just add new service to compose file or remove one. Also, does not depend on host provider, because can be run on pretty much any operating system.

2. Travis deployment

   It logs in via ssh to your remote server and runs deployment script there. Step-to-step explanation on how to set it up you can find here. Only change I made there - added environment variables export, so it’s possible to strore secret keys in Travis.

That’s it! Travis executes all compose files in the folder, and removes orphan containers.

To add a new component to the system, I need to create a new folder or add git submodule with a Dockerized app and add a docker-compose file to run it, following some rules, so nginx container can find it and create routes.

Links

• Travis deployment configuration
• Docker Let’s Encrypt Nginx proxy companion

I started using vim because I was bored. Likely, I code in go, so it’s pretty easy to switch. I mean, I tried to do Java with vim for about a week - and it’s a hell. So if your primary language requires massive IDE support, it’s not an option to completely switch to vim.

After slightly more than half a year of usage, I improved typing, navigation speed, and general comfort while I code.

Before I was using a mouse to navigate between files, scrolling them to find something, But with vim, your movements are minimal and efficient.

Also, it makes you think a bit more. Not only because you need to remember a lot of commands and combinations, but because you always try to do something by pressing the minimum number of keys possible.

I am going to describe my experience on how to start and make an overview of the configuration I use.

How to start

Basics

As a first step, to understand the basics, I suggest trying embedded vim tutor. You can run it by:

$ vimtutor

It should be installed by default.

It’s a text file divided into lessons which will teach you to navigate, edit, search and more - basics enough for daily use. I was doing this tutorial for a couple of weeks until I was confident with each lesson.

Once you got used to hjkl navigation, stop doing it. The main power of vim is navigation and clicking hhhhhh to get to another line, for instance, is super inefficient. It can be as hard as switching to vim, but you will not regret. Here is a good video about it.

Configuration

Some people suggest to use vanilla vim to start with, so you can understand which functions you luck of comparing to IDE/editor you used to and do not install a lot of useless plugins.

However, I think it makes sense to find the most popular configuration for a language you use on GitHub and use it as a base.

After some time you most likely will optimize it for your usage. I did exactly same, started with vim-go-ide and then forked it.

Configuration overwiew

Here is a list of plugins I use:

• gruvbox - really good color scheme (I prefer dark version)
• deoplete.nvim - fast-auto completion, the first thing to have when you used to IDE
• deoplete-go - go specific completion options. By default it can complete only words from the same file, recently used words, file paths
• nvim-yarp - required by deoplete in order to work with vim8
• vim-hug-neovim-rpc - same
• nerdtree - the best plugin to navigate in a directory tree. Has a lot of options from creating/deleting a file to some super weird I don’t use
• vim-fugitive - super powerful git integration
• nerdcommenter - smart commenting selected lines/parts of code
• vim-go - the main plugin if you code in Go. Includes a lot of commands, syntax checking, syntax highlighting, go tools. Make sure you read documentation
• auto-pairs - automatically close paired symbols e.x. brackets
• tagbar - overview of source code. variables / functions / classes / etc.
• fzf - if you still use grep or ag, check this out
• fzf.vim - better fzf integration for vim.
• Dockerfile.vim - Dockerfiles syntax highlighting
• vim-signify - hightlighting of changed lines of code for any vcs
• nginx.vim - nginx syntax highlighting

Links

• My fork of vim-go-ide
• How to Do 90% of What Plugins Do (With Just Vim)
• Improving Vim Speed

Problem

This problem was asked by Facebook:

Given a stream of elements too large to store in memory, pick a random element from the stream with uniform probability.

Solution

There are many variations of such problems, and before solving it, I want to show some basic examples that I met.

Most trivial one is picking one random element from an array.

Every programming language has a function to generate a pseudo-random number (int or float) within the given range. If you think of an array A as of N numbers, it’s clear how to pick random one: generate number i from 0 to N, and a[i] is the answer.

func oneRandom(a []int) int {
    i := rand.Intn(len(a))
    return aa[i]
}

Sometimes you need to pick N random elements from an array.

In this case, you can use the same approach and pick N indexes from 0 to len(A).

func nRandom(a []int, n int) []int {
    result := make([]int, n)
    for i := 0; i < n; i++ {
        result[i] = oneRandom(a)
    }
    return result
}

However, if you need to pick N random elements, they also have to be different.

In this case, you can pick N different indexes and make sure that they are different:

func nDifferentRandom1(a []int, n int) []int {
    m := map[int]bool{}
    for len(m) != n {
        j := rand.Intn(len(a))
        m[j] = true
    }

    result := make([]int, n)
    for i := range m {
        result[i] = a[i]
    }
    return result
}

But it’s not very efficient, because you can spend much time trying to pick the last index when you need to pick 8 elements from an array of length 10.

In this case, you can use the well-known zero-allocation algorithm to do so. The idea is to move elements you picked to the beginning of an array, and choose from others next:

func nDifferentRandom2(a []int, n int) []int{
    for i := 0; i < n; i++ {
        chooseFrom := a[i:]                            // define a slice to pick from
        choosenIndex := rand.Intn(len(chooseFrom))     // pick a random index from i to N
        a[i], a[choosenIndex] = a[choosenIndex], a[i]  // swap choosenElement with i
    }
    // after all, we have choosen elements at the
    // begining of an array, and probability is always same.
    return a[:n]
}

Let’s return to problem #15 and try to solve it using previous examples.

What do we have:

1. a stream of elements too large to store in memory
2. should pick 1 random element

We understand that to pick a random element from the stream with uniform probability, we need to process hole stream somehow (without storing to memory).

Well, we always can iterate over a stream (but just once).

If we knew the length of a stream, we could use the approach from the first example, that would be perfect.

In first example each element of an array have a linked number - it’s index. Moreover, we used a function that returns number from 0 to n with uniform probability within that range to pick one.

So, in this case, we can assign a random value to each element with the same probability, and choose between them based on it linked number.

Let’s do it: for each element, we generate a float number between [0..1]. Also, we remember maximum value that we generated, an element from the stream associated with it - random element.

Code

func solution(in <-chan int) <-chan int {
    res := make(chan int, 1)
    go func() {
        var result int
        var lastprob float64
        for a := range in {
            prob := rand.Float64()
            if prob > lastprob {
                result = a
                lastprob = prob
            }
        }
        res <- result
    }()
    return res
}

Links

• github

Hardest problem to solve so far: describing 30min trivial problems turned out to be much less fun than I imagined.

• Problem #8: binary tree (fun, but always same)
• Problem #9: array operations (boring and always same)
• Problem #10: go basics (3 lines wtf)
• Problem #11: a search tree
• Problem #12: recursive algorithm (always same, sometimes hard to detect)
• Problem #13: string operations (super lame)

I will keep updating GitHub, but not sure about the website.

Links

• github

This problem was asked by Facebook.

Given the mapping a = 1, b = 2, … z = 26, and an encoded message, count the number of ways it can be decoded.

For example, the message ‘111’ would give 3, since it could be decoded as ‘aaa’, ‘ka’, and ‘ak’.

You can assume that the messages are decodable. For example, ‘001’ is not allowed.

Solution

Firstly, let’s take care of a mapping. So if a = 1, then charCode = code(char) - code('a') + 1, it’s is possible, because in ASCII table letters of Latin alphabet are located one by one. In the example, I use function f(string) that returns 1 if a string can be decoded, otherwise 0.

Most of the string parsing problems are recursion based. To start with such a solution, it’s always helpful to manually solve some trivial cases, trying to use the results of a previous case:

If the length of a string is 1, there is always 1 way to decode it, so it’s our base case.

'1':
    ['1']

----------
F('1') = 1

If the length is 2, we always have 1 way with all digits separately, plus one if a number is less than 26, we also use this one as a base case.

'12':
    ['1', '2']
    ['12']
---------------------
F('12') = f('12') + 1

If the length is 3, we can use the results of previous calculations, because we already know how to deal with shorter strings.

F('123') = f('1') * F('23') + F('12') * f('3') = 3

All next cases can be calculated using previously defined:

F('4123') = f('4') * F('123') + f('41') * F('23') = 3

Code

func solution(s string) int {
        firstZero := s[0] == '0'
        l := len(s)
        switch {
        case l == 1:
                return canDecode(s)
        case l == 2:
                if firstZero {
                    // endge case for strings like '01'
                    return canDecode(s)
                }
                return canDecode(s) + 1
        default:
                return canDecode(s[:1])*solution(s[1:]) +
                        canDecode(s[:2])*solution(s[2:])
        }
}

// returns 1 if possible to decode string.
func canDecode(s string) int {
        if s[0] == '0' {
            return 0
        }
        i, err := strconv.ParseInt(s, 10, 64)
        if err != nil {
                return 0
        }
        if i >= 0 && i <= 26 {
                return 1
        }
        return 0
}

Links

• github

This problem was asked by Jane Street.

cons(a, b) constructs a pair, and car(pair) and cdr(pair) returns the first and last element of that pair. For example, car(cons(3, 4)) returns 3, and cdr(cons(3, 4)) returns 4.

Given this implementation of cons:

def cons(a, b):
  def pair(f):
    return f(a, b)
  return pair

Implement car and cdr.

Solution

The main difficulty with functional programming when you are used to object-oriented is that you don’t have clear names for things that are happening (try to explain to anyone what monad is).

Even if you never tried functional programming, it’s is possible to guess how python solution looks like. I will just put it and move to the next one in go.

def cons(a, b):
    def pair(f):
        return f(a, b)
    return pair

def car(f):
    def left(a, b):
        return a
    return f(left)

def cdr(f):
    def right(a, b):
        return b
    return f(right)

print(car(cons(3,4)))
print(cdr(cons(3,4)))

So most of the functional programming is about passing a behavior to a function.

When in the object-oriented paradigm you have and interface and two implementations, in functional you have a function (interface) that accepts another function (implementation) and calls it inside.

This particular problem is all about it.

We have cons - high order function that accepts two integers and returns another function, which accepts the third function that knows how to operate with this integers. Sounds crazy, but let’s try to give names to all these kinds of function.

First of all, we need to define choice, that knows how to pick one integer out of two.

The function that is returned by given implementation of cons I called makeChoice. This type of function knows what to do with given choice.

Given function cons constructs makeChoice function using input integers.

We should implement two functions that will apply two kinds of behavior using given makeChoice.

Now when we can name function types and understand what they do, it’s much easier to write the solution.

Code

package main

import "fmt"

func main() {

        car := car(cons(3, 4))
        cdr := cdr(cons(3, 4))

        fmt.Printf("car: %d, cdr: %d\n", car, cdr)
}

type choise func(int, int) int

type makeChoise func(choise) int

func cons(a, b int) makeChoise {
        p := func(i choise) int {
                return i(a, b)
        }
        return p
}

func car(do makeChoise) int {
        chooseRight := func(a, b int) int {
                return a
        }
        return do(chooseRight)
}

func cdr(do makeChoise) int {
        chooseLeft := func(a, b int) int {
                return b
        }
        return do(chooseLeft)
}

Links

• github

This problem was asked by Stripe.

Given an array of integers, find the first missing positive integer in linear time and constant space. In other words, find the lowest positive integer that does not exist in the array. The array can contain duplicates and negative numbers as well.

For example, the input [3, 4, -1, 1] should give 2. The input [1, 2, 0] should give 3.

You can modify the input array in-place.

Solution

To solve this, you should think what’s common between array indexes and a positive integer: it’s the same thing.

So we put each positive integer of an array at its place (i+1 since we count from 1) and then iterate again to find first missing.

If we don’t, return length plus one (=next).

Code

func solution(aa []int) int {
    if len(aa) == 0 {
        return 1
    }
    for _, a := range aa { // try to place all numbers at same index (from 1)
        if a < 0 { // we don't care, it's negative
            continue
        }
        if a >= len(aa) { // we don't care, result always < len(aa)
            continue
        }
        aa[a-1] = a
    }
    for i := range aa { // find first missing
        if aa[i] != i+1 {
            return i + 1
        }
    }
    return len(aa) + 1 // all there
}

Benchmarks

goos: darwin
goarch: amd64
pkg: github.com/ngalayko/dcp/problems/2018-07-05
Benchmark/0-4   1000000000               2.18 ns/op            0 B/op          0 allocs/op
Benchmark/100-4                 20000000                65.0 ns/op             0 B/op          0 allocs/op
Benchmark/200-4                 10000000               122 ns/op               0 B/op          0 allocs/op
Benchmark/300-4                 10000000               178 ns/op               0 B/op          0 allocs/op
Benchmark/400-4                 10000000               238 ns/op               0 B/op          0 allocs/op
Benchmark/500-4                  5000000               295 ns/op               0 B/op          0 allocs/op
Benchmark/600-4                  5000000               354 ns/op               0 B/op          0 allocs/op
Benchmark/700-4                  3000000               410 ns/op               0 B/op          0 allocs/op
Benchmark/800-4                  3000000               466 ns/op               0 B/op          0 allocs/op
Benchmark/900-4                  3000000               528 ns/op               0 B/op          0 allocs/op
PASS
ok      github.com/ngalayko/dcp/problems/2018-07-05     19.293s

UPDATE

As it was mentioned in the comments, the first solution fails in case of [3,2,4,-1,1]

It happens because when we place an integer to its position in the array, we also delete integer that used to be in that place.

To avoid this, instead of just placing the integer, I swap it with the current one and process current index one more time.

func solution(aa []int) int {
    if len(aa) == 0 {
        return 1
    }
    for i := 0; i < len(aa); i++ {
        a := aa[i]
        if a < 0 { // we don't care, it's negative
            continue
        }
        if a >= len(aa) { // we don't care, result always < len(aa)
            continue
        }
        if a == aa[a-1] { // if integer on it's place, skip
            continue
        }
        // put each integer on it's place
        // decrease i, because aa[i] is a new integer now and we need to
        // process it one more time
        aa[i], aa[a-1] = aa[a-1], aa[i]
        i--
    }
    for i := range aa { // find first missing
        if aa[i] != i+1 {
            return i + 1
        }
    }
    return len(aa) + 1 // all there
}

Links

• github

This problem was asked by Google:

Given the root to a binary tree, implement serialize(root), which serializes the tree into a string, and deserialize(s), which deserializes the string back into the tree.

For example, given the following Node class

class Node:
  def init(self, val, left=None, right=None):
    self.val = val
    self.left = left
    self.right = right

The following test should pass:

node = Node('root', Node('left', Node('left.left')), Node('right'))
assert deserialize(serialize(node)).left.left.val == 'left.left'

Solution

This one is weird. It doesn’t say we can’t use standard library, so I will just use it.

Code

type node struct {
    Value string `json:"value"`
    Left  *node  `json:"left"`
    Right *node  `json:"right"`
}

// New is a node constructor.
func New(v string, l, r *node) *node {
    return &node{
        Value: v,
        Left:  l,
        Right: r,
    }
}

// Serialize returns string representation of a node.
func (n *node) Serialize() (string, error) {
    b, err := json.Marshal(n)
    return string(b), err
}

// Deserialize returns node from it's string representation.
func Deserialize(s string) (*node, error) {
    n := new(node)
    return n, json.Unmarshal([]byte(s), n)
}

Links

• github

This problem was asked by Uber:

Given an array of integers, return a new array such that each element at index i of the new array is the product of all the numbers in the original array except the one at i.

For example, if our input was [1, 2, 3, 4, 5], the expected output would be [120, 60, 40, 30, 24]. If our input was [3, 2, 1], the expected output would be [2, 3, 6].

Follow-up: what if you can’t use division?

Solution

For this one, I have 2 solutions, with and without division. Follow-up in the problem text should mean that it’s easier to solve it if you don’t have division, but in reality, division causes many problems, since you can accidentally divide by zero.

First

Without edge cases it’s pretty straightforward: get the product of all array elements, then in the loop, divide that product by each element. As a result, you get the product of all elements except for the current one.

func solution_no_edge_cases(aa []int) []int {
    result := 1
    for _, a := range aa {
        result *= a
    }
    for i := range aa {
        aa[i] = result / aa[i]
    }
    return aa
}

a * b * c / a = b * c
a * b * c / b = a * c
a * b * c / c = a * b

Only problem - zeros. We have 2 edge cases there:

1. One zero in an array.

   In this case, the result should be all zeroes except for the 0 elements. That element should contain the product of all others.

2. Two zeros in array or more.

   This case is pretty much like the first one, but result array always contains only zeros. Because you there are always 0 elements in the product.

To avoid edge cases, we calculate the product of all elements except zero and also count them.

So if we have two or more zeros in the input array, we can return an array of 0 right away.

Otherwise, we iterate over an array to replace zero elements to the product we calculated and all other elements to zero.

If we don’t have zeros in an array, all these conditions are skipped, and we get the correct result.

Code

func solution(aa []int) []int {
    if len(aa) < 2 {
        return aa
    }
    result := 1
    countZero := 0
    for _, a := range aa {
        if a == 0 {
            countZero++
            continue
        }
        result *= a
    }
    if countZero > 1 {
        return make([]int, len(aa))
    }
    for i := range aa {
        if aa[i] == 0 {
            aa[i] = result
            continue
        }
        if countZero == 1 {
            aa[i] = 0
            continue
        }
        aa[i] = result / aa[i]
    }
    return aa
}

Second

The second solution is more complicated because it’s always O(n^2), but there are no edge cases here. Just use common logic from yesterday Problem #1, and get production of each elements pairs, except with itself.

Code

func solution2(aa []int) []int {
    if len(aa) < 2 {
        return aa
    }
    result := make([]int, len(aa))
    for i := range aa {
        v := 1
        for j, aj := range aa {
            if i == j {
                continue
            }
            v *= aj
        }
        result[i] = v
    }
    return result
}

Links

• github

This problem was recently asked by Google:

Given a list of numbers and a number k, return whether any two numbers from the list add up to k.

For example, given [10, 15, 3, 7] and k of 17, return true since 10 + 7 is 17.

Bonus: Can you do this in one pass?

Solution

If we want to find the sum of every combination of two array elements most obvious way is to create two for loops over an array and check if they satisfy our condition.

However, for such cases, you can always reduce complexity from O(n^2) to O(log(n)) by just starting the second loop from current array element, because, on each step of the first loop, all previous elements are already compared to each other.

So the solution is to iterate over an array, and for each element try to find if it adds up any next array element up to 17.

Code

func solution(aa []int, k int) bool {
    for i, a := range aa {
        rest := k - a
        for j := i; j < len(aa); j++ {
            if aa[j] == rest {
                return true
            }
        }
    }
    return false
}

Links

• github

For a long time, I wanted to start solving problems from leetcode to keep fit, but could never begin.

Also, for a long time, I wanted to start a blog, but could never begin.

Writing short articles about how I solve these problems seems for me as an excellent opportunity to start doing both, so every day I will publish my solution for the daily problem here, and to my GitHub page trying to explain why and how I solved it.