Michal Charemza

I don't use the term technical debt

2026-01-04T08:35:00Z

I generally avoid referring to anything as technical debt. While not an entirely meaningless term, I suspect that in almost all contexts there are better things to say. In this post I argue why I think the term shouldn't be used, and what should be said instead.

It means too many things

I've heard the term technical debt being used for:

Bugs that exist right now
Properties that risk introducing bugs with future development
Properties that risk slowing future development
Not having some form of quality assurance or tests
Not using a consistent architecture
Limits of scalability of a system in terms of users/usage
Things that mean security isn't as tight as it could be
Things that mean performance/efficiency isn't as good as it could be
Even just regular not-implemented user-facing features

Putting all this together "technical debt" means virtually any property of a system that can be improved in any way! It's far too vague to have meaningful conversations about it in general.

It sounds absolute

The debt metaphor makes it sound absolute: for a financial debt, who can argue that it exists and it has to be dealt with. But this is engineering dealing with humans, trade-offs and risks, all of which can change over time when situations evolve or when information is revealed, and so very much not absolute.

You may never feel the effects of the debt

Even if limiting to what I think are the most typical definitions — properties of code that slow development or risk more bugs when code is changed — it would only involve a subset of all possible changes. If the need for those changes doesn't arise then you never realise the effect of the so-called debt. This is where the debt metaphor really breaks down: other than by taking drastic actions, most of us can't escape the consequences of financial debt, but this is not true for technical debt.

It can lead to wishful thinking

I think because the term debt sounds like it will be addressed, it can lead to wishful thinking, which itself leads to affecting decisions on what you do today. Veering slightly from the point of this post into avoiding "technical debt", I very rarely make a change that is contingent on a future task in order to be acceptable.

The term can sound so negative it can lead to over-engineering

On the other hand, I suspect sometimes technical debt sounds so negative, the avoidance of it can lead to over-engineering solutions, rather than considering the fact that if something is needed later, it often can and should be done later.

It's not obviously linked to real-world consequences

From what I know the term was coined to communicate technical concepts to non-technical people. But because the term is so vague, and I think still sounds quite technical, I think this ironically makes it harder to convince that things labelled (just) as "technical debt" have real-world consequences and so should be avoided or addressed.

It can suggest an ideal architecture independent of constraints

If you don't have the resources — for example time/knowledge/money — to make an architecture, then it's clearly not the perfect architecture for the situation at hand. If it can't exist, it's not ideal in any meaningful sense, so labelling non-achievement of such an architecture as "technical debt" can be a waste of words, time, or even emotions.

What to say instead?

Say the thing: bug, scalability limitation, performance issue (ideally with at least rough numbers), security issue (stating how severe or how it would be exploited), development productivity issue (around what specific classes of features), missing user-facing feature, lack of quality assurance, and ideally along with the consequences of not addressing it. It's such a clearer starting off point for discussions around if/when it should be addressed or not.

For changes that allow you to subsequently address a need, for example some types of refactorings, make sure to say that need when talking about it: "doing X will allow us to easily or more safely do Y" for example. Make sure it's clear that you're aware that refactoring is often necessary, but is never an end unto itself.

And finally, explain rather than apologise in almost all cases! Achieving requirements within constraints is something to be proud of, even if some decisions led to one of the many things that could be labelled as "technical debt". If you've not done X, Y, or Z because they weren't as high priority as other things, that is something to feel good about, not guilty (as long as you haven't made it meaningfully harder to do those things if/when they are needed). Apologise only if you've made a mistake and would not take the same steps next time.

It's not done if...

2025-10-19T21:30:00Z

It can be extremely tempting to announce that something is done, essentially because it can make you and others smile and feel good—we've made progress! But if there are still steps to be taken it's almost definitely not going to be seen as done from the point of view of the person or people you're talking to. Saying something is done when it's not at best is not helpful, but at worst makes you seem dishonest.

Here are some examples of what not done can look like, depending on the team or task:

If there are remaining tests to write or fix
If QA/review/approval is pending
If it's not yet released to production
If no one outside of the team can use it
If there is documentation to write
If we're waiting on confirmation that it does solve the problem

If any of these are the case, I suggest using some other language. For example, you can be specific on what's been completed and what's outstanding. And if you're not sure what's outstanding: ask!

You can sometimes say almost done—it is safer than done. But I suggest only saying this occasionally, and only if you have a high level of confidence that it really is extremely close to getting over the line. If you overuse almost done, it will inevitably not be true in some cases, and people will lose trust in you.

Of course, teams should probably have an agreed "definition of done". But even if there isn't one here's one that's applicable to many situations:

It's done when there are no more steps to take

It's not a hack to satisfy known requirements

2025-10-04T14:10:00Z

I so frequently see extremely capable software engineers waste their skills with over-engineering, and it makes me feel sad. Here are some tips to help avoid the over-engineering trap, and so also to help keep me happy 😀.

Find out what is seen as valuable

We're not here to write code, but to solve problems. Find exactly what problems you are expected to solve and focus on those. Remember that code that clearly solves just those problems is in no way a hack.

Deliver value as often as possible to get feedback

Frequently delivering value is a great way to avoid over-engineering. It basically gives you regular course correction towards appropriate-engineering, because you quickly and frequently find out if there is no value for the code you're writing.

I generally argue for at least daily delivery of value; see the "Deliver value daily" manifesto (which I co-wrote).

Write and maintain integration tests that survive refactoring

One of the worst pieces of advice that I ever received was that every function should be unit tested. This is the path to designing code that you never want to change because no matter what change you make, tests will break, and thus leads you to trying to handle every possible future up-front. Please, I beg you, no.

Instead, write higher level tests close to the client/user facing behaviour that actually give you protection against breaking things unintentionally, and so free to you make changes later once you have evidence they're needed.

Avoid Object-Oriented Programming, or at least be extra careful with it

Object-oriented code is more susceptible to over-engineering if you're not careful, because every object has its own set of responsibilities on what it looks after, often quite divorced from the actual data flows in your application. So many problems have only a handful of possible cases, and yet if you look at all the classes and member functions involved in typical object-oriented solutions, they're coded up as though there were many orders of magnitude more cases, which makes them difficult to optimise for the actual cases. Inheritance-based reuse is also notoriously hard to change, and so pushes you to design the "perfect" system up front.

Don't be scared of keeping data and code separate via free functions, especially pure functions. Basic data-in data-out patterns often result in very clear separation of responsibilities, and they are often easy to refactor.

Remember you can still add in that complication tomorrow

Train yourself to realise you can always add in that complication tomorrow if you judge that it's still needed tomorrow. Sometimes the answer will be yes, but so often the answer will be no, and that's a great thing.

This is much easier when high-level integration tests are in place.

Be concious of what makes you over-engineer

Run a search for "What causes software over-engeneering?" and see which apply to you. Self-awareness is half the battle.

Get yourself a better definition of perfection

If you think you're fighting against a drive for perfectionism, give in to the drive but instead find a better definition of perfection. This is basically mine:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
Antoine de Saint-Exupéry

When should you create stuff just in case?

This all being said, there are times when you should create something just in case, and that's when all three of the following are satisfied:

There is a reasonable chance it will be useful later
It will be difficult to add in later
It won't meanginfully slow down the meeting of more likely requirements

To determine if all those hold true, you have to have an idea of what features are likely going to be needed. This is beyond the scope of code and much more do with people—users or stakeholders—that you will have to speak to. You can always choose to assume without validating the assumption, but doing so will almost definitely be due to arrogance or fear. Do not make engineering decisions because of arrogance or fear!

Ordering work: a skill and culture gap

2025-10-03T08:50:00Z

There are virtually infinite options on how to split up all but the most trivial pieces of software engineering work, but rarely is emphasis placed on the skill of making that choice. But this choice is extremely important: the order in which work is done affects how frequently feedback is received (even from just running the code yourself) and so can seriously affect how successful projects are in the long term.

Emphasis is all too frequently put on the final properties of systems, as if components are made all at once in a single step. Even skipping over any iterative versus waterfall debate, a person can only do a single thing at once (or write one prompt at once 😜): you still have to decide to write one line of code first, another line of code second, and so on. As a parallel imagine if in other professions the order in which to do work was not a primary consideration: that would be madness.

In terms of what to do about this, my main suspicion is that a cultural shift is needed, and skills will be developed from that. Engineers must be encouraged to internalise that the order of the work they do is of primary consequence, and they must be empowered to continually set that order. I think also others in the organisation must also understand that while designs and plans are in general good, feedback from real use will so often suggest changes, and it's almost always better for such changes to happen early than late (or worse, not at all).

This is one of the several reasons I (with others) have written the "Deliver value daily" manifesto. Our claim (backed up at least from our own experience) is that a focus on small increments of at most day-sized work really helps with making decisions on what to do when.

6 rules when making data dashboards

2024-02-04T13:00:00Z

I've made a few data dashboards recently, and I've realised it can be hard to know where to begin sometimes. Hopefully these rules can help focus your thoughts.

1. Know the use cases

Just like designing anything, if you don't know how it's going to be used, it's not likely you're going to do a good job.

Often there aren't actually that many use cases. But if there are - you can construct different dashboards/sections in the dashboard optimised for them.

2. Prefer "explain" to "explore"

An "explore" chart doesn't have a clear conclusion or story associated with the data - it lets the user come up with them. But often exploring the data is not want users need to do!

Instead, users often need some very specific question(s) answering, which are what "explain" charts do.

3. Use titles that make clear how each component can be used

Titles can be useful to the users, especially new users, so they know what they can use the dashboard or each visual for.

But also, if you can't come up with good titles, then you probably don't know the use cases well enough, or you don't know what you're trying to achieve, or you're maybe trying to make each visual satisfy too many use cases.

A word of caution - visuals shouldn't be so complex that they strongly depend on the title. They should be reasonably clear at a glance.

4. Keep interactivity to a minimum

Including interactivity often feels like a safe bet, because it allows more data to be discovered "just in case". But that "just in case" probably means you don't know what's important to show.

It also leaves the uses that need the interaction second class citizens, and has the consequence of making it harder to tailor visuals to any of the uses.

Instead, find out the use cases, and then if you need to create separate visuals each tailed to one (or few) use cases.

5. Make sure the users will trust the dashboard

If the users don't trust the dashboard, then they won't take action based on it. At best it's pointless, but at worst you've wasted yours and other people's time.

How to acheive trust is tricky, and is probably specific to each situation. Some ideas include making sure it has consistent layout/spacing/colours; making sure there aren't unnecessary or broken components; making sure the data is self consistent, and consistent with external data the user has access to; making it very clear up front if there are limitations or innacuracies; making it clear when the data was last updated; or making sure that it actually answers the questions it claims to. Sometimes visuals that are technically unnecessary to answer the questions, but lend weight to the veracity of the data, help acheive trust.

6. Put in the work

Especially if you're new to making dashboards, it can take many iterations of both the UI and the data to end up with something good. If you're focusing in on use cases while you're making it, doubly so. Be prepared for this!

Questions to ask yourself when talking with users

2022-08-14T16:15:00Z

I've been fortunate to work on projects where there is often a direct (or at least short) route between myself and the users of products I've worked on. These are the sorts of questions I ask myself when communicating with them.

Am I answering their question?

I might suspect that they're not asking a question where the answer will ultimately help them (an XY problem), but I can't be sure. In most cases I do really try to answer their question.

Do I understand what they're trying to do?

Even if I answer a question, understanding the context and what they're trying to do can lead to much better help. If I don't think I understand the context enough, I ask them for more details.

Am I explaining why I need the answer to a question?

I think it can be a bit frustrating to be asked to do things without understanding why. For example, if I ask for a screenshot, it's not because I don't believe the user, it's because there can be subtle bits of information along with the error message that explains what the problem is. "Oh it's that sort of 403. Right... I think I know what's up."

Can I offer alternatives, even imperfect ones?

There are often just so many alternatives that might help a user. I often try to think of the closest thing to what the user is asking for, and suggest that. (And if it's perhaps a ridiculous suggestion, I state that I'm aware it is!)

Have I asked my collegues if there is a better way?

There are always things I don't know, even about systems I've worked on for a while. A quick message to collegues is often helpful: for the user since it might result in a solution to their immediate problem, but it's also helpful for the team, since it communicates what users are struggling with, and what parts of the system might need some attention.

Am I acknowledging pain points?

You don't need to go overboard with this, but just a tiny bit of empathy goes so far. I think it can even be efficient as well - it can serve as a very clear statement that you understand what the user is trying to do and why. Then the conversation will probably move on to working around or fixing whatever the pain point is.

Am I giving an appropriate amount of technical detail?

This is a tricky one - I don't want to bore users with too much detail, or for them to think "why are you telling me this? Just fix my problem!", but also feels a bit odd to not communicate anything at all. I think in general it's helpful to give some detail. Knowing, even roughly, what the problems/limitations are can help users come up with their own workarounds.

Am I being realistic in promises?

Saying no can be hard, but it's usually so much better to be up front about what's realistic. In so many situations people are happy to plan given realistic information.

Am I over-apologising?

Apologising is tricky one to get right, and I don't think I'm quite there yet. It's probably extremely culture and situation-dependant, but so far I think I try to keep apologising to a minimum. In so many cases it just doesn't really help. Instead, I acknowledge pain points, and try to get a solution to the user's problem.

I think I might even imagine I'm talking to Marla Singer from Fight Club. She's gives the impression that she's been apolgoised to a lot, that apologies have no value to her, and she just wants to move the conversation past them:

Yeah, you're sorry, I'm sorry, everybody's sorry
Marla Singer, Fight Club

Have I checked what I'm suggesting works?

I try to check what I'm telling users, even if I'm confident about it. Or if I can't for some reason, I say I can't. Quite a few times I have been so close to hitting that "Send" button, but I decide to paranoia check X, and I realise I was about to be wrong.

How would I feel if I read/heard this if I were them?

This is easier to to with written communication, but I think worth just putting yourself in their shoes for a moment. It might be worth tweaking the message a bit (or asking yourself more of the questions above!)

How to offer better downloads

2021-02-28T07:20:00Z

To offer HTTP file downloads via your own code [rather than redirecting elsewhere], it's often easy to rustle something up. However, the default behaviour in a lot of cases may not give users as good an experience as possible. With a bit of effort, you can polish that right up, and here are 4 sets of HTTP headers that help you do just that.

content-length

If you are able to, set the content-length header with the total size of the file.

This simple header will cause browsers to tell users [estimates for] the time-to-completion of downloads. [It's also required for browsers to attempt to do range requests, as in the following section.]

Note that if streaming a file to the client, e.g. using Django's StreamingHttpResponse, then at the point the HTTP headers are generated, the full bytes of the file are not available, so this can't happen automatically. You have to explicitly determine the length of the file and set the header. For example, if the file is stored in S3, S3 returns a content-length header with all responses, and you can take the value of this header and return it to the client.

accept-ranges / range / content-range [/ content-length]

By default, if the connection is interrupted, browsers will have to restart the download from the beginning. If you support HTTP range requests, which use the accept-ranges, range, content-range headers, [and content-length header] then browsers can resume downloads from where they left off.

See the MDN docs on HTTP range requests for more information. Note that S3 supports range requests, so if your code is essentially a proxy to S3, you can proxy the headers to and from S3 to support this fairly easily.

content-disposition

By default, browsers will guess at a suitable filename for the downloaded file, typically using the last path-component from the URL. Instead, consider what would be a more helpful filename, and set something like content-disposition: attachment; filename="very-helpful-filename.csv".

accept-encoding / content-encoding

Some files compress well, for example typical CSV files. Serving compressed versions of these would often make downloads much faster. However, if the most likely thing users will do will do is immediately and manually uncompress the file, you've just made their life a tiny bit harder.

However, the browser can do this so the user will never even notice. For example, if the browser sends an accept-encoding header specifying it accepts gzip [which most modern browsers do], and if the server returns a header of content-encoding: gzip with gzipped data, the browser will automatically decompress this data on download. The user will notice nothing other than faster downloads.

Unfortunately, S3 doesn't support this sort of content negotiation. If you're storing your data on S3, and want to support both gzipped and non-gzipped versions of an object, you'll have to store them under separate keys. To avoid this, you might be tempted to compress on-the-fly. However, you then won't be able to send a content-length, or handle range requests.

That's it!

S3 is not a filesystem

2021-02-14T13:00:00Z

Something that occasionally catches developers out is the fact that S3 is not a filesystem, but a key-value store. Specifically

the keys can be any UTF-8 encoded string, between 1 and 1024 bytes long;
the values can be any binary string, beteen 0 bytes and 5 terabytes long.

Yes, you can emulate certain features of a filesystem using slashes in the keys. In fact, the AWS console does this: it creates a folder by creating a 0-byte object with a trailing slash in its key.

But, you don't have to be limited by the key-structure that a filesystem imposes. For example, if you want to:

you can have objects, with data, at both of the keys a and a/b;
and you can have objects, with data, that have keys ending in /.

Both of these are impossible in a traditional filesystem.

An example of using this is to store an uploaded file, some-unique-name.txt say, and then store derived data inside its "folder", for example some-unique-name.txt/analysis1 and some-unique-name/analysis2.

Now a warning

If you're about to leverage the fact S3 is a key-value store by having non filesystem-compatible keys, be warned: there are consequences.

You won't be able to easily sync from your S3 bucket to a filesystem.
Everything that claims to be S3 compatible may not be, and so migrating to those systems, or using them in a test environment, may be trickier. Specifically, minio [that I have used many times and generally really like] is not S3-compatible in this way.

So, there are some valid pragmatic reasons to make sure your keys are filesystem-compatible. But, I can't help the niggling feeling that it's a shame to be limited by systems you're not using.

Django's transaction.atomic()

2020-10-10T11:00:00Z

I have a confession: I assumed things about Django's transaction.atomic() that are not true, at least not true by default in PostgreSQL. I assumed that in a transaction.atomic() context as below, database statements are protected from any race conditions, and everything will Just Work™.

with transaction.atomic():
    # Database statements

But that's really not true.

Enter the world of transaction isolation levels: None with autocommit, Read committed, Repeatable read, Serializable, and "do it yourself".

Django ⬄ SQL mapping

Firstly, this post discusses transactions mostly in terms of SQL. As such, it's good to have a reasonable understanding of what Django does in terms of SQL queries.

Django	SQL
Fetch from a QuerySet	SELECT
Existing model `save`, or QuerySet `update`	UPDATE
Enter an `atomic` context	START TRANSACTION
Exit an `atomic` context without exception	COMMIT
Exit an `atomic` context with an exception	ROLLBACK

No explicit transaction, with autocommit

For completeness, without transaction.atomic(), by default Django runs in autocommit mode, which means every statement runs in its own Read committed transaction and committed immediately. While tempting to translate "auto" as "don't worry", even in autocommit mode UPDATEs can block , deadlock , or miss rows .

Read committed

The default isolation level of a PostgreSQL transaction, and so also of transaction.atomic(), is Read committed. Such a transaction has 4 important properties.

Each of its SELECTs behaves as though a snapshot of the database is taken at the start of the SELECT, containing only committed data from other transactions .
Each of its UPDATEs can block if concurrent transactions have UPDATEd the same rows, and block until those transactions COMMIT or ROLLBACK .
This blocking can lead to deadlock if concurrent UPDATEs on the same rows are done in different orders .
Its COMMIT won't fail due to concurrent transactions COMMITting the same rows .

This leaves a lot of room for race conditions, which can be roughly split into two classes.

Between multiple SELECTs in a Read committed transaction, other concurrent transactions can COMMIT, and so their changes become visible . Consider a Sock model, where each sock is either white or black.
```
with transaction.atomic():
    black_socks = list(Sock.objects.filter(colour='black'))  # A SELECT query
    white_socks = list(Sock.objects.filter(colour='white'))  # A SELECT query
```
You might be tempted to assume that due to the transaction.atomic(), each sock will be in exactly one of black_socks or white_socks. However, since each SELECT uses a different snapshot of database state, a given sock can be in both, or neither, of these lists.
UPDATEs with WHERE clauses, have more complex semantics, effectively using mutiple snapshots over their lifetimes. Similar to SELECT, they effectively take a snapshot at start to initially choose the rows. However, they then block until the COMMIT of concurrent transactions that are updating those rows; and once unblocked, take snapshots again to re-evaluate the WHERE clause; and, if it passes for any given row, performs the update on it. This re-evaluation on new snapshots means that such statements may "miss" rows .
What are typically safe from this class of race-conditions are UPDATEs where the WHERE clause is based on columns that do not change, such as primary keys. From experience, this is the vast majority of UPDATEs.

Repeatable read

A Repeatable read transaction

with transaction.atomic():
    cursor = connection.cursor()
    cursor.execute('SET TRANSACTION ISOLATION LEVEL REPEATABLE READ')
    # ...

has 4 important properties.

It behaves as though a snapshot of the database is taken at the first non transaction-control statement [e.g. a SELECT, UPDATE], which is used throughout the entire transaction, and data it sees is not affected by other transactions COMMITting modifications .
Each of its UPDATEs can block if concurrent transactions UPDATE the same rows, until those transactions COMMIT or ROLLBACK . This is the same as Read committed.
This blocking can lead to deadlock if UPDATEs are not ordered . This is the same as Read committed.
It will fail on COMMIT if a concurrent transaction committed updates to the same rows .

While attractive, this isn't enough to avoid race conditions. Multiple concurrent transactions can use the same snapshot , violate assumptions of your code, and cause bugs.

Serializable

Serializable is the strongest isolation level. It's similar to Repeatable read in that a snapshot is taken at the beginning of the first non transaction-control statement , and it will fail if concurrent transactions attempt to commit updates to the same rows .

However, it will fail in more situations than Repeatable read. Specifically, if at COMMIT the database cannot determine that the transaction could have been performed serially with respect to the read/writes of other transactions, then it will fail with a django.db.DatabaseError. This can happen even if they updated different rows .

PostgreSQL is somewhat overcautious doing this. For example, the following faily simple example can fail if performed concurrently .

with transaction.atomic():
    cursor = connection.cursor()
    cursor.execute('SET TRANSACTION ISOLATION LEVEL SERIALIZABLE')
    Sock.objects.all().count()
    Sock.objects.get(id=request.POST['sock__id']).save()

Do it yourself: select_for_update

You can "add to" isolation inside transactions using select_for_update. This blocks until the current transaction can obtain the same locks as an UPDATE would on the matching rows, which are then kept until the end of the transaction .

A use for this is in a Read committed transaction to enforce serializability, without the risk of commits failing as they could in a true Serializable transaction, but at the cost of the time of blocking, and the risk of deadlock .

At the time of writing this appears to be the only technique that is a first-class Django citizen: no need to write explit SQL.

Summary

There is no magic or one-size-fits-all approach to database transactions, and you can't trust Django to always do the right thing, even with a transaction.atomic() or select_for_update.

Thank you to a colleague of mine who basically told me what I thought I knew was wrong. Thanks also to the PostgreSQL isolation levels documentation and to the PostgreSQL Serializable Snapshot Isolation (SSI) documentation on which some of the above examples are based. These pages are good for further reading for more details on INSERT, DELETE, and other forms of SELECT which are ommitted in the above for brevity.

JavaScript task queue

2020-05-17T11:00:00Z

Here's Queue, a small utility function to queue and run tasks with limited concurrency.

const Queue = (concurrency) => {
    var running = 0;
    const tasks = [];

    return async (task) => {
        tasks.push(task);
        if (running >= concurrency) return;

        ++running;
        while (tasks.length) {
            try {
                await tasks.shift()();
            } catch(err) {
                console.error(err);
            }
        }
        --running;
    }
}

You enqueue a task by passing it to the function returned by Queue.

// Create the queue with a concurrency limit
const concurrency = 4
const enqueue = Queue(concurrency);

// In this case, each task is an async upload function...
const upload = async (file) => {...};

// ... and each task uploads a file
const files = [...];
files.forEach((file) => {
    enqueue(async () => upload(file));
});

If there is available concurrency, the task will run [almost] immediately. Otherwise, it will wait until enough tasks ahead of it have completed for it to run within the concurrency limit.

Use case

A use for this is part of a file uploader that allows arbitrary numbers of uploads. Although browsers limit the number of connections, they also limit how many requests are queued: Chrome can throw net::ERR_INSUFFICIENT_RESOURCES if you attempt to queue too many. To avoid this, you can queue the uploads up front.

Features

Can add tasks at any point after queue creation
No recursion
No polling
Small API
Small amount of code
Leverages async/await syntax: no handling of promises or callbacks, either internally or in client code

That's it!

I admit it: I like Python's asyncio

2020-05-08T18:50:00Z

Python's asyncio gets a fair bit of bad press. Some of it I agree with, but there is one aspect of asyncio I like: the API needed for a lot of common tasks is actually fairly small and clear.

Here's a small but fairly realistic program. It creates a pool of HTTP connections, and uses this to make two concurrent chains of requests.

import asyncio
import httpx

async def async_main():
    async with httpx.AsyncClient() as client:
        task_1 = asyncio.create_task(make_http_requests(client, {'some': 'data'}))
        task_2 = asyncio.create_task(make_http_requests(client, {'more': 'data'}))
        await asyncio.gather(task_1, task_2)

async def make_http_requests(client, data):
    resp_1 = await client.post('https://postman-echo.com/post', data=data)
    resp_2 = await client.post('https://postman-echo.com/post', data=resp_1.json())

asyncio.run(async_main())

Tasks

A task is an asyncio thread of execution. Multiple tasks can run concurrently: typically when a task is waiting for some data to be sent/received on the network, other tasks can progress. This is in fact the whole point of asyncio.

In the above example, asyncio.run creates the first task, and during that task, each call to asyncio.create_task creates another task that runs concurrently.

await my_function()

If you see await my_function() it means two things:

the current task will schedule my_function to run, and wait for it to finish;
in the meantime other tasks can progress, typically during network I/O.

asyncio.gather

To wait for multiple tasks to finish, pass them to asyncio.gather, and await the result.

async def my_function():

A function declared as async is very similar to a regular function, but just calling it won't do anything: it then needs to be scheduled. This is done by either awaiting its result, or passing it to another function such as create_task that will schedule it for you.

Some don't like having to write async and await, but in complex multi-task programs, I find it useful to be clear which functions allow other tasks to progress, and which don't.

async with

async with is like a regular context manager's with, except that during entry and exit, other tasks may progress. It's commonly used to create and destroy pools of network connections, often a part of HTTP or database clients.

That's it! There are more parts to the API, but for a lot of common use-cases, you don't need them.

Streaming: a skill gap?

2020-02-02T10:40:00Z

I've noticed a bit of a skill gap: I think a lot of developers are not able to code up "streaming" solutions to problems.

However, streaming can often be useful, even needed, in what are now run-of-the-mill web applications; and wonderfully, we often don't need anything fancier than the tools already being used: we just need to know how to use them.

What is streaming?

Any situation when you process data concurrently with receiving it. This process can be analyzing the data, or just forwarding it onwards.

What are the benefits of streaming?

There are two main [potential] benefits.

Speed

If you start processing the data sooner, before its all received, then you [might] finish sooner.

Support higher concurrency / size limits

Say you would like users to be able to upload 500mb files: in these days of video and hi-res images, this isn't a far-fetched requirement, even for a standard web application. If you don't forward the uploaded data onwards while it's still being uploaded, just a few users uploading concurrently could use all the memory on a server.

[You can upload directly from a browser to the underlying data store. For example, to S3 using presigned URLs. However, this has its own set of drawbacks, ommitted here for brevity.]

What are typical problems with streaming?

Streaming is not a perfect/one-size-fits-all solution: it does have its downsides.

Testing

You're testing an upload with a 5KB file, and it works. Are you sure it's streaming and will work with a 5GB file? There are two options that I'm aware of.

Actually test a 5GB file [making you have less than 5GB of memory available]. While this is quite a good "real" test, it can be slow.
Hook into both sides of the streaming process, and ensure that the target receives data before the source has sent all of its. You can do this with smaller data, and so such a test can be quick. However, this can be more brittle with respect to refactorings, i.e. the test can fail while the production behaviour continues to work.

Errors

Handling errors, i.e. communicating and responding to them, can be more difficult.

Conveniently, HTTP has some of this built-in. If streaming an HTTP body with a content-length header specifying the number of bytes, if the receiver doesn't receive that amount by the time the connection has closed, they know an error has occurred. If transfer-encoding: chunked is used, if the receiver doesn't receive a 0-length chunk at the end, they know there has been an error.

It's not perfect though: there is no way to send an HTTP status code once the body has begun to stream. But for many situations, this is enough.

What to do when an error has occured may be more tricky. With a non-streaming multi-stage pipeline, if one part fails, you can usually retry because you have the source bytes to retry with. However if streaming, the bytes have gone. To retry, have to build in a mechanism to re-retrieve them from the source.

Complexity

Especially when considering error handling, retrying, or say, efficiently dealing with bandwidth differences/variation in different parts of the stream, there could be more complexity compared to a non-streaming solution.

This being said, a) you may not need to implement such things [e.g. OS-provided TCP buffers may adequately compensate for bandwidth variation], and b) I suspect the complexity is sometimes overstated, and conflated with unfamiliarity [although it would be naive to think this is isn't a problem, as mentioned below].

Performance

Ironically, there might be a performance penalty compared to non-streaming solutions due to what could be radically different operations / orders of operations. This could be especially true if using streaming for smaller amounts of data.

Homogeneity

Each part of the pipeline needs to support streaming. It's not the default in a lot of cases: which is unfortunate since you can use code that supports streaming to process data in a non-streaming way [by just using a single "chunk"], but it's impossible to do the opposite.

Unfamiliarity

Streaming has an unfortunate problem: it's the skill gap itself.

Since fewer developers are familiar with it, issues are less likely to be spotted in code reviews, streaming behaviour may be accidentially broken [if there aren't appropriate tests on it], there are fewer people to ask for help, and unfortunately any help that is given has a higher chance of being misleading.

This is admittedly a bit of a chicken/egg situation!

What can I do?

I keep hearing my mother say, practice, Harry, practice!
Harry Kim, Star Trek Voyager

Wonderfully, I think you can get a lot of valuable experience from just a few small practice web-based projects.

A GET endpoint that responds with a generated HTTP response of several GBs, just of some fake data.
A GET endpoint that responds with a file from the filesystem of several GBs. Try with with both transfer-encoding: chunked and with a specified content-length.
Proxying a file to or from S3 through a server. Try with a plain HTTP client, not just one that is AWS-aware such as Boto3.
Downloading a Postgres table of several GBs. Try with just a single query. Try responding with CSV or JSON.
Accept a large CSV upload and calculate some basic stats on the columns while its being uploaded, e.g. min, max, mean, standard deviation.

Once you have done these, you would be in a much better place to weigh up the trade-offs to know if a streaming solution is right for any given real-world project. At the very least, you'll be in a better place to review collegues' streaming-based code.

Python generators

2020-01-12T11:00:00Z

In these days of medium data [data that is too big to fit in memory on a single machine, but could otherwise be processed by one], it's important to know what features your programming language offers to help you process data using streaming. Generator functions in Python are one such feature. [For brevity, this post will refer to generator functions as generators].

Generators vs functions

Generators differ from functions in that while a function can only return a value once, generators allow you to "return" multiple times. The keyword return is not used for this however; instead yield is.

Basic examples

def generate_words():
    yield 'first'
    yield 'second'
    yield 'third'

# Will print out: "first, second, third, "
words = generate_words()
for word in words:
    print(word, end=', ')

Often the keyword yield is itself in a loop. For example, the above is equivalent to the below.

def generate_words():
    for word in ['first', 'second', 'third']:
        yield word

# Will print out: "first, second, third, "
words = generate_words()
for word in words:
    print(word, end=', ')

Generators can then be chained together to make a pipeline.

def generate_words():
    for word in ['first', 'second', 'third']:
        yield word

def process_words(words):
    for word in words:
        yield f'{word} apple'

# Will print out: "first apple, second apple, third apple, "
words = generate_words()
processed_words = process_words(words)
for word in processed_words:
    print(word, end=', ')

Declaring a generator

A generator is declared as a function, i.e. with def function_name(, but somewhere in its body it contains at least one instance of the keyword yield. [See criticism]

Using a generator

As in the above example, you call the generator and iterate over its return value: each yield encountered results in an iteration.

Purpose of generators

They allow you to

process data in a streaming way [i.e. without loading it all into memory at once];
while separating code with different responsibilities [which is often seen as a good thing], into a pipeline of chained generators;
without having to write custom classes that implement __iter__ and __next__ [which is quite boilerplate heavy, and so can be hard to see the data processing that's going on].

The earlier basic examples show some of this: the data generation code is separate from the processing code, which is separate from the printing code.

Real-world example

A more real-world, and complex, example is constructing a CSV dump of table from a Postgres database using psycopg2, using a single database query [not using an ORM], and streaming it from a Django view. You could do something like the following, which uses a chain of generators.

import csv
import os
import psycopg2
from django.http import StreamingHttpResponse

def fetch_db_rows(dsn, query):
    # Named cursor => server side cursor. Without this, _all_ data from the
    # query will be fetched into memory, which defeats the purpose of streaming
    with \
            psycopg2.connect(dsn) as conn, \
            conn.cursor(name='my_csv_dump') as cur:
        cur.itersize = 5000
        cur.arraysize = 5000
        cur.execute(query)

        while True:
            rows = cur.fetchmany()
            for row in rows:
                yield row
            if not rows:
                break

def to_csv_bytes(rows):
    # Python's csv.writer expects a file-like object with a `write` method
    # We give it one, but the `write` method just returns the bytes passed
    class DummyFile:
        def write(self, value):
            return value

    csv_writer = csv.writer(DummyFile(), quoting=csv.QUOTE_NONNUMERIC)

    for row in rows:
        yield csv_writer.writerow(row).encode('utf-8')

def my_streaming_csv_view(_):
    dsn = os.environ['DSN']
    query = 'SELECT * FROM my_table ORDER BY id'

    # Define a pipeline of two steps
    db_rows = fetch_db_rows(dsn, query)
    csv_bytes = to_csv_bytes(db_rows)

    # Django/the web-server adds to this pipeline: adding HTTP chunked
    # headers, and sending data to the underlying socket
    return StreamingHttpResponse(csv_bytes, content_type='text/csv')

This is ok, especially in terms of separation of components in the pipeline: fetching from the database is separate from conversion to CSV, which in turn is separate from any HTTP concerns.

However, StreamingHttpResponse has some hidden behaviour [depending on configuration/web-server]. Every time it iterates over the value passed, it results in an HTTP chunk [which includes a short header] being sent through various layers of code, to the socket, and maybe even over the wire. Especially if each CSV line is small, this has quite a high overhead per byte of actual data.

We can do better.

We can send HTTP chunks of [for example] 8KB, even if each CSV row is smaller or bigger than 8KB. To do this we add a buffering step to the pipeline: we don't need to touch the database code, the CSV-writing code, or the code of StreamingHttpResponse.

def buffer(chunk_size, bytes_items):
    queue = []
    queue_length = 0

    for bytes_item in bytes_items:
        queue.append(bytes_item)
        queue_length += len(bytes_item)

        while queue_length >= chunk_size:
            to_send_bytes = b''.join(queue)
            chunk, to_send_bytes = \
                to_send_bytes[:chunk_size], to_send_bytes[chunk_size:]

            queue = \
                [to_send_bytes] if to_send_bytes else \
                []
            queue_length = len(to_send_bytes)

            yield chunk

    if queue_length:
        yield b''.join(queue)

def my_streaming_csv_view(_):
    dsn = os.environ['DSN']
    query = 'SELECT * FROM my_table ORDER BY id'

    # Our pipeline has an added step to buffer the data into 8KB chunks
    db_rows = fetch_db_rows(dsn, query)
    csv_bytes = to_csv_bytes(db_rows)
    chunked_csv_bytes = buffer(8192, csv_bytes)

    return StreamingHttpResponse(chunked_csv_bytes, content_type='text/csv')

This mechanism of being able to define and maintain a pipeline, in code, with the minimum of boilerplate, and without needing any extra libraries, in my opinion is wonderful.

Possible limitations

Each component in the above pipeline is on the same thread: none of them can actually do anything while another is processing. Specifically, between each of the fetchmany calls to the database, the database connection is sitting idle, when it could have been fetching the next batch of data. You may be tempted to do something about this by using threads[/similar] with another intermediate buffer.

My suspicion is that in a lot of situations this wouldn't increase performance. The downstream connection to the client would already have buffers, and its bandwidth is likely to be much lower than upstream to the database, so the buffers would never get empty: the downstream connection may already be sending as many bytes per second as it can. Threads[/similar] may do nothing but use more resources on the server, hurting performance and increasing costs. My recommendation is to KISS and only complicate the pipeline if you have good evidence it will help in your production cases.

Criticism

I find it odd/confusing that the only way to differentiate a function from a generator is to look inside its body and look for the keyword yield. Thinking how asyncio functions are declared with async def my_func():, I think generator def my_func(): would have been clearer and more consistent.

Mini HTTP guide for developers

2020-01-04T16:00:00Z

Frameworks often hide/abstract parts of HTTP away. I think this is often a bit of a shame: it hides what's possible with HTTP, and so can lead to effects on engineering decisions.

This short guide aims to rectify that. It details a few of the most common and useful parts of HTTP, and is aimed for developers with some experience making or receiving HTTP requests. [In this post, the term HTTP is used to refer to the bytes of the HTTP protocol, which is the same if those bytes are sent over plain TCP, or through a TLS tunnel. The term HTTPS is used only when necessary to distinguish HTTP over TLS.]

Initialising the connection

Say we ask our HTTP client to make a GET request to the URL https://example.com/the/path. Firstly, there is no such a thing as a URL in HTTP: it's just a shorthand that the client parses and uses the different components at various points in the process.

The client resolves example.com to an IP address, say to 1.2.3.4. Note that typically in many cases this would involve sending the string example.com unencrypted across the network.
Initiate a TCP connection to the IP address 1.2.3.4 on port 443: if no port is specified in an HTTPS URL, port 443 is assumed. For HTTP URLs, port 80 is assumed.
Initiate a TLS connection over the top of this TCP connection. This again would uses the domain example.com, in both SNI, and verification of the subsequently supplied certificate. In many cases, the domain name example.com would be transmitted unencrypted across the internet.
Then start the HTTP request/response process. This would use both the domain example.com, as well as /the/path. This process is detailed below.

The HTTP request

The client sends the bytes of the HTTP request message over the TLS connection. This is made of a request-line containing the method and the path, followed by a number of header key:value pair lines, a blank line, and then the body. In this case, the body is 0-bytes long, which is typical for GET requests.

GET /the/path HTTP/1.1\r\n
host: example.com\r\n
\r\n

A "line" ends with the two characters \r\n. The visual line breaks in the examples shown here are for ease of comprehension, and are not characters that are transmitted.

The HTTP response

The server would then respond with a status-line, some headers, a blank line, and the body of the response.

HTTP/1.1 200 OK\r\n
content-length: 21\r\n
\r\n
The bytes of the body

Notable headers

The important parts of HTTP are the headers: bits of metadata sent before the body of the message [usually].

Host header

Usually HTTP clients add a host header automatically from the supplied URL. It has two common uses.

CDNs or reverse proxies use the host header to determine how to route requests onwards.
Application server code uses the host header in a best-effort attempt to determine the domain the HTTP client used to make the request. Depending on the configuration of intermediate proxies, this can mean the application server may not be able to correctly determine what the original domain was.

Content-length header

HTTP is sent over TCP [or TCP+TLS], which surfaces as a stream of bytes in client code. A "stream of bytes" means that the receiver could just receive a single byte at a time, perhaps even with seconds of delay between each. The receiver has no way to know if it has received all of the bytes, or the connection is just a bit slow. For this reason, requests and responses [that can have a body], can supply a header that tells the other end how many bytes are part of the body of the message. This is the content-length header.

Often HTTP clients add this automatically if they know at the time of starting sending the HTTP message how many bytes will be sent in the body.

Transfer-encoding header

A HTTP message sender may want to send a body, but it does not know at the start how many bytes make up that body. One option is that it can wait until it knows how many bytes, and set the content-length header appropriately. However, this may involve having to buffer all the bytes in memory, which may not be possible or desirable.

An alternative is to use transfer-encoding: chunked. With this header, the body of the message is sent in chunks, each prefixed by the number of bytes in that chunk [as it happens, in hexadecimal]. This means the body transfer can be started without knowing how many bytes in total will be sent. Common chunk sizes are between 8kb and 64kb. Often HTTP clients "do the chunking" themselves, adding the chunk header before each chunk as needed.

However, it is usually better to avoid transfer-encoding: chunked and instead set a content-length header. The receiver can use this in various ways, such as to or be able to allocate resources needed at the start of downloading the body, or estimate time remaining. If the receiver needs to know how many bytes are in the body, using transfer-encoding: chunked may be forcing it to buffer the entire body in memory before it can process it further.

Wonderfully, you can often still stream bodies with a correctly set content-length, but you may need to go to a bit of effort to find the right value. For example, to stream a file you may need to query the file system explicity to find the length of the file before starting to fetch its bytes.

Connection header

HTTP/1.1 by default keeps connections open after a HTTP request/response, so they can be used for subsequent request/response, and avoid the overhead of new TCP [or TCP+TLS] connections. This referred to as persistant connections, and is often a good thing, but has downsides.

Usually servers would only keep the connections alive for a certain period of time, and then close them. This means there is a race condition: a server could have closed the connection from their point of view, but the client not be aware of this and attempt to re-use the connection, send its bytes [but the server wouldn't process them], and only later some time would the client would be aware of an error condition. The client may not know if it's safe to retry the request or not. For example, a client may have no way of determining if a POST errored before or after it was processed by the server. If designing an API, you may wish to implement some sort of unique idempotency-key for such requests. With this, the client can safely retry requests that have failed from its point of view, while the server knows not to reprocess any duplicates, and can still return the response corresponding to the original request.

Another downside is that if you don't end up re-using the connection, resources would continue to be used needlessly on both the client and the server.

If you want a smaller chance of issues like this, you may explicitly set a connection: close header. If you can deal with such issues, you may wish to design the system to take better advantage of persistant connections. For example, instead of choosing to have multiple S3 buckets each on a different domain, you choose to have one, to take better advantage of per-domain HTTP persistent connections and speed up S3 requests/responses.

X-forwarded-proto

This is a modern header: it is often added to requests by HTTP-aware intermediate CDNs or reverse proxies. If the proxy has received an HTTPS connection, it can add x-forwarded-proto: https, and otherwise adds x-forwarded-proto: http.

Without this header, the application server behind a reverse proxy would have no mechanism to know if the client made its request via HTTP or HTTPS. This may be important if you would like to respond to HTTP requests with redirects to HTTPS URLs.

X-forwarded-for

Often an application server would like to know the IP address of the client. However, if the client connects to a reverse proxy, and then the reverse proxy connects to the application server, the application server only has details of that final TCP connection. From its point of view, its TCP client is the reverse proxy. This is often not helpful.

The solution to this is that each intermediate proxy adds (to) the x-forwarded-for header in the request, setting the IP address that its incoming TCP connection is from. If there is already an x-forwarded-for header on its incoming HTTP request, it appends the IP address to this in a comma separated list before forwarding the HTTP request onwards.

This means that the application server can receive an x-forwarded-for with a long list of IP addresses in it, for example x-forwarded-for: 1.2.3.4, 5.6.7.8, 9.10.11.12. Because each server adds to the value of the existing x-forwarded-for header supplied by a potentially untrustworthy client, care must be taken before trusting any particular value in this list.

For example, you may have an application accessible behind a CDN, which adds an x-forwarded-for, so in the application server you may be tempted to trust the first IP in x-forwarded-for. However, the CDN would append to any existing values in x-forwarded-for. This means that an evil client can send a request with an existing x-forwarded-for header, set with some IP, and trick the application into thinking the client is at that IP. Knowing this, you may choose to use the last IP address in the list, thinking that this can be trusted. However, this may also not be a good choice: often applications are accessible both from the CDN, but also directly, even if just via an IP address. An evil client could connect to this with a spoofed x-forwarded-for header, and again trick the application.

Solutions to this trust issue involve only using the last N values of x-forwarded-for, where you have a mechanism to ensure that those N hops a) definitely involved certain infrastructure and b) you trust that infrastruture to manipulate any existing x-forwarded-for in a certain way.

Summary: Reconstructing URLs

Reconstructing the URL that a client used involves multiple parts of the HTTP request: the path of the start-line, the host header, as well as the x-forwarded-proto header. For all this to work, intermediate proxies must be appropriately configured.

Summary: Streaming

HTTP is often enough for streaming: you may not need anything fancier. If you can determine the full length of the body, set the content-length header; otherwise, use transfer-encoding: chunked.

Summary: HTTP is leaky

All non-trivial abstractions, to some degree, are leaky.
Joel Spolsky

HTTP is a leaky abstraction, exposing information on the lower-level TCP [or TCP+TLS] connection via the x-forwarded-* headers; giving the ability to control that connection via the connection header; and requiring one of content-length or transfer-encoding headers to make up for the fact that TCP doesn't have any concept of message length.

If you want to take full advantage of HTTP, you should be aware of these; compensate for them; and even be able to leverage them when needed to avoid unnecessary time, memory, code, or infrastructure use.

The core S3 API is beautiful

2019-12-31T08:10:00Z

One of the beautiful, and maybe even genius, things about the core S3 API: it's just an HTTP PUT to store an object, and an HTTP GET to fetch it. You need a few headers, but that's it.

Structure of an S3 PUT request

PUT /key/of/the/object HTTP/1.1\r\n
host: my-example-bucket.s3-eu-west-1.amazonaws.com\r\n
authorization: ...\r\n
content-length: ...\r\n
x-amz-content-sha256: ...\r\n
x-amz-date: ...\r\n
\r\n
The bytes of the object

Structure of an S3 GET request

GET /key/of/the/object HTTP/1.1\r\n
host: my-example-bucket.s3-eu-west-1.amazonaws.com\r\n
authorization: ...\r\n
x-amz-content-sha256: ...\r\n
x-amz-date: ...\r\n
\r\n

This means that as long as you can inject the right headers, you can use any HTTP client to make the requests. Here's an example using Python requests.

import os
import requests

# from https://gist.github.com/michalc/ccb87856363a895fd1fadf52ab4cdcec
from aws_sig_v4_headers import aws_sig_v4_headers

host = 'my-example-bucket.s3-eu-west-1.amazonaws.com'
service = 's3'
region = 'eu-west-1'
path = '/key/of/the/object'
pre_auth_headers = {}
query = {}
data = b''
headers = aws_sig_v4_headers(
	os.environ['AWS_ACCESS_KEY_ID'], os.environ['AWS_SECRET_ACCESS_KEY'], pre_auth_headers,
	service, region, host, 'GET', path, query, data,
)
response = requests.get(f'https://{host}{path}', headers=headers)

If the core wasn't so straightforward, I suspect S3 would not be anywhere near as popular, and S3-compatible storage providers may not even exist.

I don't often say this: but thank you AWS.

Questions to ask yourself when considering a dependency

2019-12-29T12:40:00Z

What is the underlying data transformation I need?

If it's trivial, or almost trivial, consider writing it yourself.

What does it do in my specific case?

Consider the actual data transformation you need done, and go through it in the dependency's codebase. Often libraries have extremely generic components to support lots of use cases. You however, may just have the one, and so this genericness is unnecessary. Generic components are often difficult to debug and difficult to optimise: make sure it's worth having them.

What would I do if I do need to make changes to it?

Ensure you look through its code to see how much of a project it would be to make changes to it, ideally with some specific changes in mind. Keep in mind that in many cases, the dependency authors have no obligation to address your issues, so being aware of what it would take to make certain changes yourself is important. If it would be extremely time-consuming, consider not using the dependency.

Am I treating code in the dependency as not part of my software?

It's an easy bias to fall into: out of sight, out of mind. But dependencies are part of your software: they take your CPU cycles, your memory, their bugs are your bugs, their security issues are your security issues, their updates are your updates. Consider reviewing its code as you would any other PR into your codebase.

How much of its API am I actually using?

If you've using 1 function of a 30 function API, you may be introducing a lot of dead/unreachable code into your software, as well as something that may not be as tailored to your use case as it could be. This means that if changes need to be made to tailor to your case, they would likely be harder than they would otherwise be. Consider just writing it yourself, or if licensing allows, just including that function in your codebase.

What's the coverage like when I run my tests on it?

Consider introducing the dependency, but then running your test suite with code coverage including the dependency's code. If it's low, you would be introducing a lot of code into your project that is not covered by your own test suite.

Can I write a simpler version myself that's enough for now?

What has been specified as "needed" often isn't really needed quite yet: often it's acceptable to release something incrementally. Consider writing a simpler version, geting feedback, and then moving to the fuller/more complex versions armed with confirmation on exactly what complexity is needed. You can iterate on your own-rolled solution, or decide that introducing the dependency really is the best move.

How much does it offer over its dependencies?

Sometimes dependencies just offer a thin layer over their own dependencies. Consider depending on those directly [and then asking yourself these questions for all of them].

Am I adding it because I'm scared of implementing it myself?

Consider spending some time trying to do it to overcome the fear. You might realise it's not such a big and scary thing. Keep the scope focused: your aims are often not the aims of the dependency's authors, such as creating a generic library to be used in a lot of cases. Even if you do think you will need generic components, this can often be done later when you have confirmation of your requirements.

How long would it take for me write the functionality I need?

If your estimate is a few hours or days, consider just doing it. Even if you abandon this and decide to use the dependency in the end, this is not necessarily time wasted: you would have learnt more about what the dependency does, and so what your own software does, and puts you in a better position for supporting a wider range of future requirements.

Am I adding it because its marketing suggests it does a lot?

Often dependencies market themselves as doing/having done a lot [even just via a README], which indirectly suggests that attempting the same thing yourself would be foolish. This may be misleading in two ways: they may have not done a lot; or they may well have done a lot, but you might be depending on just a tiny sliver of that work. Consider keeping in mind your specific use case, the data transformation you need it to perform, and looking through its code.

Can I still add it later easily?

If yes, consider implementating part of it yourself, and adding it later when you have more evidence of its usefulness.

Am I adding it "just in case" I need some of the functionality?

Be very careful of this. It's often easier to add a dependency later than to remove it, so consider deferring the addition until you have good evidence that it's a benefit.

Am I treating recent commits as evidence of free work?

It's easy to look at recent commits and see the author as another member of your team. In most cases, they are under no obligation to do anything about your issues. It's a risk to depend on this. This risk may be required or acceptable, but it may be unnecessary or too high.

Am I treating a lack of recent commits as a dead project and so something to avoid?

There may be nothing wrong with a project that has had no recent activity. Say it has a small API, few or no dependencies, you look through the code and its fine: then there may be little to do on it. It may be just be finished! Or at very least, finished enough for you to use.

You might not need a class

2019-05-09T20:35:00Z

When writing Python, sometimes you need to store/manipulate state. There are two typical options:

Plain data structures [dictionaries, lists, sets etc.], passed to functions which perform mutation.
Instances of classes, i.e. objects, with the state as members, mutated using instance methods.

However, there is a third way:

A class-like function [I didn't come up with this term], with the state stored in a closure.

A class-like function is a "constructor" that returns "public" functions that can manipulate its state.

def MyClass(...)  # Capitalised to make it clear it's a "constructor"

    state_1 = ...
    state_2 = ...

    def func_1():
        nonlocal state_1
        ...

    def func_2():
        nonlocal state_2
        ...

    def func_3():
        ...

   return func_1, func_2  # Typically, 1 to 2 funcs

Used as, for example

func_1, func_2 = MyClass(...)

Such class-like functions have a few nice properties.

They are less flexible than classes. Yes: this is a nice property when dealing with state. You can't just define a few more methods, add @properties, override methods in a subclass etc. State is often seen a source of bugs, and sometimes it's better to really limit how it's mutated.
To test it, you are strongly pushed to only test the public interface, i.e. the functions returned from the function, rather than private implemenation details.
They encourage stronger separation of responsibilities between code that mutates state, and code that doesn't. I would be wary of too strongly touting benefits of "separation of responsibilities"; but separating stateful and non-stateful code, all-things-being-equal, is good.
You can name the returned functions whatever you like, even _ if not needed.
There are cases when it's faster.
They allow you to escape self-hell [although you may need more nonlocal].

As examples, lowhaio and aiodnsresolver both use class-like functions.

I admit it: I like Python's ternary operator

2019-03-09T10:00:00Z

Lots of people hate Python's ternary operator. If laid out as:

selected_value = value_1 if condition_1 else value_2 if condition_2 else value_3

I can see why: it's hard to see what's going on. However, laid out differently, it's a different story:

selected_value = \
  value_1 if condition_1 else \
  value_2 if condition_2 else \
  value_3

You can much more clearly see what the possible values are, and under what conditions they're used.

And I claim it's safer. The clarity itself is a contributing factor to this. But also, compared to an if-elif-else statement, it's harder to get a NameError by accidentally not assigning a value to a variable. This is especially true with complex cases, when you might otherwise have multiple nested if-elif-else statements.

Reducing risk of release day

2019-03-02T22:30:00Z

Ideally, changes of behaviour are released incrementally to base your next steps on feedback. But sometimes you think this just isn't possible. Maybe it had been decided that large public behaviour changes must all go live at once; or sometimes planned changes are so large, you might initially think that there is no alternative other than for it all to go live at once, a few days, weeks, months [or even longer!] down the road. Ironically, it is exactly for the sorts of changes where you might have difficulty seeing how it could be split up that are the most important to split up: they are complex and so high risk.

Three common classes of such changes are:

a user-facing feature that is seemingly small, but actually needs large scale internal code changes;
a larger user-facing feature-set that needs lots of UI and backend changes [often released at once for marketing reasons];
an internal property of an application that is seemingly all-or-nothing, such as bumping the version of a framework through a breaking change that requires large scale changes in the codebase.

You can automated test, QA, or load test these large changes all you like, but these checks are rarely a replacement for feedback from production: in terms of bugs but especially for performance. The performance characteristics of large swathes of code that has not run in production by large numbers of users can rarely be predicted.

However, it is often still possible to split up the changes to get production feedback on a lot of code before release day. As is typical for many [all?] changes, each change should be:

small: ~1 day's work;
safe: reviewed/tested/QA-ed as you deem appropriate;
released: so the code actually runs in production, and its result used.

To split the work, on what seem to be "all at once" problems, into such pieces, there are two classes of techniques, detailed below.

Long-running feature branch [but keeping it as small as possible]

A long-running feature branch is often the traditional way of doing large changes, periodically merging in any changes from the master branch, and releasing by merging to master at the end. This is by far the worst technique. Not only are you essentially maintaining two versions of the application [similar to The Big Rewrite], but come release day, it will all hit production at once!

However, a variation of this can often have great properties that reduce risk on release day. Instead of just periodically taking changes from master into the feature branch, you also do it other way around. In more detail...

As you're working on a feature branch, always think about what could be released [and used in production] first, without changing the public behaviour. Put another way, try to refactor* the production code so it can run both the original behaviours, and the new ones, and release the refactoring before the actual feature is released.

Ideally, you can plan this: do the refactoring up-front, and then move onto the new behaviour that uses the refactored code. However this is often not possible: not until having written some of the feature do you then realise some parts of the code need to be refactored, and this should/could be released first. It can be tricky to reorder this: the refactoring done second is to be released first.

There is a technique that is especially helpful here: keep each commit small, releasable, and put test additions/changes in the same commit as the corresponding production code change. This requires a bit of diligence, and often use of git rebase -i to edit and squash commits. But if you do this, then it's far easier to re-order them safely, again using git rebase -i: the safety is achieved by running the tests* for each re-ordered commit to make sure everything works. So, if you do refactoring at the end of a feature branch, with small, releasable commits, you can then re-order them so the refactoring is put at the beginning.

There may be conflicts when you re-order, but if you fix them, run tests* on each commit, and it all works, then you know

the refactoring doesn't break any of the existing code or the feature;
the refactoring can be usefully released first.

So you do exactly that: you separate the refactoring commit(s) into a separate branch [git cherry-pick can be used for this], and go through your process to merge this into master and get it released. You then rebase your feature branch onto master, and ensuring to fix conflicts and making sure tests pass for each commit. At this point, master will contain the refactoring for the change, and the feature branch will just contain the [non refactoring] change. Repeat for as many refactorings as needed.

The rebase onto master, as opposed to a merge: this replays [the diff of] each commit from the feature branch, one by one, and with each you fix conflicts and can run tests. Here is where the benefit of rebase [over merge] with small releasable commits is revealed: you get smaller conflicts which are usually more managable.

Another benefit to this comes if you're not the only person working on this code. Getting each refactoring into master as soon as it's known to be required, means the rest of the team can rebase onto these changes, and deal with any conflicts sooner [and so typically with less or simpler code] rather than later. If each person in the team has been writing small, releasable commits on their own branches, then any conflicts can often be safely and quickly sorted out.

The most extreme example of this that I've done was for Django upgrade of a fairly complex application that was coupled to a lot of subtle/semi-private Django behaviour. To limit the risk of the Django upgrade + application code changes all going live at once, I repeatedly refactored the application code so the same code worked in both versions. Ultimately, there were about ~20 refactoring branches, each reviewed, QA-ed, and released, before the actual Django upgrade itself.

Note: it was crucial that the actual code changes are run in production to get value from this technique. If I had code like the below,

if DJANGO_VERSION == NEW:
  ...
else:
  ...

I would have been effectively incrementally adding a feature flag with lots of code behind it, which would have all been switched on on release day. This may be acceptable for an initial investigation, but before the actual production switch of Django version, it should be refactored to not have this conditional behaviour, as discussed below.

Feature flag [minimising code behind flag]

An alternative to a feature branch is a feature flag. This is in many ways similar to a feature branch, as in the feature code is kept from running by most users. However, with a feature flag the code is deployed to the production application, but it's only run by certain users with a certain flag enabled, or perhaps by hitting a certain URL.

This is often acceptable at the beginning of a feature [say, to get feedback on the UI of a feature before it's fully designed or working], but always consider what will happen down the line on release day. On release day, the feature flag will be effectivly enabled for all users, and potentially lots of code will be run by lots of users that wasn't before.

For this reason, don't enable the feature flag for everyone until the code behind the feature flag is minimised. Often it's possible to have not much more than just the UI that enables the particular feature. The rest of the code was run by large number of users long before the public release.

A great benefit of this technique is that often it is possible to incrementally refactor* a lot of the code out from behind the feature flag. Even if at the beginning there might be a fair bit of code behind it, you can go through several release/QA cycles, moving more and more of the code into production before release day.

A frequent example of this is changing a feature that uses 1 of "something", to using many. You can have a feature flag that enables the UI for this, but often to get this working requires changes through multiple layers of code, and possibly even the database.

What you can often do is to make a configuration option that limits the maximum of the "something", and configure all users to have a maximum of 1. Then you can work, layer at a time, making it work with N, but because each user is configured to have a maximum of 1, they will see no difference. Each layer can be tested*, QA-ed, and released. If there are problems, you have a lot of chances to get feedback before release day. Only when all the layers of the code are done, can you change the configuration option for production users.

In this particular case, once you know of nothing more to be done to the code, you don't even have to release the feature to all users all at once. You can slowly bump up the maximum for some users, in order to be able to get feedback from production, in terms of behaviour of the UI, from any reported bugs, and from performance. If you don't have a production use-case for each user to have a different maximum, then you can then set the maximum globally and remove some of the per-user configuration.

When to use which technique?

I don't know of a hard-and-fast rule: each case is different. However, some considerations:

A long running [but kept small] feature branch may not be feasible when more than one person needs to work on that feature branch. A feature flag is probably more helpful here: multiple people can more easily work on branches changing code behind the flag / refactoring it out from behind the flag.
Long-running feature branches, even if as small as possible, are harder to work with when there are lots of conflicting master branch changes: you may hit The Big Rewrite issues of maintaining two applications. A feature flag may be better here: impact on the "other" application will be visible in the code exactly when making changes, rather than making it a problem down the line when merging/rebasing.
A feature flag may not be helpful for a change that needs a lot of refactoring, but which is itself small in terms of final change that will be released on release day, and can/should be done by one person. A long-running [small] feature branch will likely be suitable in this case.

Addressing common critiscms

There are a a few common critiscms of the frequent-release techniques:

Each release introduces risk, and so releasing often is more risky than less.

At first glance, this argument makes sense. However, these techniques push you to split up what you will be releasing anyway, rather than releasing more things [which could be more risky].

Frequent releasing == testing less. This is risky! We don't do risk here.

Make sure each release is tested/QA-ed as you would any other change. This is not "move fast and break things", this is "do what you need to do to minimise the risk of each change that is going to happen anyway".

This is wasting QA time: you'll testing the same thing multiple times

If a change is deemed "large", then I think it's more appropriate to spend longer on QA: this is time spent reducing risk.

* It's important to have high-level feature/integration tests to support refactoring and safe rebasing. Low-level tests [often called unit tests] are less helpful.

S3 path/pseudo-folder locking

2018-11-18T18:50:00Z

AWS S3 is a key/value store, with operations that only operate on a single key at a time. There is no native concept of a folder: the closest thing is a group of keys with the same prefix. These facts mean that in applications that treat S3 as a filesystem, operations on such pseudo-folders, such as a renames or copies, are not atomic: if performed by different users at the same time, corruption of the key structure can occur. Here, I present a method to mitigate the chance of this based on locking.

Some familiarity with S3 is assumed, especially the PUT/GET/DELETE APIs for objects.

This post is not finished: expect lots of typos and little sense

Pseudo-folders, paths, and key prefixes

We describe pseudo-folders through an example. Consider 4 objects stored in an S3 bucket, with slashes in their keys. Other than the fact that they are in the same bucket, there is no structure.

- $ a/b/c $
- $ a/b/d $
- $ e/f/g $
- $ e/f/h $

However, since these keys have slashes, and they have common prefixes, we can seem them as though they are part of a tree [also known as a hierarchy] of paths, below the root path $/$, where the lowest branches in the tree map to the objects in the bucket.

$/$
- $/a$
  - $/a/b$
    - $/a/b/c → a/b/c$
    - $/a/b/d → a/b/d$
- $/e$
  - $/e/f$
    - $/e/f/g → e/f/g$
    - $/e/f/h → e/f/h$

The pseudo-folders in this example are those at paths $/$, $/a$, $/a/b$, $/a/b/c$, $/e$, $/e/f$ and $/e/f/h$.

This list of pseudo-folders includes the paths that map to objects: /a/b/c and /e/f/h. While a traditional filesystem does not allow a path to be both a folder and a file, S3 does not forbid the equivalent: for example both a/b and a/b/c could be keys to objects. This fact means that every path should be treated as a pseudo-folder.

Note that List Objects API allows you to list objects in a pseudo-folder, i.e. objects with the same key prefix, using the prefix and delimeter options, and the AWS console uses this to provide a reasonable illusion of folders. However, there is no API that provides atomic operations on all the objects on such lists.

Notation

S3 object keys will not start with a slash. Corresponding paths and pseudo-folders will always be written in full, and starting with a foward slash /.

Scope

We consider 4 categories of operations on pseudo-folders that we would like to be atomic.

Write & Delete: A write of one pseudo-folder
Rename: A write of two pseudo-folders
Copy: A read of one pseudo-folder, write of another pseudo-folder
Read: A read of one pseudo-folder

We don't try to maintain atomicity in the case of the server going down mid-way through such operations. We also don't concern ourselves with the eventual consistency properties of some operations on S3. We also only consider locking schemes, and only consider schemes where locking and unlocking on a given path takes a number of operations that is independent of the number of locks currently held, and independent of the number of descendents the path has, i.e. is constant-time.

Single-object GETs

It should be noted that a GET to a single object in S3 is atomic: it will either succeed with a consistent object, or fail with a 404. Therefore GETs to single objects won't typically need to be protected by locks. Note that whenever we mention a read of a pseudo-folder, we are concerned with a read that need an atomic view of the entire pseudo-folder. For example, if we needed to make a zip file from all of its contents.

Single-object PUTs however, even though they are also atomic, depending on the strucure of keys the application enforces, may need locking in order to maintain.

Locking interface & usage

Knowing the categories of operations we are interested in, we can design an API for a class, PathLock. If implemented, we could write something like the below.

import asyncio
from path_lock import PathLock

lock = PathLock()

async def delete(path):
  async with lock(read=[], write=[path]):
    ...

async def write(path, ...):
  async with lock(read=[], write=[path]):
    ...

async def rename(path_from, path_to):
  async with lock(read=[], write=[path_from, path_to]):
    ...

async def copy(path_from, path_to):
  async with lock(read=[path_from], write=[path_to]):
    ...

async def read(path):
  async with lock(read=[path], write=[]):
    ...

Each path argument can be some object that represents the path of a pseudo folder, such as PurePosixPath.

Granularity

As is typical with locks, we have the problem of making the lock as granular as possible: it should only block operations that would be unsafe to run concurrently, and other operations should progress unhampered. In our case unsafe combinations of operations on a pseudo-folder are "writes and writes", and "writes and reads". We aim to make a locking system that blocks these, but allows everything else to proceed

At first thought, it might appear that a read/write lock on each pseudo-folder could be enough. However, given that pseudo folders can be nested, it requires some thought to actually implement such a lock around such operations in order to acheive a granular lock and constant time locking.

We'll start with an example, and then generalise.

Consider a copy of $/a/b$ to $/a/b'$. This is a read of the pseudo folder at the path $/a/b$, i.e. from all possible keys prefixed with $a/b/$, and a write of $/a/b'$, i.e. to keys prefixed with $a/b'/$.

$/$
- $/a$
  - $/a/b$ (read)
    - $/a/b/c → a/b/c$
    - $/a/b/d → a/b/d$
  - $/a/b'$ (write)
    - $/a/b'/c → a/b'/c$
    - $/a/b'/d → a/b'/d$
- $/e$
  - $/e/f$
    - $/e/f/g → e/f/g$
    - $/e/f/h → e/f/h$

During this copy, the following operations are not compatible, and should be blocked to ensure atomicity:

reads of the pseudo-folder $/a/b'$
reads of any descendant pseudo-folders of $/a/b'$
reads of the ancestor pseudo-folders of $/a/b$ and $/a/b'$, namely $/a$ and $/$
writes of the pseudo-folders $/a/b$ and $/a/b'$;
writes of an any descendent pseudo-folders of $/a/b$ and $/a/b'$;
write of the ancestor pseudo-folders of $/a/b$ and $/a/b'$, namely $/a$ and $/$

All other read and write operations are compatible.

We can write these rules more succinctly using the concept of lineage. We define $\mathbb{L}(p)$ as the lineage of a path $p$: the union of all of $p$'s ancestors, descendants, and $\{p\}$ itself. We also define $R(A)$ as all the read operations on the set of paths $A$, and $W(A)$ as all the write operations on the set of paths $A$. Using these terms, we can say that a copy from the pseudo-folder at path $/a/b$ to $/a/b'$ should block

$W(\mathbb{L}(/a/b))$
$R(\mathbb{L}(/a/b'))$
$W(\mathbb{L}(/a/b'))$.

We generalise this further and state that for a path $p$

$R(\{p\})$ should block $W(\mathbb{L}(p))$
$W(\{p\})$ should block $W(\mathbb{L}(p))$ and $R(\mathbb{L}(p))$.

We can express these rules a different way, and say that for any path $p$

$R(\mathbb{L}(p))$ should block $W(\mathbb{L}(p))$
$W(\mathbb{L}(p))$ should block $W(\mathbb{L}(p))$ and $R(\mathbb{L}(p))$.

The second rule above implies the first, so we just need:

$W(\mathbb{L}(p))$ should block $W(\mathbb{L}(p))$ and $R(\mathbb{L}(p))$.

We can see that out aim is to be able to construct a read/write lock on $\mathbb{L}(p)$ for all paths $p$.

We can also express this as a compatibility table, showing what operations should, and should not, run concurrently for two tasks. We also define $\mathbb{L}^c(p)$ as the set of all paths not in the lineage of $p$ in order to better show how locking schemes differ in the concurrency they allow.

$W(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}^c(p))$	✓
$W(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✓
$R(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✓
$R(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✓

This table also gives us a way to compare locking schemes. All schemes must block the 2 incompatible combinations of operations, but that leaves 4 combinations they should allow for maximum concurrency, and so we can assign each potential scheme a score out of 4.

Global exclusive lock: 0/4

As an initial example, we can consider an extremly simple implementation of PathLock that ignores its arguments and defers to a single instance of asyncio.Lock.

As you might expect, this doesn't have great concurrency properties:

$W(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}^c(p))$	✗
$W(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✗
$R(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✗
$R(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✗

But the implementation is tremendously simple. Keeping in mind that single-object GETs don't need to be locked, this may be good enough for many situations.

Global read/write lock: 2/4

Better locking can be acheived by using a global read/write lock, with two modes $\overline{R}$ and $\overline{W}$, with compatibility table

	$\overline{R}$	$\overline{W}$
$\overline{R}$	✓	✗
$\overline{W}$	✗	✗

[To distinguish read and write locks from the similarly named operations, we write locks with a horizontal bar.]

which could be implemented as below.

As you might expect, this allows more concurrency than the global exclusive lock.

$W(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}^c(p))$	✗
$W(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✗
$R(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✓
$R(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✓

As when using the global exclusive lock, this may well be good enough for many cases.

Ancestor-locking and read/write locks on each path: 3/4

To improve upon the above, we construct an algorithm inspired by [1]. Instead of a global read/write lock, we maintain a read/write lock for each path $p$.

	$\overline{R}(p)$	$\overline{W}(p)$
$\overline{R}(p)$	✓	✗
$\overline{W}(p)$	✗	✗

When we need to access, either read or write, to a pseudo-folders, we acquire the write lock $\overline{W}(p)$ on the paths of the pseudo folders, and for all of the ancestor paths of the pseudo-folders, acquire a read lock $\overline{R}(p)$ on each.

In order to access pseudo-folders at paths $P$, we acquire the locks

$\overline{W}(p)$ for all $p$ in $P$
$\overline{R}(p)$ for all $p$ that are ancestors of the paths in $P$, except any in $P$

To avoid deadlock, the locks are acquired ancestor-first, and then in lexographical order on path-component name; and each task attempts to acquire at most one lock at a time [proof]. For example, if a task needed to to access $/a/b$ and $/a/b'$ it would

acquire a read lock on $/$,
acquire a read lock on $/a$,
acquire a write lock on $/a/b$.
acquire a write lock on $/a/b'$.

If a concurrent task wanted to lock anything in $\mathbb{L}(/a/b)$, it would get blocked. For example, if another task tried to access an ancestor or $/a/b$, such as, $/a$, it would try to acquire the write lock on $/a$, and get blocked by the original task's read lock on $/a$. Alternatively, if a task tried to access a descendant of $/a/b$, such as $/a/b/c$, it would get blocked by the write lock on $/a/b$.

A downside of this is that there is no difference between reads and writes, which means that concurrent reads to the same lineage are forbidden, as can be seen in the below compatibility table.

$W(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}^c(p))$	✓
$W(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✓
$R(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✗
$R(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✓

However, keeping in mind the single-object GET exception, this may be acceptable.

Note also that if a path isn't locked, then the lock object itself doesn't need to be retained in memory. As in the implementation below, using a weak-reference cache can be used for these.

Ancestor locking and Read/write/ancestor locks on each path: 4/4

We can improve on the above by maintaining a different sort of lock on each path. Instead of a read/write lock with two modes, $\overline{R}$ and $\overline{W}$, we construct one with 4: $\overline{R}$, $\overline{W}$, $\overline{R_A}$, read-ancestor, and $\overline{W_A}$, write-ancestor.

	$\overline{R_A}(p)$	$\overline{R}(p)$	$\overline{W_A}(p)$	$\overline{W}(p)$
$\overline{R_A}(p)$	✓	✓	✓	✗
$\overline{R}(p)$	✓	✓	✗	✗
$\overline{W_A}(p)$	✓	✗	✓	✗
$\overline{W}(p)$	✗	✗	✗	✗

In order to read the pseudo-folders at paths $P_R$ and write the pseudo-folders at paths $P_W$, we acquire the locks

$\overline{W}(p)$ for all $p$ in $P_W$
$\overline{W_A}(p)$ for all $p$ that are ancestors of the paths in $P_W$, except any in $P_W$
$\overline{R}(p)$ for all $p$ in $P_R$, except any in in $P_W$ and any ancestors of $P_W$
$\overline{R_A}(p)$ for all $p$ that are ancestors of the paths in $P_R$, except any in $P_W$, and any ancestors of $P_W$, and any in $P_R$

To avoid deadlock, these lock are acquired ancestor-first, and then in lexographical order on path-component name; rand each task attempts to acquire at most one lock at a time [proof].

Using this set of locks, we then can acheive a perfect concurrency score of 4/4, and acheive the following concurrency table.

$W(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}(p))$	✗
$W(\mathbb{L}(p))$	$W(\mathbb{L}^c(p))$	✓
$W(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✓
$R(\mathbb{L}(p))$	$R(\mathbb{L}(p))$	✓
$R(\mathbb{L}(p))$	$R(\mathbb{L}^c(p))$	✓

As per the version that uses read/write locks, if a path isn't locked, then the lock object itself doesn't need to be retained in memory: a weak-reference cache can be used.

Proof that ancestor-locking is deadlock-free

The locks are always acquired along a sequence: ancestor first and then lexographic, and therefore the locks form a total order. So we prove that for any algorithm where each task acquires its locks in a strictly increasing sequence along a total order, waiting for at most one lock at a time, that deadlock is impossible. The proof doesn't depend on properties of the modes of the lock, and so can be applied to both of the ancestor locking methods presented in this post.

Conditions

Let $L=\{L_i\}_{i \in S} $ for some $S \subset \mathbb{Z} $ be a set of independent locks, each with at least one mode, and let $<_L$ be a strict total order relation between elements of $L$. Let $T = \{T_i\}_0^{M-1}$ for some $M \in \mathbb{N} $ be a set of tasks, where each acquires locks in $L$ in a strictly increasing order according to $<_L$, and at any given moment is waiting to acquire at most one lock.

Statement

It is not possible for $T$ to be in deadlock.

Proof

Assume $T$ are in deadlock. We'll show that this leads to a contradiction.

Without loss of generality, we index the $ \{L_i\}_{i \in S} $ such that they form strictly increasing sequence according to $<_L$

\[ L_0 <_L L_1 <_L \ldots .\]

By definition of deadlock, $M \geq 2$, each $T_i$ has acquired at least one a lock, and each is waiting for a lock held by another task in $ T $. For each $T_i$, let $ A_i $ be the indexes of the locks that $T_i$ has acquired, and let $w_i$ be the index that it waits for. Since all tasks acquire their locks in increasing order,

\[ L_a <_L L_{w_i}, \forall a \in A_i \]

or equivalently

\[ a < w_i, \forall a \in A_i \tag{1}\label{eq:a} .\]

Without loss of generality, we index the tasks such that each $ T_i $ is waiting for a lock held by $ T_{i+1 \pmod{M} } $. So each $w_i \in A_{i+1 \pmod{M}}$, and so by \eqref{eq:a} $w_i < w_{i + 1 \pmod{M}} $. This is true for all $ i \in {0 \ldots M-1} $, and so

\[ w_0 < w_{1 \pmod{M}} < \ldots < w_{M-1\pmod{M}} < w_{M \pmod{M}} \]

and therefore

\[ w_0 < w_{M \pmod{M}} .\]

However, $M \pmod{M} = 0$, which implies \[ w_0 < w_0 .\]

This is a contradiction. $\square$

[1] Ritik Malhotra; An Efficient Locking Scheme for Path-based File Systems; https://people.eecs.berkeley.edu/~kubitron/courses/cs262a-F14/projects/reports/project6_report.pdf

Python asyncio read/write lock

2018-11-11T14:50:00Z

In this post, I present an implementation of an asyncio read/write lock [also known as a shared/exclusive lock].

Since this post was written, the lower-level class, FifoLock, has been released separately.

What is a lock?

When you have concurrent tasks, there may be parts of the code that have to be protected from being run by multiple tasks concurrently. A typical example is non-atomic reading from a file, concurrent with non-atomic writing to that same file. If a reading task reads the file half-way through a writing task, the reader may get a corrupt file: seeing some of the file before the write, and some of the file after.

A way to prevent this is to use what's called a lock. Just before such code that needs to be protected, each task attempts to acquire the lock, and just after the code, it releases the lock so other task can acquire it. If a task cannot acquire the lock, it will wait until it can. [There are other possibilities if a task can't acquire the lock, but for simplicity, we just assume it waits.]

Exclusive lock

There are different sorts of lock, but a typical example is an exclusive lock. Only one task can acquire this lock, all others must wait. Using an exclusive lock is a way to prevent corruption when multiple tasks are reading from/writing to a file.

Python's standard library comes with an exclusive lock. This can be used, for example:

import asyncio
lock = asyncio.Lock()

# ...
async def locked():
    # ...
    async with lock:
       # only one task can be here at any given time

Note that a lock in this context isn't really analogous to a lock on a door. An exclusive lock is better compared to a talking stick or a talking spoon, in that only the person who has acquired the talking stick can talk, others must wait until they acquire it. For other sorts of locks, it's a bit more complex and the analogy can break down.

Read/write lock

Using an exclusive lock is often enough to prevent corruption, but it may block tasks that would otherwise be safe to proceed. For example, concurrent readers on a file is often safe, and only if there is a writer should other tasks be blocked. To allow this, we use a different sort of lock: a read/write lock.

A read/write lock is made of of two parts: a read lock and a write lock. The read lock can be acquired by multiple tasks concurrently, but only as long as no task holds the write lock; and a write lock can only be acquired if no tasks hold the read lock and no task holds the write lock.

This can be summarised in the following table, showing what parts of the lock can be held concurrently by two tasks, A and B.

B A	Read	Write
Read	✓	✗
Write	✗	✗

Note that this table isn't quite enough to completely define how the lock behaves. When there are multiple tasks waiting to acquire the lock, some algorithm must be implemented to decide which tasks can be acquired. The implementation presented here allows tasks to proceed in a first-come-first-served order: a queue of tasks waiting to acquire the locks is maintained.

Note also that this type of lock isn't limited to reading or writing, hence this sort of lock is often called a shared/exclusive lock. However, it is so often used in the context of reading and writing, it is convenient and typical to call this a read/write lock [or similar].

Read/write lock: usage

The usage is very similar to the built-in exclusive lock, except that the lock object is callable with a required argument, an object representing which of the Read and Write lock modes is desired.

from asyncio_read_write_lock import FifoLock, Read, Write

lock = FifoLock()

# ...
async def read():
    # ...
    async with lock(Read):
       # multiple tasks can be here, but only if there are no writers

# ...
async def write():
    # ...
    async with lock(Write):
       # only one task can be here at any given time, and only if no readers

The lock class FifoLock is general, and can support more complex locks than read/write, although here we just use it as a read/write lock.

Read/write lock: comparison with aiorwlock.

There is already a read/write lock available for asyncio at aiorwlock. There are differences, which may be important for your use-case.

This implementation has no support for Python versions less than 3.7, while aiorwlock appears to support earlier versions. This lack of support is a feature: it allows for simpler code.
Python 3.7's contextlib.asynccontextmanager is used internally. It allows for very concise creation of an asyncio context manager, without sacrificing flexibility or ability to handle errors.
This implementation has no support for acquiring a write lock already held by the current task: attempting to do so would result in a deadlock. My use cases do not require this, and my instinct is to try to write client code so that this is not required. This would make the code of the lock more complex, and I suspect in a lot of cases using this would make client code more difficult to reason about.
It appears that the algorithm aiorwlock uses to acquire a lock is O(n) with respect to the number of tasks already holding a lock. This implementation is O(1) with respect to tasks holding a lock. Of course, this fact alone does not mean that this implementation is better or faster in practical terms.
Locks are acquired on a first-come-first-served basis, compared to aiorwlock giving preference to waiting writers: giving preference to waiting writers can starve the readers if the write lock is continually requested. Which of these is more appropriate depends on your use case, but my instinct is that unless you are sure there will will never be enough writers to starve the readers for any meaningful length of time, it's better to avoid such a bias.

Read/write lock: implementation

The implementation is below.

You might not need Boto 3

2018-10-28T11:15:00Z

It is not a requirement to Boto 3 in order to communicate with AWS from Python: you can make requests using any HTTP client, as long as you can work out the correct headers. Here's a function that does just that [with some caveats].

For example to PUT some data to S3 using aiohttp:

Why use this, and not Boto 3?

Your application is event-loop based

This was my original reason for writing this: Boto 3 blocks the event loop. You can patch it or wrap it, and there are libraries that do this, but that's even more code in your application, when you could have less.

You can use the HTTP client you're already using in the rest of your project

You may prefer that all your outgoing requests to go through the same function / library as the rest of the application. Say, for consistent logging, proxy configuration, or things like enforcing a global limit on outgoing connections.

Free to boost performance: disk space, memory, and speed

An install of Boto 3, with botocore, takes up at least an extra 35mb [and more depending on what other dependencies you already have]. If you're already using a HTTP library, then the function above adds 2.5kb [although yes, there could be more code at the call sites].

Or, you want to tease out as much performance as possible, in terms of speed or memory usage. You might not need to parse the entire JSON or XML from results if you don’t need to: you are free to be as performant (/hacky) as you like by using regex or just string matching against results; or you can use a faster JSON or XML parser; or you would like the option of doing these things in the future without lots of changes to your application or test code.

You don’t want to wait for fixes or features to make it into Boto 3

Boto 3 is not perfect [as no project is of course!], but also the developers don't have the same priorities as you. Having to wait for an upstream library update, where you have no influence over or knowledge of when the update will happen, is often not good place to be.

For example, AFAIK aws-chunked uploads to S3 are not yet supported by Boto 3. [To be fair however, I suspect it's often of limited use, because you need to know the content length ahead of time, unlike standard HTTP chunked transfers.]

Another example is being able to change how/when the SHA-256 hash of the payload is calculated. The hash could be computed incrementally, and not all at once just before upload, and in a recent project I changed the signing function and surrounding code in order to be able to hash the payload as data comes into the application. AFAIK, this isn't possible with Boto 3.

You want limit the bug surface area

If something isn't working, is it your code, Boto 3, botocore, or AWS? You might want to limit the scope of investigations. For example, on a project I was involved with, repeatedly creating the Boto3 S3 client appeared to cause a memory leak.

You want it to to be possible to understand what the application is doing

You might have security requirements that code in your application must be reviewed. Doing this for every version of Boto 3 you need may not be a great option.

You want the power to change the higher-level interface

Boto 3 presents a higher-level interface to AWS that might not quite match up with the rest of your application. For example, how Boto 3 uses generators, exceptions, or even naming, may not be how the rest of your application is structured, and you may judge that consistency in your codebase to be valuable.

You would like to directly apply the documentation from AWS

The AWS documentation has a lot of examples: at least one on every action I've looked it. I've not seen this in Boto 3's documentation.

You don't want another layer of abstraction

Abstraction layers are often not free in terms of performace, ability to reason about behaviour, bugs introduced, or ability to change as your requirements change. [This is admittedly a succinct way of presenting several of the above points]

You have a mocking/testing setup with other HTTP requests, and don’t want to include another one

Moto is great, but this is yet another dependency, and obviously is only mocking Boto, and so only mocking calls to AWS. You may want to be completely free as to where you put the boundaries of your test, and a consistent approach when dealing with boundaries in tests is valuable. Also, using Moto also means you're even more coupled to Boto 3.

You use temporary security credentials, and want greater control over state

They are typically cached for some period of time, and you might want to manage state, especially state relating to security, very particularly. Or you might not even want to cache the temporary security credentials at all. [Note: I'm not sure if there will be some rate limiting issues here if you hit endpoints to retrieve the temporary credentials too frequently]

Is this battle tested?

No, but I have used it [or at least very similar code] in a few projects: communicating with S3, ECS, Elasticsearch, and IAM.

Any gotchas?

The AWS APIs are not completely consistent with each other. Some use JSON, some XML, some are more REST-ful than others, some have their actions specified in a query string, or some in a HTTP header. However, the documentation appears thorough for each API, and the APIs are quite good in terms of HTTP status codes, which means often you don't care about exactly what's returned. other than the status code.

The actual name of the service, passed in as the service parameter to to aws_sig_v4_headers, seems to require an educated guess. For the cases I've tried it seems to match the hostname used in the AWS API endpoints. For example, for a call to the Elasticsearch endpoint of es.us-east-1.amazonaws.com, then the service would be es.

When should I use Boto 3?

If you decide that the part of it that you're using really does add value on top of the AWS API, and that value is worth not having the upsides listed above.

Should this be factored out to a shared library?

Maybe. However, I have opted to not do this for my cases so far, to ensure each is a flexible as possible with only the complexity it needs. For example, this approach has allowed me change the code in a single project, which required incremental hashing of payload, without affecting others.

Design decisions and limitations

It's an API exposing a single function, but still had some thought. This was designed for my use cases: you may have others, and so may need a different function [or functions].

Dictionaries of string -> string are used for the headers and queries, This is concise, flexible in terms of manipulation I require in constructing API requests, all the HTTP libraries I've worked with support this, and no extra dependencies are needed. However, this means that duplicate header and query keys are not supported. However, I don't have a use case for duplicate header or query keys.
Payload must be passed in as b-string/bytes instance in all cases, including the empty b-string for GET requests without a body. There are no assumptions made on what HTTP methods can have payloads, and no automagical conversion, for flexibility.
The x-amz-content-sha256 header is only needed for S3 [from what I can tell], but doesn't seem to matter if you send it anyway to other APIs [also from what I can tell]. The hash of the payload is required to be calculated anyway, it's sent anyway as part of the Authorization header, and for my cases, this bit of extra work and bytes sent is an acceptable trade off for having a simpler and consistent signing process for different AWS APIs. If the day ever comes where this trade off is not acceptable, I have the power to change this.
There is no automagical conversion of region + service to host name [or vice-versa]. Instead, there is explicitness, flexibility, and less dynamic behaviour in order to make things easier to reason about.
The path is not normalised in terms of removing duplicate slashes. For requests to S3, this shouldn't be done anyway, and for other APIs I'm happy to just put the requirement of constructing normalised strings up the call stack in my code: I was doing this anyway so this is no extra work.
Requests to non-S3 APIs that have characters that need URL-encoding in the path apparently won't work: they should be URL-encoded twice, and this function encodes them once. To use this function in such cases, you would need to encode the path before it being passed to this function, or use a different function. I've not needed to handle this case, so the function above doesn't handle it.
There are a lot of arguments to the function. I considered alternatives, such as wrapping some up in data structures, or putting the onus on the caller to put together parts of the signature, but this made things more, rather than less, complex. Hiding complexity often gives you nothing but more complexity.
Internally, there are some functions defined that are technically unnecessary since they are called once. I felt it was more important to show the flow of data, and what calculations depend on what previous calculations, in order to be able to tweak it if necessary. You can of course change it to not have these if you prefer.
It uses Python 3.6's f-strings, and so does not support earlier versions of Python. I have no use case for earlier versions of Python.
The function is not pure: calls datetime.datetime.utcnow(). You could easily separate it into a pure function that takes in the current time if you prefer. I didn't have a use case for this, so I didn't do it. For testing, I'm happy with tests that use FreezeGun. However, with that exception, all the expressions inside the function are referentially transparent. [I would refer to this as an application of pragmatic purity].

Questions to ask yourself when handling exceptions

2018-07-23T23:00:00Z

Can I think of any combination of input where the exception can be thrown?

If there is no combination of input that you can determine that can cause the exception, then if somehow it does get thrown, your assumptions on how the program behaves have been violated. Consider letting the error bubbling up, and defering to a general error handler. In a web context, returning a 500 would be a typical example of how a general error handler would behave. Wonderfully, this is the default behaviour for many web frameworks.

Is exiting safest?

In some situations, you may even determine that exiting is safest. For example a web-based application may try to access configuration on load, and raise an exception if it can't, which if unhandled, would [by default in many languages] cause the program to exit. In a blue-green deployment pipeline, it's desirable for the application to exit on load if misconfigured, since then the deployment will fail, and the existing deployment will remain live.

Am I avoiding a 500?

500s can be helpful. If it's best to abort because your assumptions have been violated, 500s often get to the front of the queue in front of getting addressed. You're also not wasting your client's time, since the problem is on the server, not with the client as the 4XXs suggest.

Am I handling this exception this because I don’t known the range of inputs to this bit of code?

Find out what the range is. If it's hard to work out, you have just made it even harder to understand what bit of code runs under what conditions. This isn’t a "code for the future" argument, it’s making sure you know what the code does right now, for today’s behaviour [although this would also help many futures].

In some cases, it may be appropriate to iteratively refactor surrounding code to make it easier to reason about.

[This is also an argument for type safety as well, since often it's easier to be sure on ranges of inputs. However, it's not possible or realistic in many situations.]

To test the handler have I had to mock an implementation detail?

It might not be possible for the exception to be thrown in production based on input. Consider not handling the exception where it can be thrown, and instead letting it bubble up to a general handler.

Does testing the handler feel unnecessary?

You may already suspect the handler itself is unecessary. Consider removing the handler, and letting the exception bubble up to a general handler.

[Listening to instincts can be risky, but one that pushes you to have less or simpler code, I think is worthwhile to examine]

Am I subtly introducing features?

If you're catching an exception and then marching forward with a default, you have just added a feature. Make sure this is what you want: there is often a long term cost of this. Is there really a need for this feature? How have you decided the default? Often the default is chosen as the original behaviour before the change: is the best option for the current range of behaviour of the code?

Defaults may be [currently] dead code, or they may be introduce complexity. If you do want defaults, be careful of littering them throughout your code: this makes it hard to reason about their behaviour now, and hard to change later. Better is to have a specific normalization phase, where the input is converted to some standard form. If you do have checks on the data after this phase, defer to a general error handler if the data is not as it should be. However, be mindful that if you can't cause these checks to fail through any combination of input, you may have effectively dead-code.

Am I writing exception handlers for a general library that is used on one case?

If yes, you may be wasting time. It may be better to just handle the case you need to handle.

Can the case the exception handler is designed to handle, be handled earlier?

If the exception can happen though some combination of input, it might be better to refactor the code so the error case can't happen in the bowels of your algorithm/processing, but instead happens earlier. This is already hinted at in other questions, but a rough order of preference of when it's good to deal with such things:

Compilation
Tests
Start of the run of the program
Input validation
Input normalization
Input processing

I'm handling low-level exceptions by raising higher level/custom exceptions. Am I sure hiding the real cause is helpful?

It's often tempting to provide a "pure" level of abstraction by hiding everything the code is actually doing. However, consider what someone will need to do to investiagte this exception. If they would need to look into the lower-level cause, you have just made extra work for them unnecessarily.

As a guideline, if a layer of abstraction helps you reason about today's behaviour, feel free to add it. If it doesn't, or if you can only see how it's helpful in some future version or use of the system, be very cautious.

Am I increasing the risk of security holes?

One of the most important aspects of secure code is your understanding of what it's doing. If your assumptions on how it behaves are violated, this is evidence that the code is not as secure as you thought. Be very sure how you would like to handle this.

Keep security in mind if you are deciding to add features, which you may be doing by handling exceptions. You are increasing complexity, which makes it harder to understand the code, and again, less secure.

Am I lowering the chance of a negative consequence, by increasing the chance of a worse consequence?

Exceptions are often handled for "safety", under the assumption that letting it bubble up is "unsafe". It's often not so trivial as this: you have to reason about each case and determine what is safest in that situation. There is no perfect or always-applicable way of doing this, but the below list is in decreasing order of "safety" for many situations.

Not having a feature
A feature is broken by aborting
A feature is broken by appearing to work, but it does something unexpected
Leaking data

Be very cautious that you're not needlessly increasing the risk of items further down this list. A data leak can have extremely long term consequences, and "I didn't understand the code, but I was avoiding breaking a feature for one of our customers" will not fly at the subsequent public inquiry. I'm not even kidding.

Structured data in environment variables

2018-07-07T12:30:00Z

I recently wrote a small Python application that had to be configurable so it can run differently in different environments. As is often done, I used environment variables for this.

However, for various reasons beyond the scope of this post, it was helpful to have structured data stored in the environment: lists, dictionaries, and even lists of dictionaries. So when the application runs, it should be able to extract this and get something that looks like:

config = {
    "FOO": [{
        "BAR": "setting-1",
        "BAZ": "setting-2",
    }, {
        "BAR": "setting-3",
        "QUE": "setting-4",
    }],
    "FIZZ": [
      "setting-5",
      "setting-6",
    ],
    "BILL": "setting-7",
}

This could be done by encoding such a structure into a single string, perhaps using JSON, throwing it into a single environment variable, and decoding it in the application. However, this would be fairly painful to edit and difficult to debug in many situations.

So I came up with a simple format where each setting is in its own variable, but itsname defines where it is in the structure. Specifically each name is

a double-underscore separated list of path components;
and where each level of components defines a dictionary unless all of them parse as integers, in which case the level defines a list.

For example, the above config could be defined in bash by:

export FOO__1__BAR=setting-1
export FOO__1__BAZ=setting-2
export FOO__2__BAR=setting-3
export FOO__2__QUE=setting-4
export FIZZ__1=setting-5
export FIZZ__2=setting-6
export BILL=setting-7

and to convert this list of environment variables to nested Python dictionaries and lists, you can use normalise_environment.py:

import os
config = normalise_environment(os.environ)

This way of incorporating structure into environment variables has some nice properties.

The double underscore is likely to not cause issues because underscores are used in environment variables frequently.
No (extra) escaping needed. You can set them easily on the command line, any old bash script, Travis, CircleCI, CloudFoundry, ECS task defintions.
You can then read them really easily, which is good for debugging.
If you don't currently use any double underscores in environment variable names, you can put your entire environment through normalise_environment.py, and it will be unchanged. This helps in moving existing config to a more structured form in small steps.
Although the function normalise_environment.py is helpful in Python, it's by no means necessary. Any code that can read environment variables can access their contents, without extra steps or dependencies involved.

What to call this

I'm not sure how best to refer to this technique (it's almost so trivial calling it a "technique" is a bit much...). It's not quite flattened since the structure is preserved. Storing the variables denormalised is the best I have, since it can be argued that the location of each variable in the structure is denormalised.

Programming design choices in the face of changing requirements

2018-02-25T12:30:00Z

In an Agile environment, you are likely keeping in mind the following 3 Agile principles when working.

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Working software is the primary measure of progress.

So if asked to make a feature, say Feature A, you split the work up into smaller, working, and deliverable pieces. Keeping also in mind the simplicity principle,

Simplicity--the art of maximizing the amount of work not done--is essential.

you decide to split the work so you do the simple but useful things first:

Feature A₁: some subset of the final feature set that is useful but simple and quick to code up
Feature A₂: an extension of A₁ delivering all the required behaviour and complexity

After you've done Feature A₁, keeping in mind the Agile principle

Business people and developers must work together daily throughout the project.

you demonstrate the functionality so far. Then comes the feedback

This is great! We can really see how this solves our problems. Feature A₁ acheives 90% of what we need. The last 10% from Feature A₂ is quite low priority now, and we're even talking about never doing it. Instead, can we make a start on Feature B?

Fortunately, when coding up Feature A₁, you also kept in mind to

Welcome changing requirements

so you didn't make a start on the complexity for Feature A₂ because you suspected the requirements would change after Feature A₁ was seen by the business people. You also kept in mind

Continuous attention to technical excellence and good design enhances agility.

so you didn't make Feature A₁ in any way a hack: it is safe, tested, what it does and why it does it is extremely clear, and there isn't any unnecessary code. You have spent no time working on something that isn't useful, and you could not have given the developer of the next feature a better gift.

Type simplicity

2018-01-04T20:00:00Z

There is a "feature" of type-safe language: it is often an effort to use union types. This has the consequence that the developer is given a little push to avoid such types, and consider alternatives.

In this post I argue that the simpler, non-union, types and corresponding code are often easier to reason about, and less likely to have bugs. Therefore, even in a in a type-unsafe environment, you should give yourself this push for simpler types.

What are union types?

They are types that have two (or more) cases. For example, in Haskell, you have to be explicit, via a bit of effort, as to "which" type is used. A basic situation is using Either, which allows a Left type and a Right type:

a = if condition then Left 3 else Right "three"

In this case, a would have the type Either Int String. [This is slightly inaccurate due to typeclasses, but this isn't relevant here.]

When in an type-unsafe environment, variables don't have types, just the data they refer to does. However, it is often useful to mentally note down the all the types of variable's data and construct "what the type of the variable would be" if the environment had the concept of variables having types.

For example, in Python, you can write:

if condition:
  a = 3
else:
  a = "three"

where the type of a would be Union[int,string] (using Python's type annotation syntax). The similar Javascript code:

var a;
if (condition) {
  a = 3;
} else {
  a = "three";
}

would result in the type of a being the union type number | string (using Flow's type annotation syntax).

A more complex, but more realistic example, would be an object with differerent keys:

var obj = {}
if (condition_1) {
  obj.key_1 = "value_1";
}
if (condition_2) {
  obj.key_2 = "value_3";
}
if (condition_3) {
  obj.key_2 = "value_3";
}

Slightly shockingly, this is a union of 8 types:

{}
| {key_1: string}
| {key_1: string, key_2: string}
| {key_1: string, key_2: string, key_3: string}
| {key_1: string, key_3: string}
| {key_2: string}
| {key_2: string, key_3: string}
| {key_3: string}

[You can write a similar type more concisely in Flow. The types are written out here to make clear all the different cases.]

Special mention: Maybe/Optional/None/null

These are a common union type that crops up in code. In Haskell again you have to be explicit adding a Just to wrap a value:

a = if condition then Just 3 else Nothing

However in Python you can just set the values as desired,

a = None
if condition:
  a = 3

where the type would be Union[int, None] or more concisely Optional[int]. Similarly you write in Javascript:

var a = null;
if (condition) {
  a = 3;
}

where the Flow type annotation would be ?number.

Special mention: optional arguments with default of None

It is common to use optional arguments to function that have a default of None:

def my_func(a=None):
  ...

You should be conscious of the fact that, unless None is the only value ever passed, the arguments will be of a union type.

Special mention: variable only sometimes defined

In imperitive languages, it's often possible to only sometimes define a variable. For example the_filter is only sometimes defined below.

if condition:
  the_filter = filter_function

In Python, it would causes a runtime error if you try to access the variable when it's not defined. However, it's useful to keep in mind that there are 2 cases, so the type of the variable can be usefully classed as a union, of the defined and not defined cases.

When and why should they be avoided?

"Avoided" is perhaps too strong. However, since they result in a number of different cases the code has to deal with, choosing an alternative that reduces the amount of code that has to deal with multiple cases may result in code that is easier to reason about, and even improve effectiveness of tests and manual QA.

Alternative: Single simple type

Often, types are simple in an initial version of the code. They are then transformed into union types when adding a feature, where the original case should behave as before. Wonderfully, it is often possible to refactor the old case to be a special case of the new.

For example, if you're adding a feature where an existing list is filtered by a filter based on some conditions, you may have written:

selected_filter = None
if condition_1:
  selected_filter = filter_function_1
elif condition_2:
  selected_filter = filter_function_2

if selected_filter is not None:
  data_filtered = filter(data, selected_filter)
else:
  data_filtered = data

The type of selected_filter would be Union[function, None]. You can rewrite this to always set a filter, using a constant function for the original case:

selected_filter = \
  filter_function_1 if condition_1 else \
  filter_function_2 if condition_2 else \
  lambda x: True

data_filtered = filter(data, selected_filter)

In this version, the type of selected_filter would just be function. There are fewer cases in the code, all things being equal, this makes the code easier to reason about.

Also, the difference in what code is run between cases is extremely small: every single case will call the filter function. Therefore tests written before the filtering feature was added would still call filter. If the tests pass, this is evidence that the new behaviour will work just as expected in all cases.

Further still, calling filter creates a new list. If there is code mutating lists somewhere, it is crucial to make sure the correct list is being mutated. Always creating a new list rather than sometimes creating a new list helps to avoid subtle bugs relating to this. Tests can't cover every combination of cases, but we can maximise their value by making sure differences between the cases they do test is minimal.

[There are arguments against using the ternary operator, especially the Python ternary operator due to the order of arguments. However it does lend itself to making sure that variables are always defined, and I find the above layout for the nested case fairly easy to parse to understand the different cases and values.]

Alternative: Multiple simple types with separate code paths

In some cases the behavior of the cases of the types are quite different, but for historical reasons are munged together. When processing the type, there could be runtime checks for something already known up the call stack / by the client. An alternative would be to split the code at the earlier point, using different types, with two entirely different code paths for the different types.

For example, a "Send email" button may have been written first, and a "Save draft" feature added later. The "Safe draft" button POSTs similar, but not the same, data to the original endpoint on a server. This is often done in the name of DRY, and to avoid touching existing code as much as possible. However, sending email and saving draft are now tightly coupled together and hard to reason about, so making changes are likely to be slow or introduce bugs.

Better would be to have a separate endpoint for drafts. Two data types would be in-play, an "email to send" type, and "email to save" type, and the server wouldn't need to dynamically determine which: it's known by the which endpoint the data was POSTed to. The two cases are separate from the point of hitting the buttons: each endpoint would only contain code relevant to that case, without duplicated runtime tests for things already known. Therefore each should be easier to reason about and easier to make futher changes to. There may be some duplication on the server, but this can be minimised by factoring out common code from the two endpoint handlers if desired.

Questions to ask yourself when writing tests

2018-01-01T20:00:00Z

Am I confident the feature I wrote today works because of the tests and not because I loaded up the application?

If no, consider instead writing higher level tests that each test more of the code.

I've tested a number of cases manually. Are these cases covered by tests?

If no, write tests covering these cases. This may be some combination of higher and lower level tests. Don't worry if some code is tested multiple times. Often, testing glue code is worth the cost of this re-testing.

What would the tests have looked like if I wrote them before I wrote the code?

It's not always possible to envisage what the tests should be before writing the code: some tasks require a bit of exploratory work. However, if you're writing tests after the fact, considering what you could have written beforehand may help writing better tests, since it may push you towards testing behaviour rather than implementation details.

If there were no tests in the codebase at all, which ones would I write now to test my changes?

It's easy to be affected by what's in the codebase already, especially if it was written by you. Try to avoid this bias by imagining a clean slate, and what a "perfect" test would be. There may be contraints so you can't acheive this: often time or language constraints, but you might be able to do something better than what there is already.

Are the boundaries of the code under test, themselves tested?

Each boundary introduces an assumption, which introduces risk, since it might be false. Consider higher level tests that each test more of the code. This may mean the same code may be tested from different levels, but this is a reasonable, since it means the risk of incorrect assumptions is lower.

Have I just made something public in order to test it?

If yes, consider instead writing higher level tests that each test more of the code.

Am I testing something trivial?

If yes, consider instead writing higher level tests that each test more of the code.

Have I just mocked something that is fast and deterministic?

If yes, consider not mocking it. Mocking introduces assumptions, which introduce risk. Code may be tested multiple times at multiple levels, but having fewer assumptions is often worth this cost.

For every part of my code change, if I break the code, will a test fail?

100% code coverage is not enough for this, since that does not ensure components work together. To avoid a high number of low-level tests, consider instead writing fewer higher level tests.

Am I avoiding testing something because it's hard to test?

If it's hard to test automatically, it is likely be hard to reason about or test manually. Thefore the risk is higher, and it's actually more important to test such things.

Is the None/null case tested?

Often it's better to design the production code so there is no None/null case. However, if there is one, it should be tested.

Are boundaries of the problem domain tested?

The most common of these are off-by-one errors, so often good to check each side of a numerical boundary. Yes, some code will be re-tested, but the increase of confidence is worth the cost.

What is QA about to do? Can I add automated tests for that?

If you're about to hand over to QA with a list of things to test, consider writing automated tests for them.

Have I just copied and pasted the code of the function into the test?

If yes, then the test doesn't have much value. Consider instead a higher level test that asserts on more code.

What situations would cause the test to fail?

If the only realistic answer is that the developer deliberately made a change to some code and the behaviour of the software continues to be correct, then the test is likely testing implementation details. Consider writing a higher level test, covering more code.

If I refactor implementation details, would the tests pass?

If no, consider instead writing higher level tests that each test more of the code.

Defining "implementation detail" is partially an art. However glue code that just passes data about, that doesn't actually do the data processing or output that a feature requires, can often be usefully classed as an implementation detail.

Can I add likely features with the tests still passing?

If no, the tests are likely to assert on too much. For example, "snapshot" tests that assert on the entire state of the DOM, or maybe an entire PDF.

Better are "targeted high level tests". For example, instead of asserting on the entire state of the UI, just test certain parts of it, such as what is displayed in a drop down after a click on some element. [This is not an E2E test: remote services can still be mocked.]

Working out what a "likely" feature is partially an art, and benefits from experience and domain knowledge. This is a reason for developers to be quite close to client contact, be involved in design decisions or at least have as much contextual information as possible before starting to code. Often they are very business-logic related: adding or "tweaking" what data is allowed as input, stored, given as outputs, or associated UX changes to allow these. They are rarely low-level or infrastructure changes: changing database, data transfer formats, or code framework.

What would a developer do if the test fails?

If the answer is that the test is likely to be changed to pass with the current production code's behaviour, then the test doesn't add any value, and it either asserts on extremely trivial behaviour, or it asserts on too much, such as a snapshot of an entire DOM. Instead consider a "targeted high level test".

Are my tests helping me achieve my aims?

If they don't, write different tests. Potentially, you might have to restructure the production code as well.

The list monad's bind

2017-06-10T10:00:00Z

The word monad can be seen as scary, but you don't particularly need to worry about what it means in the general case in order to use the implementations of >>= (bind). For example, the list monad, where >>= is defined as below.

(>>=) :: [a] -> (a -> [a]) -> [a]
xs >>= k = join (fmap k xs)

When evaluating >>=, each element of xs is passed to k, which returns a list, and then the lists from each invocation of k are concatanated back together for the final value. This means that >>= can be used as an alternative to filter, map, or be used for local transformations of the list, such as repeating each element.

As a filter

Often you would like to filter a list based on a predicate. This can be done with >>= as below.

filtered = list >>= \x -> if predicate x then [x] else []

This is equivalent to the below use of filter.

filtered = filter predicate list

As a map

>>= can also be used to map a function over the elements of a list, i.e. end up with a list of the same length, but each element has been passed through another function. An example of this is below.

mapped = list >>= return . mapper

This is equivalent to just using map.

mapped = map mapper list

As a local transform

The cases where >>>= shines are those that can't be achieved using a single higher-order function. For example, repeating each element twice in a list.

repeated = list >>= \x -> [x, x]

Should I use `>>=`?

My instinct is to use the least general code that solves the problem you're tackling, and solves it clearly. So while using >>= for when there isn't a single other function that can do the job is good, using it when map or filter would do, while interesting, I would probably avoid.

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.
Ian Malcom, Jurassic Park

Hidden structure

2017-05-22T10:00:00Z

Trying to understand functors, I came accross the initially strange fact that functions are functors. Which means, roughly speaking, you can "map", another function over it. So given a function

g :: r -> a

and another function

f :: a -> b

you can "map" the second function over the first to get another function, which is defined as the composition of the two.

fmap f g :: r -> b
fmap f g = f . g

Similarly, and more familiarly, you can take a list

xs :: [a]

and a function

f :: a -> b

and map the function over the list to get another list.

fmap f xs :: [b]
fmap f xs = [f x | x <- xs]

(I suspect this isn't how fmap is defined on a list in the real source, but I find list comprehensions quite easy to understand.)

The suprising thing is that you can express two seemingly very different activities, mapping a function over a list, and composing a function with another function, in Haskell using the same function: fmap. Here "the same" is defined as having the same type signature.

fmap :: Functor f => (a -> b) -> f a -> f b

Although yes, the implementation of fmap for functions and lists are indeed different.

The beauty and the horror of Haskell monads

2017-05-06T10:00:00Z

This post assumes some understanding of and familiarity with Haskell monads.

The beauty

There are a number of patterns that come up frequently in programming. Some of these are:

Combining IO actions, making the result of each available to later ones: IO monad
Building up, or reading from, a peusdo-mutable state or configuration: Reader, Writer, and State monads
Running a sequence of functions where the later are skipped if the previous doesn't return a meaningful result: Maybe monad
Running a function over all possible combined values of a number of lists: List monad

The beautiful thing is that problems that at face value are completely different actually share a common structure. They can be decomposed into their domain-specific components, and >>= (bind) and return, where, although different for all of the above examples, adhere to certain laws. Once you are familiar with any given instance of >>= and return, then it's really quite amazing how much can be achieved, and communicated, with just a few lines of code.

The horror

The "once you are familiar with any given instance of >>=" hints at the horror. The >>= and return functions are (terribly?) overloaded. Each of the monads use the same notation, but they can all do radically different things. To know what any >>= actually does, you have to infer its type from the surrounding code, and look to its definition. Similarly a do-block's behaviour depends radially on the definition of >>=: and it might be even worse, because >>= isn't even present in the code.

Taking and modifying some examples from What I Wish I Knew When Learning Haskell, we can see a use of the Maybe monad,

main :: IO ()
main = putStrLn . show $ example

example :: Maybe Int
example = do
  a <- Just 3
  b <- Nothing
  return $ a + b

and an example of the Reader monad.

import Control.Monad.Reader

data MyContext = MyContext
  { foo :: String
  , bar :: Int
  } deriving (Show)

main :: IO ()
main = putStrLn . show $ runReader computation $ MyContext "hello" 1

computation :: Reader MyContext (Maybe String)
computation = do
  n <- asks bar
  x <- asks foo
  if n > 0
    then return (Just x)
    else return Nothing

The monadic values example and compution definitions have similarities: do notation, <-, and return. However, these similarities are misleading. Take for example return from the Maybe monad. Its definition is

return = Just

while the Reader monad's definition is

return a = Reader $ \_ -> a

Similarly, because of the different implementations of >>=, line breaks in the do-notations result in different code. Hence how monads are sometimes described as programmable semicolons.

There is a lot of non-explicit behaviour going on. In code reviews, I often use the term "magical" if there is non-explicit behaviour, and I usually suggest it is to be avoided. Yes, >>= and return are likely to abide by the monad laws, and they certainly do in these examples, but I'm unsure if this is enough to understand what's going on.

As a parallel, Mike Acton, in his talk on data-oriented design, suggests only using C++ operator overloading that is "super obvious", such as adding vectors. Uncle Bob also teaches us to be more explicit.

The name of a variable, function, or class, should answer all the big questions. It should tell you why it exists, what it does, and how it is used.
Robert C. Martin

There are reasonable arguments that monads in Haskell, by overloading >>= and return, break these guidlines. Of course, perhaps I will feel differently once I have been working with Haskell for longer. We will see!

Haskell do notation protects you from parentheses

2017-04-21T10:00:00Z

Starting in Haskell, I wanted to lean away from do notation, to make sure I knew what was going on under the hood before taking syntactic shortcuts.

However, I have found a small stumping block in my quest for de-sugaring. Function application $ and nested do notation results in quite clear imperative-style code. For example, E2E tests using Test.Hspec.Webdriver.

main :: IO ()
main = hspec $
  describe "E2E smoke test" $ do

    sessionWith config "integration" $ using allBrowsers $ do
      it "checks all text in p" $ runWD $ do
        openPage "http://web_test/test/files/index.html"
        e <- findElem $ ByCSS "p"
        e `shouldHaveText` "Some HTML"

      it "checks all text in p strong" $ runWD $ do
        openPage "http://web_test/test/files/index.html"
        e <- findElem $ ByCSS "p strong"
        e `shouldHaveText` "HTML"

Desugaring isn't just a case of removing do and changing the line breaks to >> or v <- expression to expression >>= \v ->. It turns out that

if there were nested do blocks, and multiple lines in the outer block, then you need parentheses around what was a line in the outer one;
if there were usages of $, then the expression to the right of $ will need to be wrapped in parentheses, and $ removed.

This is due to the facts that

the end of a do block, and each newline in a do block, essentially has a lower precendence than $, which in turn has a lower precedence than >>;
explicit lambdas, which would come from desugaring <-, take everything from -> to the end of the expression as their body.

To explain: when in a do block, the second argument of $ is from the $ to the end of the line. However, when not in a do block, the second argument of $ goes beyond the end of the line, which is ignored, beyond any >> and >>=, all the way to the end of the containing expression. Similar logic applies for the body of lambdas.

This means that the above E2E tests, but without do notation, look like the following.

main :: IO ()
main = hspec $
  describe "E2E smoke test" $

    sessionWith config "integration" $ using allBrowsers $
      (it "checks all text in p" $ runWD
        (openPage "http://web_test/test/files/index.html" >>
        findElem (ByCSS "p") >>= \e ->
        e `shouldHaveText` "Some HTML")) >>

      (it "checks all text in p strong" $ runWD
        (openPage "http://web_test/test/files/index.html" >>
        findElem (ByCSS "p strong") >>= \e ->
        e `shouldHaveText` "HTML"))

Not the end of the world of course, but certainly a point in the do-notation column for clarity, and certainly conciseness.

Long files are not the enemy

2017-04-02T10:00:00Z

I have seen multiple developers leap to the conclusion that long files of source code are automatically bad, leading them to separate the code, virtually as-is, into separate files. This post suggests that this view is too simplistic, and other things should be considered before moving code about.

So why do developers think long files are bad?

My suspicion is that upon seeing a long file, there is often an assumption that it's a moving part. Any working with it means you have to reason about it as a whole, this whole is too big to reason about, and therefore you should split it up to make it possible to reason about each of them separately. So firstly...

A file is not necessarily a moving part

I don't have a strict definition of what a code moving part is, but roughly it's something that you have to reason about as a whole, and that interacts with other code or the outside world via some interface. Examples include

pure function,
group of functions that call each other or communicate via some state,
script that is run in response to a web request,
class, especially if its instances maintain an internal state,
Angular directive or a React component.

You can have one of these in a file, multiple in a file, or even one spread across multiple files.

If there's a problem, you're just moving it about...

Consider a 1000 line file that has 40 functions in it. These functions might not call each other, read or change state, and they don't have any side effects beyond those they advertise, such as touching a database. Each of these functions I would class as a moving part, and each of these function might have problems. In another file, these problems would be exactly the same.

... and maybe making it even worse

Consider if those 40 functions each change and read state, call each other in a complex web of booleans and optional arguments, none of them having a clearly separate purpose, and the behaviour of any part of the code is very brittle with respect to changes in other parts. This makes the 40 functions into a single moving part, and splitting these functions, as they are, into separate files actually spreads this moving part about. I suspect it is likely that this makes it harder, and not easier, to reason about.

... and makes it harder to see its history

Once you have moved code about, it makes finding the history of it just that little bit harder. If there are problems with the code, the history of the lines may help work out why things are the way they are, and so guide you in the choices you make when making further changes. It's actually with the worst code that the version control history is the most valuable.

So what is the enemy?

Smells/rot in the moving parts themselves. Maybe each moving part has too much responsibility, blurry responsibility, or acheives its goal(s) using too much state or crazy overcomplicated logic. Any one of the tens/hundreds of possible problems code might have that aren't that it happens to be near another moving part in the same file.

What should I do about it?

Refactor as you would, but keep in mind that moving code as-is, especially bad code, has consquences that might not all be positive.

Final semi-stolen quote

To steal/adapt Sean Kelly's point on microservices...

You don’t need to introduce another file as an excuse to write better code.

Avoid stateful variables

2017-02-12T10:00:00Z

Coming from an imperative programming background, or working in an imperative code base, it can be tricky to write purer code. However, there is one guideline I try to follow that gently prods me in the "right" direction.

Avoid stateful variables

Following this guideline leads to a number of things that I think makes the code "better".

Stateful variables

What I mean by stateful variable is a variable whose value changes once set. This include simple re-assignment,

a = f()
a = g(a)

but it also includes calls that change the internal state of an object after it has been created.

a = MyClass()
a.setFoo('bar')

Both of these mean there is a state that changes over time in the code. It might be short-lived, just in a single function, but it can still make the code harder to reason about. There is no, easily determined, single expression that can replace a stateful variable wherever it is used. You have to think about the entire history of the variable, and what caused it to change. Using a more technical term, stateful variables cannot be used in a referentially transparent way.

Once I have identified a stateful variable in code I am about to write or change, I consider alternatives.

Alternative 1: Declare more stateless variables

A way to avoid stateful variables is to declare more stateless variables. These are variables that don't change once set. For example,

a = f()
a = g(a)

could become as below.

a = f()
b = g(a)

This might be a bit more complex if you have branching involved.

a = f()
if test():
  a = g(a)

This above could become as below.

a = f()
b = g(a) if test() else a

Or, if you can change g, it might make sense for it to handle a boolean,

a = f()
b = g(a, test())

or even for it to call a function that returns a boolean.

a = f()
b = g(a, test)

Alternative 2: Declare fewer variables by inlining

A simple technique is just to not use as many variables. The code

a = f()
a = g(a)

could just be as below.

a = g(f())

A more horrible example is if you have a stateful variable that depends on another stateful variable.

condition = condition_1()
if condition and condition_2():
  condition = condition_3() 
a = f()
if condition:
  a = g(a)

This abomination is equivalent to the below, assuming all functions have no side-effects.

if (condition_1() and not condition_2()) or (condition_1() and condition_2() and condition_3())
  a = g(f())
else:
  a = f()

Yes, there is a little bit of duplication. but I wouldn't fear this, especially for such a tiny amount of duplication. It's case-dependent, but it may well be worth seeing exactly the cases certain code is run, and this might mean a little bit of duplication.

duplication is far cheaper than the wrong abstraction
Sandi Metz

Alternative 3: Create objects as needed rather than changing them

a = MyClass()
if test():
  a.setFoo('mary')

a = MyClass(foo='mary' if test() else None)

This code might not be strictly equivalent, but I would put in the effort to make sure MyClass works with foo being None.

As a sidebar, the None case above makes foo look look it might be an optional argument, which isn't great, for the reasons outlined in the post about optional arguments. Ideally, you would be able to factor out the code from MyClass so that it doesn't need to treat None as a special ase.

Alternative 4: Use higher order functions

In some cases you have a "pipeline" where a value is repeatedly modified for various reasons. For example a list of filters that are each applied conditionally.

a = f()

if condition_1():
  a = filter_1(a)

if condition_2():
  a = filter_3(a)

if condition_3():
  a = filter_3(a)

if condition_4():
  a = filter_4(a)

filtered = a

In this case, you can construct a structure of functions, and run them using a reduce.

a = f()

def conditional_filter(condition_filter, value):
  return condition_filter['filter'](value) if condition_filter['condition']() else value

condition_filters = [
  {'condition': condition_1, 'filter': filter_1, },
  {'condition': condition_2, 'filter': filter_2, },
  {'condition': condition_3, 'filter': filter_3, },
  {'condition': condition_4, 'filter': filter_4, },
]

filtered = reduce(conditional_filter, condition_filters, a)

Going overboard

I don't follow the above alternatives in all cases. There are times when making stateless code would be less clear, would be too much of a project, or would make the code too rigid.

For example, in the case of Alternative 4 above, there are reasons why this final pure code is not better than the original. It's remove some state, and added the ability that it's really easy to make the list of conditions and filters dynamic, but at the cost of making it harder to do something different/hacky in the middle of the filtering process. Say, save a file or make an API request. This would of course make the code slightly horrible, but in a real-world situation with time constraints and other priorities, this might be the best option.

Also, you should consider who else will be working on this code, and their familiarity with working with higher order functions like reduce. Of course, this should be covered in programming 101, but for me at least, it really wasn't.

In cases where a stateful variable is to be used, I usually try to make sure its lifetime is as limited as possible, make it so it is modified in consitent ways, and it is modified for consistent reasons.

As everything, it's a trade-off.

Non atomic deployments

2017-02-06T10:00:00Z

When designing a deployment strategy for your web application or site, keep in mind that browsers do not request a site atomically. They ask for the HTML, and then, some time later, ask for the resources such as scripts and stylesheets. If a deployment is fully atomic, as in only one version of a site is accessible at any given time, this time period can introduce a race condition:

A build/deployment is started.
A browser requests a HTML page, and the server supplies it.
A deployment is completed. Old versions of static resources such as stylesheets and scripts are no longer accessible.
The browser requests the static resources from the original HTML. The browser either gets 404s or scripts or stylesheets that don't match the CSS, resulting in broken site.

There are a number of ways of avoiding this.

Inlining everything in the HTML. Not a crazy idea for extremely small sites, but once you get fairly heavyweight scripts or images, this makes time to view the page quite high.
Use versioned or hashed file names for resources, and don't make versions from previous deployments inaccessible until some time after the deployment.

The second way is the method used for this site, with file names with the hash of the file contents in to avoid having to store version numbers anywhere. This was originally setup for cache busting, but it has this wonderful benefit of supporting multiple versions of assets at the same time.

Cleaning up

A problem is eventually cleaning up the old versions of files. However, because online storage is so cheap, it can often be a viable option to just never clean up the obsolete resources. If you're spending an extra £O.O1 per month, my suspicion is that as long as the files don't contain anything sensitive or have a restrictive license, it's just not worth the time to setup, and maintain, anything to do it. An argument against this however, is that lots of dead files can make it harder to debug issues.

Also up until recently, I didn't know/couldn't think of a way to delay deletion of such files without too much infrastructure, such as a server running cron, Jenkins, or scheduled Lambda function. This would have to be tied to the build system so it knows what to delete. This can all get a bit complex, especially when there must be some mechanism to abort the delete in case you go back to a previous version with the same hashed filename. All a bit much to setup / maintain / monitor just to delete a few files.

However, if you serve static assets from an S3 bucket, there are two features that make this really easy that require very little additional infrastructure.

Object tagging. You can individually tag each object with a user-defined tag.
Lifecycle management. You can setup rules that delete all object that have a tag, some specified time after they were last modified.

Using these, you can do the following.

Setup a lifecycle configuration rule that deletes objects in the bucket that have a certain tag, a certain amount of time after they were last modified.
Before each the deployment, fetch a list of all current objects in the bucket.
When deploying the static resources, make sure each has a versioned / hashed file name, and make sure to clear any tag that would have resulted in its deletion.
For all existing resources that weren't overridden and that didn't already have the tag, copy the resources to themselves, adding the tag. The copy, while a bit hacky, is the only way I can see of updating the last modified time. This avoids objects being deleted sooner than the minimum time after they become unreferenced that you feel is appropriate.

Following the above strategy means that once a resource is obsolete, it will remain in the bucket until some time after the deployment, when S3 will automatically delete it. If you update the site again, and the same resource becomes un-obsolete and re-published, removal of the tag means S3 won't delete it.

That's it! A low-infrastructure mechanism of cleaning up after non atomic deployments.

The best infrastructure is the one that doesn't exist
Jeff Nunn

Overriding methods is spaghetti in disguise

2017-01-09T10:00:00Z

A common pattern I have seen is having some default behaviour in a base class, that is sometimes overwritten in subclasses, and sometimes not.

class Base(object):
  def colour(self):
    return "red"

class ClassA(Base):
  pass

class ClassB(Base):
  def colour(self):
    return "green"

a = ClassA()
print a.colour()  #  "red"

b = ClassB()
print b.colour()  #  "green"

The usual reason for this is that the Base case was written first, and then new behaviour was needed that didn't quite match up with the original.

Why not do this?

The simple case above isn't too bad since it's, well, simple. A more complex case however, is when you have more subclasses, more methods calling each other, and each subclass overriding a different subset of the base methods.

class Base(object):
  def colour(self):
    return "red"

  def size(self):
    return "medium"

class ClassA(Base):
  pass

class ClassB(Base):
  def colour(self):
    return "green"

class ClassC(Base):
  def size(self):
    return "small"

a = ClassA()
print a.colour()  #  "red"
print a.size()    #  "medium"

b = ClassB()
print b.colour()  #  "green"
print b.size()    #  "medium"

c = ClassC()
print c.colour()  #  "red"
print c.size()    #  "small"

Or, a similar pattern, but utilizing the fact that all methods in Python are effectively virtual.

class Base(object):
  def colour(self):
    return self._colour()

  def _colour(self):
    return "red"

class ClassA(Base):
  pass

class ClassB(Base):
  def colour(self):
    return "green"

class ClassC(Base):
  def _colour(self):
    return "blue"

a = ClassA()
print a.colour()  #  "red"

b = ClassB()
print b.colour()  #  "green"

c = ClassC()
print c.colour()  #  "blue"

At first glance, the code doesn't look like traditional spaghetti: there are no if statements, no booleans being passed around, no optional arguments, and no global variables. It all looks quite clean.

In spite of these, I judge this code to have problems: hidden inconsistency, hidden coupling, historical bias, poor separation of responsibilities, and poor abstraction.

Hidden inconsistency

When calling any method on an instance of the subclasses, some objects will call Base class code, and some not, depending on what is defined in the subclass. Each subclass now has unique properties and a unique data flow that must be reasoned about. This might be unavoidable, each case must in some way be different, but my suspicion is that that having this inconsistency "hidden" in the inheritance heirachy tree by nature of some methods being overridden and some not, means more time will need to be spent later when adding/changing behaviour.

Hidden coupling

Perhaps a problem with code re-use via inheriance in general, it's difficult to change code in the base class without affecting the subclasses. This is however especially true when each subclass uses a different subset of the base class methods. The ones that do use a Base methods are actually the ones that don't override them. Hence it's more accurate to term this hidden coupling.

Historical bias

The code in the base class is most likely only there because it was written first. Very obvious and reasonable when you're writing it, but come to look at the code in 2 years, it's not immediately clear why the code is the way it is. Trying to untangle spaghetti is tricky, often trying to work out the reasons the original programmer wrote what they wrote in order to reduce the risk when changing it. Having the extra reason of "that's what I did first" is making the future's job harder.

Poor separation of reponsibilities

The code in the base class controls certain aspects of the application. Except in the cases where this is overridden. This suggests that responsibility has not been separated well.

Poor abstraction

Although I think it's often unhelpful to think in such terms, overriding Base methods results in the statement that an object "is a" Base becoming a bit meaningless, and therefore unhelpful. The "stuff the object does", as defined in the base class code, only holds for some of the instances of the subclasses.

Should I put in an intermediate class?

It's tempting to try to model the problem using more "is a" relationships and break the tree down further to factor out common behaviour. My suspicion is that you're just making things less clear and harder to change later, because you have increased the number of layers in the system and the contracts between them.

So what are the alternatives?

They depend on the code in question, but they all involve moving the code out from the base class, so that each subclass explicitly states what it uses. For example the code could just be in pure functions not associated with any particular class...

def red():
  ...

def blue():
  ...

def medium():
  ...

class ClassA(object):
  def colour(self):
    return red()

  def size(self):
    return medium()

class ClassB(object):
  def colour(self):
    return blue()

  def size(self):
    return medium()

... or if they do need to be part of a class, they could be composed at import time...

def red():
  ...

def blue():
  ...

def medium():
  ...

class Base(object):
  def colour(self):
    return self._colour()

  def size(self):
    return self._size()

class ClassA(Base):
  _color = red
  _size = medium

class ClassB(Base):
  _color = green
  _size = medium

... or without any inheritance involved...

def red():
  ...

def blue():
  ...

def medium():
  ...

class ClassA(object):
  red = red
  medium = medium

  def colour(self):
    return self.red()

  def size(self):
    return self.medium()

class ClassB(object):
  blue = blue
  medium = medium

  def colour(self):
    return self.blue

  def size(self):
    return self.medium

... or even with just a single class that is wired up at runtime.

class Class(object):
  def __init__(self, colour, size):
    self._colour = colour
    self._size = size

  def colour(self):
    return self._colour()

  def size(self):
    return self._size()

def red():
  ...

def blue():
  ...

a = Class(colour=red, size=blue)

The examples above all "wire up" functions to the classes/instances. However, you can use objects if you want or need to.

What about type safety?

One thing that some of the alternatives above have lost is type information, even though they deliberately share the same public API.

In Python, you might not need to want to worry this too much. However, you might want/need a stricter type system, for example for

type hints to check before runtime, say using mypy;
checking types at runtime using isinstance or similar;
run time protection against forgetting to implement a required method.

If so, you have an option: deriving from one or multiple abstract base classes.

from abc import ABCMeta, abstractmethod

class Colour(object):
  __metaclass__ = ABCMeta

  @abstractmethod
  def colour(self):
    pass

class Size(object):
  __metaclass__ = ABCMeta

  @abstractmethod
  def size(self):
    pass

class Class(Colour, Size):
  # Any of the options above

C++ has something similar: you can define a class with only pure virtual functions.

class Colour {
public:
  virtual void colour() = 0;
};

class Size {
public:
  virtual void size() = 0;
};

class Class : public Colour, public Size {
public:
  void colour() {
    ...
  }
  void size() {
    ...
  }
}

Java, although often encouraging of code reuse using inheritance, has quite nicely got pure abstract base classes built right into the language as interfaces.

interface Colour {
  public void colour();
}

interface Size {
  public void size();
}

public class Class implements Colour, Size {
  public void colour() {
    ...
  }
  public void size() {
    ...
  }
}

That's it!

E2E Tests: Test as a user would

2016-12-25T10:00:00Z

This post contains a few recommendations on how to write certain aspects Protractor E2E tests. Specifically how to depend less on the internals of the page. You can't entirely not depend on internals: they all depend some amount of HTML, and currently you have to choose and interact with the elements of the page somehow. However, some internals are better than others.

The point of E2E tests

First a quick recap on a/the purpose of E2E tests. They ensure that aspects of the application work as expected even if there are later changes: they protect against regressions as you're developing the application. Essentially they should fail if some aspect of a user-facing feature changes in an undesirable way.

However, they should not fail if some internal of the pages changes that doesn't break a user-facing feature. This doesn't necessarily directly affect the primary purpose of the tests: they can still protect against regressions. However, failing E2E tests when there is no breakage of a user-facing feature can be a horrible time-sink. With a bit of thought, the risk of this can me minimised, or at least reduced.¹

To this end, there are a few features of Protractor that I think should not be used.

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.
Ian Malcom, Jurassic Park

Don't depend on ng-repeat or ng-model: instead, select elements as a user would

The Protractor documentation suggests selecting HTML elements by ng-repeat or ng-model attribute contents. This means if someone comes along and does a bit of refactoring, or uses something other than ng-repeat or ng-model, the E2E tests will fail, even though the behaviour, from the users' points of view, is perfectly fine. I treat this similarly to the test depending on a variable name or process on the server. It seems quite silly for the test to be fragile with respect to that!

Instead, consider using minimal CSS or XPath selectors to pick elements as close as possible to how a user would, that would survive internal refactorings. For example, the first table after a h2 element with text content "Section 1".

table = element(by.xpath('//h2[text()="Section 1"]/following::table[1]'))

Or, if you have used a custom element for your UI widget directives, the first <my-widget></my-widget> in the first <nav></nav> of the page.

widget = element(by.xpath('//nav[1]//my-widget[1]'))

Consider how specific you want the selectors to be. If you have only one my-widget in the page, you can be more minimal in the selector.

widget = element(by.xpath('//my-widget'))

This makes the test less fragile to moving the element around the page. If it's crucial to test where this is in the page, you could always have an explicit assertion for that, perhaps in a separate test.

Another technique is to choose elements using ARIA roles.

dialog = element(by.css('[role="dialog"]'))

This has the handy benefit of gently pushing you to add accessiblity attributes.

Another technique is selecting elements using cssContainingText, say to find button that also has the text "Go to stage 2".

button = element(by.cssContainingText('button', 'Go to stage 2'))

And as a last, but perhaps still common, resort, using a plain class.

message = element(by.css('.message'))

If using classes I recommend considering that they are as semantic as possible. This isn't just to make the HTML "nice": it's so that the tests continue to pass as long as the element behaves as it should, independent of any changes or internal refactoring or renaming of non-sematic (with respect to the UI) elements on the page.

I don't think it's a perfect solution: you might want to tweak the semantic names for custom elements and classes, and in such cases the tests would still fail, but I suspect it's better for a lot of cases that depend on ng-repeat or ng-model.

Don't wait for Angular: instead, wait for the interface as a user would

If you're using Protractor, consider setting set browser.ignoreSynchronization = true for all tests. Yes, really. The magic Protractor gives you for the simple cases just isn't worth it, because the cases of having some sort of polling, or non-Angular asynchronous code running, just makes you have multiple sorts of tests.

Instead, wait for changes in the page elements to appear on the screen. You can do this explicitly in your first few tests, but then you might want to factor the waiting out once you are clear what you're waiting for and commonality between the cases is clear. Your tests will be less flakey to changes you might make in the application. Even, if you eventually decide to swap out Angular!

browser.ignoreSynchronization = true;
browser.wait(function () {
   return element(by.css('[role="dialog"]')).isPresent()
});

Of course, if you're not using what Protractor builds on top of Selenium, then I would seriously consider just using plain Selenium, or a different framework like WebdriverIO.

¹ I realise I don't have quantitative data to back this claim up.

Reasons to not use page objects in E2E tests

2016-12-18T10:00:00Z

Page objects seem to have gained a bit of traction in AngularJS E2E testing, especially since they seem to be officially recommended. In this post I offer a few reasons not to use them.

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.
Ian Malcom, Jurassic Park

Reason 1: They obscure behaviour

Consider the official example below.

var AngularHomepage = function() {
  var nameInput = element(by.model('yourName'));
  var greeting = element(by.binding('yourName'));

  this.get = function() {
    browser.get('http://www.angularjs.org');
  };

  this.setName = function(name) {
    nameInput.sendKeys(name);
  };

  this.getGreeting = function() {
    return greeting.getText();
  };
};

Because of the behaviour of ElementFinder, the elements are actually only retrieved when you call setName or getGreeting. Of course you know this, since you know how Protractor behaves, and can see the code of the page object. But you could have this information much closer to the test rather than elsewhere. When you have more complex page behaviour, with elements appearing and disappearing due to non-trivial behaviour, in order to keep the tests maintainable, it is important to make sure that they are doing what you think they're doing.

I think the behaviour of ElementFinder that attempts to retrieve the element from the page when the element is first interacted with is good for simple cases, but slightly too magical for testing more complex UIs. Hiding this behind your own layer makes this situation even worse.

Reason 2: They obscure state

By nature of being objects, they promote keeping an internal state, exposed by an interface. You might not be able to get away from having some state in your tests: they last for some "real" period of time. However, you should be able to keep it all in the main body of the test, keeping only the state you need for that particular test.

Consider the example above. As mentioned, the elements are actually found on the page when you call setName or getGreeting, but you are offered no way to "refind" an element, say if the application removes them and new ones are added to the DOM. Of course, you can create a new instance of that page object, or add a method to do it,

this.refindNameInput = function() {
  nameInput = element(by.model('yourName'))
};

but doing this means that you are completely leaking how the page object handles state internally. Encapsulation has offered little but another layer of code between the test and what it's doing.

Reason 3: They obscure the interactions with the UI

The page object example in the documentation suggests wrapping the non-semantic UI actions such as typing in an input and getting text from an element...

nameInput.sendKeys(name);
...
greeting.getText();

... with functions with semantic names.

this.setName = function(name) {
  nameInput.sendKeys(name);
};

this.getGreeting = function() {
  return greeting.getText();
};

The main problems is that it's needlessly hiding how the object is interacting with the elements on the page and where the object gets its data. This means it's less clear what the test is testing.

As the application grows, and you have more elements and ways of interacting with them, combinations of key presses, mouse movements, and different and more complex ways of giving output, this can lead to your page objects exposing a lot of what you meant to hide in the first place.

Reason 4: You are spending time coding for a case of many E2E tests that should never happen

It is usually recommended to not have that many E2E tests. If you have a very small amount of E2E tests for each part of your application, then spending time for some future "what if we do this 100s of times" cases is effort expended that has the primary effect of making it harder to reason about each test that you do have.

Reason 5: They might not be the right abstraction for your cases

A "page", at least by name, may not be the right abstraction for finding or interacting with elements in an application. You might have a widget to test in a left menu panel, but it can also appear on the right, accessible from a different selector. Or it might appear at different times. It might also behave slightly differently depending on where it is or when it appears. Or you might even have different browser instances in play, say to test a login/registration flow, and need to control different ones, having them all in-play at once.

Just like optimization, premature refactoring, also known as speculative generalization, can be bad. Instead, I suggest doing continous/opportunistic refactoring. Until you have 3 tests or more, you might not really be able to see what is common to them, and you might get yourself stuck in multiple layers of asynchronous test code and related page objects that aren't particularly suited to your case.

Keep in mind: there might not even be a "right" abstraction for all your tests, and that's ok!

Of course, not planning up front has its criticisms.

I find that weeks of coding and testing can save me hours of planning.
Someone

However, I think using page objects just to have encapsulation and because others do it doesn't quite meet the definition of what I think is planning. At least good planning!

What to do instead of page objects?

Always keep in mind what you're trying to do in the tests.

Finding element(s)
Interacting with element(s)
Asserting something about element(s)

If you have a few tests of a repeated non-trivial group of actions, like a login, or maybe a drag and drop action between one part of a page and another, then factor them out to functions, not objects, that group them. I would really consider not factoring out much more than that.

For example, consider the following test.

describe('my homepage', function() {
  it('should greet the named user', function() {
    browser.get('http://mypage.com');
    input = element(by.xpath('//label[text()="Name"]/following::input[1]'))
    input.sendKeys('Julie');
    greeting = element(by.xpath('//label[text()="Greeting"]/following::p[1]'))
    expect(greeting.getText()).toEqual('Hello Julie!');
  });
});

You can see, right here in these few lines of code, what the test is doing: how the elements are being found and how Protractor is interacting with them. I think each test of your suite should be this clear. This does mean they will be some repeat calls to ElementFinder functions, or maybe some repeated locators, but so be it!

This doesn't sound very DRY... what if/when I need to change something?

Consider the reasons for changing/adding test code.

Changing something trivial in the UI, like text or where an element is on a page
Changing something non-trivial in the UI, like how a non-trivial group of widgets work together
Adding a new test
Fixing an existing test, for example if it is flaky/occasionally fails

If your main concern is the first case, and you're worrying that you might have to do a bit of a search-replace in your tests, then my suspicion is that you're focusing on the wrong thing. At most it's a few minutes of something a bit boring. I think it's much more important to make the tests easier to deal with if your doing something less-trivial, which are the other 3 cases. In those cases, clarity of what the tests are doing is paramount.

Say no to more optional arguments

2016-12-10T16:00:00Z

Often you want to pass in an extra variable to an existing function

def my_func(a):
  ...

so it supports a new use. You usually want the old use to remain the same, so you add a optional argument with a default value. Often, in Python at least, this default is None.

def my_func(a, b=None):
  ...

So after adding this, you would have two (or more) call sites.

# Original call site
my_func(a=arg_original)

# New call site
my_func(a=arg_new_1, b=arg_new_2)

and quite likely inside the function there will be at least one new block of code that modifies its behaviour in the new case.

def my_func(a, b=None):
  ...
  if b is not None:
    ...

This is a run-time check, even though you know, at write-time, that the case you want this to run is in the new call site. Having write-time knowledge that you convert to a run-time check is often needlessly increasing number of branches in the code¹. I suspect it is a response to personal bias against touching existing code, and it biases the code itself: the original use case is clearer than than the new one. All this, I suspect, is likely to make further changes to the code more difficult.

One small improvement is to not have a default value in the parameter list, so the function looks like

def my_func(a, b):
  ...
  if b is not None:
    ...

and change the original call site to explicitly pass None.

# Original call site
my_func(a=arg_original, b=None)

However, exactly because you can pass None, and the function handles this case separately, b is still a de-facto optional argument to the function.

What to do about this depends on the actual code in question. However, often there is code in the function that can be factored out to functions called from outside of the function in question, ideally each returning meaningful values.

# Original call site
my_func(a=arg_original, b=foo())

# New call site
my_func(a=arg_new, b=bar())

so the code of my_func no longer has a special case of handling b being None.

In some cases, if you're working in a language that allows lambda functions, it might be clearer to extract out a lambda function that you pass in.

# Original call site
my_func(a=arg_original, b=lambda x: foo(x))

# New call site
my_func(a=arg_new, b=lambda x: bar(x))

And of course, if you're working in an object oriented language, it might be appropriate to go the whole hog and pass in an object.

# Original call site
my_func(a=arg_original, b=new Foo())

# New call site
my_func(a=arg_new, b=new Bar())

These changes have benefits that they keeps the reponsibility of the original function small, chances are it has limited the number of branches, and it does not bias the code base: the new call site is as clear and deliberate as the original call site.

¹ This statement only has relevance if number of branches is seen as something to be deliberatly avoided, a view which may be oversimplisitc, and in some cases, counter-productive. I am wary of following it too strictly.

Pragmatic Purity

2016-12-09T16:00:00Z

The code below follows a common pattern in impure code.

if (condition) {
  doFooImpurely();
} else {
  doBarImpurely();
}

If you can extract out common logic from doFooImpurely and doBarImpurely into a meaningful intermediate variable that can be calculated purely, so the code is of the following form

const intermediateValue = condition ? getFooValuePurely() : getBarValuePurely();
doImpurely(intermediateValue);

then I would recommend considering it. Even if the total cycomatic complexity of the code is equal to what it was, I usually judge the latter to be better because

the pure and impure parts of the code are separate, and so better due to the standard separation of concerns arguments;
depending on what is inside doImpurely, there are likely to be fewer impure code paths: this is often better since there are fewer ways state can change or side effects can happen.

The benefit is greater in more complex cases.

if (condition) {
  doFooImpurely();
} else if (anotherCondition) {
  doBarImpurely();
} else {
  doQuxImpurely();
}

If you can extract out the pure logic, the code can then take the below form.

const intermediateValue = 
   condition        ? getFooValuePurely() :
  (anotherCondition ? getQuxValuePurely() : getBarValuePurely());
doImpurely(intermediateValue);

If you're not a fan of the nested ternary operator, you can easily restructure the pure part of the code: make it a series of if statements, extract out to another function, etc. A really nice benefit of pure code is that you can really easily move code about, and have confidence that it doesn't negatively affect the code that "does stuff", i.e. the impure code.

Blue-Green Deployment of Static Site on S3

2016-10-30T16:00:00Z

This post contains the high level steps for the site architecture and deployment strategy for blue-green deployment for a static site hosted on S3.

There are a few things that are required for a reasonable blue-green strategy.

No downtime between releases.
Atomic deployment. A visitor sees a whole working version from before or after the deployment. This can be tricky because a user doesn't request a site atomically, i.e. first the HTML is requested, and then some time later all the static resources.
E2E tests run on a certification environment that has the bare minimum of changes between it and production.

To acheive this, you can place the entire site behind an API Gateway. Although the documentation and marketing material hints that it is only suitable for APIs to be consumed programatically, there is nothing stopping it being used for an API to be consumed by people, i.e. a site made up of HTML (and indirectly, CSS, JavaScript, images and fonts). A few features of API Gateway make this possible.

API gateway has the concept of stages, where you deploy and API to a stage, and then, later, atomically update another stage to that deployment.
API gateway can proxy requests to S3, so the entire HTML of the site can placed behind the API gateway.
API gateway supports custom domains and the ability to map paths to specific stages.

With these features in mind, the overal architecture of the site can be as follows.

You have all the HTML pages of the site defined as an API, with appropriate resources defined. A Swagger definition file is a reasonable choice for this.
2 stages setup for the API, each with a custom domain. The public domain of the site has an API mapping from the root path to the production stage; and another domain, which is used for certification, has an API mapping from the root path to the certification stage.
A single S3 bucket for the assets, i.e. everything except the HTML. These must have versioned file names, such as having a fragment of the md5 hash of the contents. This S3 bucket can be behind a Cloudfront distribution if you would like.
Two S3 buckets for the html. One for production and one for certification.

A strategy to deploy using this is below.

Deploy static assets to the asset bucket, making sure that each is has a file name that is versioned with an MD5 of its contents, and existing assets are not deleted.
Determine which bucket is in production. One choice is to actually embed this as a mock route in the Swagger definition file, and download this in the build script, so you have an up to date file.
Upload all HTML to the bucket that isn't in production: the certification bucket.
Modify the Swagger definition file so requests for HTML are proxied to the certification bucket. You could use your favourite template language for this.
Deploy the API, as defined by the modified Swagger definition file, to the certification stage. Take note of the deploymentId.
Run E2E tests.
If they all pass, update the production stage so its deploymentId is set to the certification stage.

At every stage above, a visitor to the public site will fetch either the pre-release HTML, or the post-release HTML. In both cases, all the assets will load correctly.

Note: To keep this post brief, not part of this strategy is removal of assets that are no longer used.

That's it!

Defer node stream processing

2015-12-31T10:00:00Z

There might be times when you have a chain of piped streams where you want to delay a part of the chain until a promise is resolved. You can do this with a function that returns a simple Transform stream.

function waitFor(promise) {
  return stream.Transform({
    objectMode: true,
    transform: function(file, enc, cb) {
      var self = this;
      promise.then(function() {
        self.push(file);
        cb();
      }, function(err) {
        cb(new Error(err));
      });
    }
  });
}

You would need to pipe to and from the return value of the above function, and it will effectively delay the downstream piping until the passed promise is resolved. This can be useful in a gulp task where you don't want to split it up into sub-tasks, but still want one part of a stream to be processed after another.

As a example, taking part of the gulpfile for this blog, I defer publishing html files until images and stylesheets have been uploaded, so any visitors mid-update don't see a broken site.

gulp.src('blog/**/*.html', {cwd: BUILD_DIR, base: BUILD_DIR})
  .pipe(awspublish.gzip())
  .pipe(waitFor(Promise.all([binaryResourcesDone, textResourcesDone])))
  .pipe(publish({
    'Cache-Control': 'max-age=' + 60 * 5 + ', no-transform, public'
  }));

Animating elements moving between lists

2015-07-19T10:00:00Z

Animating elements between parts of an application can be tricky. The beauty of ng-repeat means we can declare that a list, such as a ul, should represent a data model, such as an Array of data, and the list keeps up to date with whatever changes we make to the underlying model. The tricky bit comes when we want to view not just current state of the model, but transitions between states, such as

removal of an item from a list,
addition of an item to a list, or
moving an item from one list to another.

Pre-Angular 1.4, it was fairly straightforward to animate addition or removal of items from an ng-repeat powered list using .ng-enter and .ng-leave transitions. With 1.4, we can now animate moving a peice of data from one list to another.

The key to this is the ng-animate-ref attribute. If the ref-value on an outgoing element matches an element on an incoming element, then Angular clones the outgoing element, inserts the clone positioned absolutely on the page, and using CSS transitions moves it from the position of the outgoing to the position of the incoming.

What outgoing and incoming mean is that their addition to and removal from the DOM are subject to .ng-enter and .ng-leave transitions, either on themselves, or on a parent element. For example, we can animate the heights of elements to or from 0 when they are added or removed from a list:

}
<ul class="list">
  <li class="item" ng-repeat="item in listA" ng-click="toB(item)">
    Item: \{{ item.id }}
  </li>
</ul>
<ul class="list">
  <li class="item" ng-repeat="item in listB" ng-click="toA(item)">
    Item: \{{ item.id }}
  </li>
</ul>

with CSS transitions defined as:

/* New element set to 0 height after addition to DOM... */
.item.ng-enter {
  transition: 0.1s linear all;
  height: 0;
}

/* ...then transitioned to 30px */
.item.ng-enter.ng-enter-active {
  height: 30px;
}

/* Existing element set to 30px height just before removal... */
.item.ng-leave {
  transition: 0.1s linear all;
  height: 30px;
}

/* ... then transitioned to 0 */
.item.ng-leave.ng-leave-active {
  height: 0;
}

The above rules work because at the appropriate points in the addition and removal of elements from ng-repeat lists, Angular adds the classes on the elements and allows the browser to perform the CSS transitions.

Once the list elements are subject to .ng-enter and .ng-leave transitions, we add a wrapper span to each element, with an ng-animate-ref attribute containing the ID of the item, so Angular can match each element leaving the DOM with one entering.

}
<ul class="list" title="List A">
  <li class="item" ng-repeat="item in listA" ng-click="toB(item)">
    <span class="item-contents" ng-animate-ref="\{{ item.id }}">Item: \{{ item.id }}</span>
  </li>
</ul>
<ul class="list" title="List B">
  <li class="item" ng-repeat="item in listB" ng-click="toA(item)">
    <span class="item-contents" ng-animate-ref="\{{ item.id }}">Item: \{{ item.id }}</span>
  </li>
</ul>

Note that depending on the transitions we want, we might not need the extra wrapping span, but it keeps the code as flexible as possible, and avoids any ambiguity about what transitions are happening on what elements: the enter and leave transitions take place on the .item elements, and the animations between the lists happen on the .item-contents elements.

We can control the CSS transition with a single style rule on the .ng-anchor-in class that Angular adds to the cloned element.

.item-contents.ng-anchor-in {
  transition: 0.2s linear all;
}

You can have a more complex transition if you want by also having a transition on .ng-anchor-out, but this is not necessary for this case. You can see the docs for ngAnimate for more information.

You can see this in-action, together with the the rest of the boilerplate Javascript and CSS, in the below Plunker. Each item cam be moved by clicking on it.

That's it! There isn't really much to it.

Throttled variable changes

2014-12-20T10:00:00Z

Usually shadowing introduced by prototypical inheritance of $scope is something to be avoided, often the source of bugs cause by not having a dot in models. However, there is a way it can be used to throttle variable changes in templates.

Thottling variable changes is something you might want to do to avoid flickering of fast changing variables, so the user has a chance to see each state of the interface for a reasonable length of time. A "loading" message or spinner is quite typical, but if it ends up being shown for an extremely short amount of time, say 10ms due to a request having been cached, it can suggest something is a bit broken, or at least not give the user a great experience as it could be.

A small directive can do this, using throttle from Lo-dash or Underscore.

app.directive('throttle', function($window) {
  var WAIT_TIME = 750;

  return {
    scope: true,
    link: function(scope, element, attrs) {
      var name = attrs.throttle;

      function setValue(value) {
        // Shadow the value in the child scope
        scope[name] = value;

        // The leading edge of the _.throttle callback
        // is called within a digest, but later ones are not
        scope.$$phase || scope.$apply();
      }

      scope.$parent.$watch(name, $window._.throttle(setValue, WAIT_TIME));
    }
  };
});

which can be used as follows.

}
<div throttle="state">
  <div ng-if="state == 'loading'">Loading...</div>
  <div ng-if="state == 'loaded'">[Loaded template]</div>
</div>

Then in the parent controller you can set the throttled variable, state in the above case, as often as you like, and you can be confident that the effect won't be seen in the template more often than every 750ms.

Benefits of this technique over some others are given below.

The same name of variable is used throughout the template as in the controller that sets the variable. So you can really easily introduce this to existing templates, or remove it if you want to later.
The logic is purely in the template. The controller and factories/services don't need any code to throttle the variable changes.

Limitations or drawbacks of this method are below.

You can only use this for variables without a dot in them. However, my use case is for simple state variables that just contain a string representing the current state of the interface, which don't need to be set anywhere but in the parent controller, so they don't need a dot.
The private variable $$phase is used to create a safeApply equivalent. This could be worked-around, but most likely by re-implementing _.throttle.

Same-domain iframe communication

2014-07-27T09:19:00Z

It's possible for Angular apps to communicate, where one is running in an iframe of a parent, using standard scope events. This means you can treat an iframe much like a custom directive, responding to $broadcast-ed events, or $emit-ting its own.

This technique relies on the child app being able to access the $scope of the iframe element in the parent app. This is possible by using $window.frameElement to access the iframe element from within the child window, and the angular.element scope function to get its $scope. The child app can then call the parent app's $on and $emit functions.

For simplicity, I've opted to create a small factory in the child app that returns the scope of the iframe in the parent app. This way, it can be used in a very similar way to $rootScope.

appInIframe.factory('$parentScope', function($window) {
  return $window.parent.angular.element($window.frameElement).scope();
});

Then in any controller in the child app that wants to communicate with the parent, you can use these functions as per the following example.

appInIframe.controller('ChildController', function($scope, $parentScope) {
  $scope.messages = [];
  $scope.message = function() {
    $parentScope.$emit('from-iframe','Sent from iframe');
    $parentScope.$apply();
  };
  $parentScope.$on('from-parent', function(e, message) {
    $scope.messages.push(message);
    $scope.$apply();
  });
});

The only thing you should remember is that you need to trigger a digest manually in each app.

The controller in the parent app can then use standard $broadcast and $on to commnicate to the iframe.

appInParentWindow.controller('ParentController', function($scope) {
  $scope.messages = [];
  $scope.$on('from-iframe', function(e, message) {
    $scope.messages.push(message);
  });
  
  $scope.message = function() {
    $scope.$broadcast('from-parent', 'Sent from parent');
  };
});

You can see an example of this technique in a plunker.

Benefits of this technique over some others are given below.

No additional global functions added to either the parent or the iframe window. Although yes, it does depend on angular being global in the parent window.
The parent controller is ignorant of the fact that events go into an iframe.
IDs are not used by parent or child. Because the parent uses the standard $broadcast and $on functions, targeting a specific iframe from the parent controller uses exactly the same techniques you might use to target a custom directive.

AngularJS Transclude

2014-07-27T09:19:00Z

Transclusion allows a directive to move clone(s) of the contents of a directive to an arbitrary place in the DOM. Usually this is at the original location of the directive, wrapped in some extra elements, but it doesn't have to be. So why move the element? You might want components to still have access to the original scope, with any models and methods, but for page-layout or CSS reasons be elsewhere.

}
\{{model.variable}}
<move>
  Contents bound to local scope \{{model.variable}}
</move>

Example components that use this could be

Context menu
Dialog box
Sidebar
Alert box or notification area.

One thing to keep in mind is that you'll have to make sure you remove the element from the DOM when the original scope is destroyed. The high-level code would look like the following.

app.directive('move', function() {
  return {
    restrict: 'E',
    transclude: 'element',
    link: function(scope, element, attrs, controller, transclude) {
      transclude(scope, function(clone) {
        // Inject the element somewhere in the page
      });
      scope.$on('$destroy', function() {
        // Remove the element from the DOM
      });
    }
  };
});

There are a few techniques that can be used to communicate the clone of the element to the new location.

Directly walk the DOM to inject the clone, jQuery-style.
Use require to pass the clone to another directive by calling its controller directive
Fire off the clone using an Angular event $emit or $broadcast. Another directive would be listening to this event and inject the clone.

I should admit, I've not used this technique for anything but a few tests, so there could be some consequences about this that I haven't encountered.

AngularJS Promises

2014-04-12T09:19:00Z

AngularJS promises are an extremely powerful tool. They allow you to make a multi-layered and complex system based on asynchronous functions, with error and in-progress notification handling, all without getting into callback hell.

This post attempts to explain both creating and using AngularJS promises. It assumes some familiariy with AngularJS, specifically defining and injecting services and using controllers.

What are AngularJS promises?

At the most low-level, a promise is an plain old Javascript object (POJO) with then and finally functions:

{
  'then': function(successCallback, errorCallback, notifyCallback) {
    // Black box code
  },
  'finally': function(finallyCallback) {
    // Black box code
  }
}

and they are returned from functions whose meaningful result will be found or calculated later, i.e. asynchronously. The function will choose which of the callbacks to call, depending on success or failure of the asynchronous action. In case of success, the successCallback is called, and the promise is said to be resolved with a result. In case of failure, the error is called, and the promise is said to have been rejected with an error.

You should note:

You won't need to construct a promise yourself using object notation. As will be discussed, they will be created via function calls.
Angular promises have other functions, but they just shortcuts, so are ommited for this post.

Using promises

A good first step in understanding promises is to use ones that are created by an existing service. A common use of a promise in AngularJS, are ones created by the $http service.

var promise = $http.get('/my-url');
promise.then(function(result) {
  // Do something with the results of the GET request
});

In the preceeding example, the success of the GET request will cause the $http service to resolve the promise with the result of the GET request. This will then call the successCallback passing this resolved value as the first parameter.

It's quite common to not assign promises to an intermediate variable. The following is equivalent to the preceding example.

$http.get('/my-url').then(function(result) {
  // Do something with the results of the GET request
});

In the above examples, only the successCallback is used from the promise. If we want to handle a failure then

$http.get('/my-url-that-might-fail').then(function(result) {
  // Do something with the result of the GET request if it succeeds
}, function(error) {
  // Do something with the error if it fails
});

Although not currently possible, if $http ever ends up being modified to give in-progress notfications, then this could be used as

$http.get('/my-url').then(function(result) {
  // Do something with the results of the GET request
}, function(error) {
  // Do something with the error
}, function(update) {
  // Do something with the update
});

If you want to perform some action after the $http request has completed, whether it succeeded or failed, then you can use the finally function.

$http.get('/my-url').finally(function() {
  // Do something after either success or failure
});

The success, failure, notify callbacks can all be ommited, or passed null if you have no action to be performed

Chaining Promises

The real power from promises comes from chaining. The first key to understanding this is

The then and finally functions each return a new promise, known as a derived promise.

An example of using derived promises:

var promiseA = $http.get('/my-url');
var promiseB = promiseA.then(function(result) {
  // Do something with the results of the GET request
});
promiseB.then(function(result) {
  // Do something with the resolved result of promiseB
})

As with the case of a single promise, it is quite common to not assign the derived promises to intermediate variables. The following is equivalent to the preceeding example:

$http.get('/my-url').then(function(result) {
  // Do something with the result of the GET request
}).then(function(results) {
  // Do something with the resolved result of promiseB
})

The second key to understanding chains are:

Derived promises are resolved/rejected with the returned resolved/rejected value of the callback that was run.

In practice, this means there are 3 possible ways of controlling the derived promise. It can be resolved immediately, it can be rejected immediately, or its own resolution/rejection can be deferred further until a 3rd promise has been resolved/rejected, in which case the derived promise is resolved/rejected with the 3rd promise's resolved/rejected value.

Resolving a derived promise immediately

If you return anything but a promise from the callback that is run, the derived promise will be resolved immediately with that returned value

$http.get('/my-url').then(function(result) {
  return 'my-immediate-value';
}).then(function(results) {
  // results === 'my-immediate-value';
})

Be aware that not explicty returning a value means that you have returned undefined, and the derived promise will be resolved with undefined.

$http.get('/my-url').then(function(result) {
  // Not returning a value.
}).then(function(results) {
  // results === undefined
})

The above applies to both the success and the error callbacks. Returning any non-promise value from the error callback means that the derived promise will be resolved, and not rejected:

$http.get('/my-url-that-does-not-exist').then(function(results) {
}, function(error) {
  return 'my-immediate-value';
}).then(function(results) {
  // results === 'my-immediate-value'
})

As with the success callback, not returning a value from the error callback means the derived promise will be resolved with undefined

$http.get('/my-url-that-does-not-exist').then(function(result) {
}, function(error) {
  // Not returning a value.
}).then(function(results) {
  // results === undefined
})

I've done the above accidentally: it derives a resolved promise from a rejected one, which might be be desirable.

Rejecting a derived promise immediately

You may want to reject a derived promise, even if the original promise was resolved. This is done by returning the result of $q.reject().

$http.get('/my-url').then(function(result) {
  return $q.reject('my-failure-reason');
}).then(function(results) {
  // The code never gets here
}, function(error) {
  // error === 'my-failure-reason'
});

If you want to then fail the derived promise from an error callback, then you can do the same thing in it:

$http.get('/my-url-that does not exist').then(function(results) {
  // The code never gets here if the GET was unsuccessfull
}, function(error) {
   return $q.reject('my-failure-reason');
}.then(function() {
  // The code never gets here if the GET was unsuccessfull
}, function(error) {
  // error === 'my-failure-reason'
});

Deferring a derived promise

A powerful aspect of derived promises is that their resolution/rejection can be deferred until another promise has been resolved/rejected. This is done by returning a promise from the success or error callback. For example, if you want to run $http.get calls sequentially, and then do something after the final is successful, you can do this by returning the result of $http.get from the callback:

$http.get('/my-first-url').then(function(results) {
  return $http.get('/my-second-url')
}).then(function(results) {
  // results here are the results of the GET to /my-second-url 
});

Because each then call again returns a promise, you can easily add to this chain:

$http.get('/my-first-url').then(function(results) {
  return $http.get('/my-second-url')
}).then(function(results) {
  return $http.get('/my-third-url')
}).then(function(results) {
  // results here are the results of the GET to /my-third-url 
});

Rejection/error handling in promise chains

If a promise is rejected, then every subsequent promise in the chain will be rejected, until one is reached with an error callback.

$http.get('/my-first-url-that-fails').then(function(results) {
  // Never called
  return $http.get('/my-second-url')
}).then(function(results) {
  // Never called
  return $http.get('/my-third-url')
}).then(function(results) {
  // results here are the results of the GET to /my-third-url 
}, function(error) {
  // Error callback called
});

When specifying an error callback, be careful what you return. If you return a non-promise value, which includes undefined by not specifying a return value, then the derived promise for that callback will be resolved, and not rejected.

$http.get('/my-first-url-that-fails').then(function(result) {
  // Never called
  return $http.get('/my-second-url')
}, function(error) {
  // Error callback called
}).then(function(results) {
  // This *is* called, because the previous
  // callback returned undefined
  return $http.get('/my-third-url')
});

Layered APIs

A common use of promises is chaining them via layered APIs. A typical pattern in AngularJS is to have calls to $http functions in a service, so controllers are not aware that $http is used.

MyController -> MyService -> $http

You can do this using a structure like:

// In MyService
this.fetchResults = function() {
  return $http.get('/my-url');
};

// In MyController
$scope.fetchResults = function() {
  MyService.fetchResults().then(function(results) {
    // Do something with results
  });
}

However, this means that the controller will be exposed to HTTP headers and statuses. To hide this lower-level detail, you can add post-processing in the service via a derived promises:

this.fetchResults = function() {
  return $http.get('/my-url').then(function(results) {
    // Just return the HTTP body
    return results.data;
  );
};

You can also include some of your own error handling, so in the case of a failed request, the controller can be ignorant of any details of HTTP:

// In MyService
this.fetchResults = function() {
  return $http.get('/my-url').then(function(results) {
    // Just return the http body
    return results.data;
), function(error) {
  return $q.reject('Oh no!');
});

// In MyController
$scope.fetchResults = function() {
  MyService.fetchResults().then(function(results) {
    // Do something with results
  }, function(error) {
    // Do something with error if it occurs
    // which would be 'Oh no!'
  });
}

Creating Promises

If you're not chaining onto an existing promise, you might need to create a new, non-derived, promise. You can do this by calling $q.defer(). This returns a deferred object:

var deferred = $q.defer();

The deferred object contains a promise, and methods to call to control that promise:

{
  'resolve': function(result) {
    // Black box code
  },
  'reject': function(error) {
    // Black box code
  },
  'notify': function(update) {
    // Black box code
  },
  'promise': // Promise as described above
}

When you want to resolve the promise, you can call the resolve function. Similarly for the reject and notify functions.

Using this, a simple timer (ignoring the existence of $timeout) can be written as

var simpleTimeout = function(time) {
  var deferred = $q.defer();
  $window.setTimeout(function() {
    deferred.resolve('Done!');
  }, time);
  return deferred.promise;
}

Which could be used as:

simpleTimeout(1000).then(function(result) {
  // Code gets here after 1 second, and
  // result === 'Done!'
});

You could also have a timer that sends a notification:

var simpleTimeout = function(time) {
  var deferred = $q.defer();

  $window.setTimeout(function() {
    deferred.resolve('Done!');
  }, time);$

  $window.setTimeout(function() {
    deferred.notify('Half way there!');
  }, time/2);

  return deferred.promise;
}

Which could be use as:

simpleTimeout(1000).then(function(result) {
  // Code gets here after 1 second, and
  // result === 'Done!'
}, null, function(update) {
  // After 1/2 second, code gets here
  // and update === 'Half way there!'
)};

Exceptions thrown in callbacks

A not very documented feature of Angular promises is that when exceptions are thrown in the callbacks, via throw, then the derived promise will be rejected with that thrown value.

$http.get('/my-url').then(function(results) {
  throw 'my-failure-reason';
}).then(function(results) {
  // The code never gets here
}, function(error) {
  // error === 'my-failure-reason'
});

This will also trigger Angular's registered exception handler, so it's not quite equivalent to using $q.reject().

ECONNRESET Errors in node.js

2013-08-14T19:00:00Z

The Problem

Running a basic node.js + express + socket.io setup, I would occasionally get ECONNRESET errors. After a bit of trial and error, I found that this seems to have been caused by running socket.io and express on the same port, port 80.

Solution

I opened up port 81 on the server, and ran socket.io through port 81. My code to set things up now looks like:

var httpPort = 80;
var socketIOPort = 81;

var express = require('express');
var app = express();
app.listen(httpPort);
var socketIO = require('socket.io');
var io = socketIO.listen(socketIOPort);

I then had to update the client-side socket.io Javascript to connect on port 81, and so far I've never had these errors since. For reference I'm running node.js v0.10.0, Express v3.3.4 and socket.io v0.9.16.

Dependency-Based Cache Invalidation

2013-07-25T09:00:00Z

The problem

With an ever-increased user and code-base, with ever increasing amounts of data, we are approaching limits of what any single relational database can handle over at intelligentgolf. Caching output of database queries and/or generated HTML must be part of any solution, as it's clearly innefficient to be re-calculating the same things again and again on every page load. While caching things for a set amount of time is easy, the problem is that users demand to see their updates as soon as new data is in the system, for all but a few cases. So the cache must be invalidated. To pull out the obligatory quote:

There are only two hard things in Computer Science: cache invalidation and naming things.
Phil Karlton

So, liking a challenge, I set out to design a caching system, for PHP, with the following desired qualities:

Users never see out of date content.
Not to tightly couple the cache code to any other logic.
Not to have to maintain any function that invalidates the cache when a model / database rows is/are updated.
Avoid long processes on update of data.
Any caching code that explains a dependency to the caching system must be next to the code that uses this dependency.
Be flexible enough to cache HTML or arrays of data. Therefore the system can be used to both cache either expensive database calls, or expensive operations in the front-end webserver.
Be able to address the slowest / most frequent accessed pages, or parts of pages, first, without a rewrite of the entire code-base.
Be suitable for non-trivial cases, where cached content can depend on other potentially cached results and results of other functions. HTML pages often depend on data from multiple tables, expensive database queries and/or post-database processing. This is what I understand Russian doll caching to be.
Be suitable for a distributed architecture on virtual servers, such as a load-balanced front-end web servers on AWS.
The cache is a single key => value store, where the only access for data is by its primary key, such as Memcached
Ultimately, the dream is that entire HTML pages could be loaded without calls to the underlying database.

What I do think are acceptable trade-offs:

A slight slowdown of code on generation of a result when there is a cache-miss, to accomodate the caching system. The cache could be primed if this is too long.
Cached results can, in some circumstances, be needlessly recalculated, as long as the chance of this quickly decreases with time.

Key-Based Invalidation

The above requirements seem to lead to the option of key-based cache invalidation. This is where items are never expired from the cache. Instead, before fetching an object from a cache, the key that identifies the cached object, is itself amended with a date, version, md5, or some unique modifier, that will change whenever there is a change in the underlying data that the cached object itself depends on.

This can work well for dependencies that are known, or can be appropraitely assumed, before any cached result is started to be calculated. For example, the current time or date, any query string parameters, configuration options, or MD5 of code or template files. However, there can be dependencies that are only determined by querying the underlying database, which is precisely what is to be avoided.

An example of this is a list of pages created by a given user, found by the following SQL.

SELECT id, pagename FROM pages WHERE pages.created_by = 1234

What pages those are are only known after the SQL is run.

Dependency-Based Invalidation

To address this issue, for any new result that is cached, stored with it is a list of dependency identifiers, each with a version identifier, specifying the version at which the dependency was when the cached resulted was computed. Stored separately, is a list of current versions of all the dependencies, which is a) kept updated with the latest version of any dependency, and b) checked before accepting any cached data as a cache hit.

All far so good, but still things needs to be designed.

Dependency Version Identifiers

How to identify the current version of any dependency needed some thought. I considered MD5 of any dependent data, version numbers, and using a plain-old last-updated time. I decided that last-updated time was the best option. It has certain properties that make it suitable:

Although it can't be regenerated from the data if lost, it can be safely assumed to be the current time, which expires any dependant cached results. Version numbers can't be safely assumed to start from 0, and therefore must be stored permanently.
It doesn't depend on the actual data, like MD5, so it can be used for any source, including "abstract" or remote sources that don't really have anything suitable to take an MD5 of.
When a cache-miss occurs, checking the last-updated time for any dependency can often only occur after the dependency has been accessed. This means there is race-condition between these, when a second process could have updated the dependency, along with its last-updated time, and potentially causing the first process to store a cached result with an incorrect dependency last-updated time. In order to avoid this, the time at the beginning of the cache-miss can be compared with the last-update time of any dependencies. If any dependency has been updated since the start of the miss, then the result found can be used in the rest of the flow of the page, but not cached. This wouldn't be possible with version numbers or MD5s.

However, there are a few drawbacks.

Updating last-update time as an atomic operation would be a bit more complicated than incrementing a version number, but still possible using Memcached's CAS tokens.
A cluster of virtual servers can't be guaranteed to be running at precisily the same time. However, if we can assume a maximum difference in times, say, 2 seconds, then in the case of a cache-miss, the new results will only be stored if all of the dependencies were last updated more the 2 seconds before the start of the miss.

Dependency Idenfitiers

The structure of a dependency identifier is also cruicial. The code that updates a depedency must use the same identifier that code that uses it. If identifing data by its unique primary key, this is fairly easy. It could be of the form

 [table-name]:[primary-key-value]

For example, "users:1234" could be the identifer for a row in the users table.

However, it gets complicated when accessing data by any other means where accessing data by a non-unique key. For example, if wanting to generate a list of pages created by given user in a CMS, this list could be ultimately generated by a SQL like

SELECT id, pagename FROM pages WHERE pages.created_by = 1234

Then not only would the dependency list have to include all these pages, but because the list of pages can be increased, or items removed from it, there must be some dependency on the "WHERE" clause. So my design is to have the identifiers in the following format.

 [table-name]:[column-name]:[column-value]

I must then ensure that whenever a row is added or removed from the database, then, for every possible column that could be used as a foreign-key, or in a where-clause, then that dependency has to be treated as updated. For example, marking page with id 2145 as deleted would need 2 dependencies updated:

 pages:id:2145
 pages:created_by:1234

I was hoping to avoid having anything but one update of a last-update time for any given dependency, to make data updates fast, but I've not worked out a way around this. At least, I suspect the maximum number of updates to last-update time would be equal to the number of columns of the table being updated, so it shouldn't increase with the amount of dependent cached items increased.

Coming in part 2 are some implementation details, along with, hopefully the code.

About This Site

2013-07-14T09:00:00Z

A few technical details about this site.

Front End

A couple of fonts are included from Typekit. and a few Fontawesome icons are used as well.
Disqus for comments, as below.
Repsonsive by way of a single @media query to change the position of the content relative to its title.
At the moment, no HTML5 tags other than the DOCTYPE. It's on the todo list. HTML is also a bit too complicated as in a previous version there was some over-complicated Javascript to show all posts on a single page

Generation

Uses LESS for more consistent CSS and easier writing.
Generated by Jekyll as static files.
Assets are concatanated, minified, and md5s added to filenames, to allow far-future expires / age headers on assets, by Jekyll Assets.

Hosting

Hosted on Amazon S3 in Ireland, uploaded by s3_website, gzipping as much content as possible.
Uses AWS Cloudfront as CDN, just in case this is ever popular far from Ireland :-). This includes serving the HTML through the CDN, whose filenames don't change and so I'm invalidating any changed HTML on uploade automatically using s3_website.
The DNS for charemza.name is hosted by Amazon Route 53. Their ALIAS records allow the content on the root domain charemza.name, without the www, to be fronted by the CloudFront CDN.
My email for michal@charemza.name is hosted by Google Apps for Business.

Version Control

Git, available at https://github.com/michalc/blog.

Performance

Running a simple test from LoadImpact, from Singapore, shows:

The first loads are quite slow, at 2 seconds. I assume the local CloudFront location is empty.
The later requests hover at 20ms. Not too shabby.

JQuery UI Datepicker Incorrect Century

2013-01-27T07:55:30Z

This is a known bug bugs.jqueryui.com/ticket/8873, but I couldn't find it on a search, so it might be helpful to post here.

I was creating a datepicker for a date of birth field, with no default value, and with a yearRange of "c-100:c-1". However, the year dropdown kept on jumping to 100 years before the actually selected year. I changed the yearRange option to "c-100:c" and it then worked fine.

Thinking Fast and Slow

2013-01-20T07:55:30Z

Thinking Fast and Slow is a pop psychology book, but one that leans on the academic side, detailing psychological experiments and applications of their consequences. Its main theme is how the assumptions of traditional Behavioural Economics, the main one of which is that people are rational, are flawed, and it gives many examples of results that show this. Here the use of the word rational specifically means that people will decide to do the thing that will give them a best chance of increase their utility, which from my understanding roughly means a measure of happiness. Often this behaviour is attributed to how System 1 and System 2 processes in the mind behave, which roughly mean "instinctual and instant" and "slow and conscious" respectively. System 1 is subject to a number of biases, and a number of experiments that reveal them are explained and discussed.

It's an extremely rich book, and one that I found quite eye opening. I thought I was fairly self-aware of the reasons why I do things, and that I am fairly rational. However, many times when reading I realised that I am guilty of the same irrational behaviour that everyone else seems to be. One the more self-help side, it gives suggestions on how to combat such System1 processes.

As a possible criticism, I'm unsure about the separation of all thoughts into 2 distinct categories. Perhaps in the future that will be revealed to be an over-simplification.

AWS Load Balancer

2013-01-19T07:55:30Z

A word of warning to those who use AWS Elastic Load Balancing. The default configuration is that any request that hasn't had any data sent from the web server for 60 records, results in the balancer resending the request. The reason for this, presumably, is so that if a server has dies mid-way through a request, the user still would get their request processed by, potentially, another server.

However, at intelligentgolf, we do some processing that can result in requests that quite naturally last longer than 60 seconds: sending mass email, or interfacing with external systems, which require 100s or 1000s of external connections to interface with 3rd party legacy systems. In some of these cases, we found that duplications were happening, even though we were sure that the form wasn't being submitted twice.

So, what we do in a lot of cases now:

Use a method on the server to detect if a submit button was pressed twice, and effectively ignore the second. This is a good practice in any case, AWS load balancing or not. Javascript checking is not enough here, as the duplications happen due to behaviour of the load balancer, the browser is completely unaware of the duplicate request.
Do the batch processing in the background, using a cron job, or other similar method.

\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}^c(p))\)	✓
\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✓
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✓
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✓

\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}^c(p))\)	✗
\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✗
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✗
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✗

\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}^c(p))\)	✗
\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✗
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✓
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✓

\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}^c(p))\)	✓
\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✓
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✗
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✓

\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}(p))\)	✗
\(W(\mathbb{L}(p))\)	\(W(\mathbb{L}^c(p))\)	✓
\(W(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✓
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}(p))\)	✓
\(R(\mathbb{L}(p))\)	\(R(\mathbb{L}^c(p))\)	✓

	\(\overline{R_A}(p)\)	\(\overline{R}(p)\)	\(\overline{W_A}(p)\)	\(\overline{W}(p)\)
\(\overline{R_A}(p)\)	✓	✓	✓	✗
\(\overline{R}(p)\)	✓	✓	✗	✗
\(\overline{W_A}(p)\)	✓	✗	✓	✗
\(\overline{W}(p)\)	✗	✗	✗	✗

	\(\overline{R}\)	\(\overline{W}\)
\(\overline{R}\)	✓	✗
\(\overline{W}\)	✗	✗