Sunk Cost Fallacy: chasing a half-baked idea for much too long

This is a part of a series where I review the work I’ve done over the past couple of years.

1. Two-year review: to plan a strategy you must first have a theory of how the hell things work
2. Out of the Software Crisis: two year project review
3. Sunk Cost Fallacy: chasing a half-baked idea for much too long (this page)
4. The Intelligence Illusion: stepping into a pile of ‘AI’
5. A print project retrospective: the biggest problem with selling print books is the software
6. Thinking about print
7. Disillusioned with Deno
8. An Uncluttered retrospective: Teachable is a mess and I need to pick a lane

Colophon Cards

It all began well.

In the first half of 2022 I worked, part-time, on a research project called Colophon Cards that was funded by The Icelandic Centre for Research’s Technology Development Fund.

I’d applied for a grant to create a design prototype of a note-taking app. I wanted to combine academic research into knowledge management and retrieval with direct user research and testing of my own. To my own surprise, I got the grant.

It went well, as far as these things go.

The research itself included:

• Forum and subreddit studies to try to discover some of the core problems people were having. This was during a minor note-taking fad as the field rediscovered the decades-old technology of “backlinks”.
• Direct user research
• User testing of paper and digital prototypes

And at the end I was left with a few different design prototypes I was relatively satisfied with.

I documented the research on the project site and on my blog, but I didn’t publish the design prototypes because I wasn’t sure what I was going to do with them next.

So far, so good and it left me with an established, if minor, presence among note-taking enthusiasts.

Building on this should have been straightforward. It would have had to wait a bit as COVID recovery led to an impromptu book.

But once that had settled the sensible thing would have been to continue the work in early 2023 with newsletter essays on knowledge management and note-taking as well as taking one of the design prototypes and turn it into a technical prototype to test the waters.

But there was one wrinkle – or a crossroads of sorts.

A crossroads where I took the wrong turn.

This wrong turn would end up derailing the entire year, which I then compounded by spending too much of my time on another misguided project.

The wrong turn

I made a few mistakes with the project. The name was the first one. Nice as the name “Colophon Cards” is, it was tied a bit too strongly to a specific design metaphor. The project would have benefited from a bit more freedom to deviate from that metaphor. The assumption that the project would inevitably target power users with every complex features you can think of was also misguided.

The basic design I ended up with instead prioritised findability and retrieval. Instead of having complex structural and organisational features, it would prioritise the speed and effectiveness of finding and retrieving the specific note you needed. I would concentrate on a single use case for note-taking instead of all of them: memory aid.

That is, what you would normally call bookmarking, but with the added ability to store ad hoc notes and maybe the occasional arbitrary file.

This design was a very good start to a potential project. It addressed a recognised need (plenty of bookmarking tools in business) while still being clearly differentiated from the rest.

The wrong turn came because I decided I had to solve the “offline problem”.

Classic developer scope creep.

The “offline problem”

People generally expect their notes to be available immediately and that opening a note shouldn’t rely on a network connection.

For example, a recurring complaint from Notion users is that it doesn’t have an offline mode. (Personally, if I were them I would have prioritised offline support over bullshit “AI” features, but that’s just me.)

In the research I did, my impression was that offline use was one of the more important requirements for expert note-takers. The note-taking software scene has been fairly competitive in the past. Many people I interviewed talking about having tried out several apps over the years. The need to store your notes offline was, in part, a symptom of distrust.

This problem is much less relevant if your primary use case is bookmarking and similar storage and retrieval tasks, but nevertheless I set out to “solve” this problem for my app.

This was a bad decision both in terms of UX design, software design, and just overall execution of the idea. It resulted in massive over-engineering and overthinking on my part.

Not in the least because there are too many ways of making “offline” work.

Synchronisation

This is probably the most straightforward approach to the problem: Store data locally, then sync to server. It’s limited for collaborative applications as synchronising whole files means that when edits are simultaneous, one user’s updates will wipe out the others, but it works well for single-user applications.

It’s not simple, but it is the simplest of the various local-first approaches.

CRDTs

Conflict-free Replicated Data Types are another approach. These are data types that are specifically designed to allow for distributed work. Explaining their details is well beyond the scope for this document, but overall they do actually work and make distributed collaboration between multiple users possible.

Their downside is that they’re quite slow, memory intensive, and require substantial file space, relative to normal file editing. Existing CRDT libraries, such as Yjs are some of the most optimised pieces of JS code you’ll ever see and even that hasn’t been enough. Both the Yjs and automerge projects have invested a lot of work in even more optimised rewrites in Rust.

Distributed version control

Related to CRDTs are distributed version control systems such as Git, Darcs, Mercurial, or Fossil.

Many of these actually use CRDTs in some way, though not usually formally defined ones.

One of the benefits of a distributed VCS based on three-way merges of a file tree (like git) over CRDT systems like Yjs are the lower performance and memory requirements during edits.

There is no overhead when you’re working with the files directly and the system only comes into play during commits, merges, and syncs (pull and push).

I liked this aspect and had the idea of how to build on it.

I did an experiment where I had a go at making a web API native version control system that used the Origin Private File System, SubtleCrypto.digest(), and fetch to create a no-dependency version control system that worked in the browser sandbox and used Hybrid Logical Clocks to resolve merge conflicts.

The first version was hacky and took an unreasonable number of shortcuts, but it worked well enough for me to become enamoured by the idea and became convinced that I needed to use it for something.

The problem is that a bookmarking service doesn’t really need any of this. In fact, very few apps genuinely need any of this. It’s not a big enough improvement on existing methods for creating collaborative apps for the investment in it to make sense.

But I was convinced I could turn this into something meaningful, so I decided to come up with an app idea – a collaborative document editing service – and see if I could get funding to develop it further.

The wrong move

This was close to the opposite of what I should have done. I had decided on the outcome first and then, for several months, spent much too much of time I didn’t really have to spare looking for ways to accomplish an objective that wasn’t backed with any meaningful evidence.

Of course, it kept getting rejected. But with each funding application I put more work into the idea and more thought in the system’s design and the more invested I became in it.

It’s pretty close to being a textbook case of the sunk cost fallacy, which means I can’t even say that I don’t know what I was thinking. I do know: “All this work I’ve done can’t have been a waste! I just have to do more work to get it going!”

These are not rational thoughts, and it took me much too long to see that.

Let’s review the principles I outlined in the first post in this series:

1. Affordable risk.
2. Co-creation.
3. Leverage your means.
4. Turn lemons into lemonade.
5. Control the unpredictable.

While you could say that my push towards getting funding a half-baked research idea was clearly an affordable risk – after all I stayed in the black and paid my bills – it didn’t fulfil any of the other principles. It also had a substantial opportunity cost: pretty much anything else I could have done would have made me more money.

The reason why the project needed funding was the only app complex enough to need this complex a solution was too ambitious for me to accomplish with my means.

I worked alone on the grant applications and the code, avoiding even a gesture towards co-creation.

By relying on research grant applications I ceded control over what I would do next to an anonymous board of assessors.

And I definitely wasn’t turning lemons into lemonade.

More importantly, from the perspective of more practical frameworks like LaunchFTW, I was not addressing a clearly defined problem for a clearly defined audience that I understood and could reach.

So, what would the right move have been?

What I should have done

Similar to what my follow-up to Out of the Software Crisis should have been, I should have begun by looking at what I could accomplish and control with my means and resources.

Grant applications and chasing funding inevitably means ceding control. Anything that requires funding is also by definition beyond my existing means.

The first step would have been to continue to research the various problems people have with bookmarking, knowledge management, and similar tasks. If, in my research, I found somebody with a problem that I knew how to solve, I could have written a blog post or newsletter entry on how you would solve that problem.

It would also have made sense to write more about the issues with knowledge management and retrieval (i.e. note-taking) that specifically concern software developers. That would have built on the audience I got by publishing Out of the Software Crisis.

In the longer term this might have led to a book. Boxes and Boards: project-oriented note-taking methods from the creative industry would have worked as a title and would have tied together the research I’ve already done.

In terms of software development, the version control experiment should have been punted to the side, or turned into a throwaway open source repository at most.

Colophon Cards itself is tougher. Slowly building it into a Minimum Viable Product would have made sense, but building an audience on bookmarking and note-taking would have to have been a priority as that would have both helped a potential future SaaS and also opened up other opportunities in the future, like a book or an online course.

Turning it into an open source project might have mitigated some of the risk or at least bought it some mindshare. Cards might have had more value as a free software project than as a SaaS in terms of building an audience and selling other services.

But, I didn’t do any of these things.

What should I do now?

As I mentioned in the project review I did of Out of the Software Crisis, good ideas don’t necessarily have an expiration date. You might lose some momentum, but a fundamentally sound idea often remains fundamentally sound even after momentum is lost.

Writing about knowledge management, note-taking, and bookmarking is still something I should do more of. I have the research and much of it is still relevant to software developers, even in this era of LLM-induced nonsense.

Working towards a Colophon Cards MVP still makes sense, although it should be a relatively low priority task.

And I really should go through what I put together for those grant applications and see if any of it can be cleaned up to throw into a repository.

But, having made the mistake in the first place actually opens up some opportunities. That’s what the Lemonade principle exists to remind us of. I know have fairly extensive knowledge about the Origin Private File System APIs, CRDTs, distributed collaboration, hashing, diffing, binary diffs, and three-way merges that I could try to apply in some way. Whether that means writing about these concepts and APIs or creating simple local-first tools for people, I don’t know.

But the fact remains that I now have skills purely because of my misguided dedication to a half-baked idea.

That’s useful.

And it’s a powerful reminder that even big mistakes can have some benefit.

Next: the mistake of involving myself in the AI Bubble.