Senko Rašić

Random thoughts | about

I've recently installed a new laptop, which was an opportunity for me to revisit and revise the default software I usually install on any new workstation I set up (I use the term “workstation” here to mean a laptop or desktop machine that I can comfortably use in my daily work — software development in web/backend, AI, audio/video/streaming and related areas).

Here's my latest setup, roughly in the order of installation:

Debian / Ubuntu

I prefer using a Debian-based distribution, ideally Debian stable if all the hardware is supported. Right now Debian 12 (current stable) is pretty old and doesn't support the latest (Meteor Lake) hardware. Instead of mucking around with testing or unstable (which can be fun, but the fun can strike at inopportune times), I've installed the latest Ubuntu (24.10) which supports almost everything out of the box (Linux users will not be surprised to heard I had to tweak some driver options to get suspend/resume working).

The OS installation is pretty standard, the only non-default option I pick is to encrypt the whole disk. Having a non-encrypted disk on a device that can be easily stolen is a no-go for me. On the desktop workstation, I also set up SSH so I can remotely access it from elsewhere, but don't open any ports on the router.

I don't like Ubuntu's tweaks to the GNOME desktop and Snap packages, so if installing Ubuntu I do remove those. In general, I prefer setting up the apt repositories for 3rd party packages (getting auto-updates and all the other apt goodies). If the app doesn't have a repository but has a .deb package, I'll install that. I can also live with flatpak packages, and as a last resort, I'll manually install the app into /opt/<app> (if it's a GUI app or has many files) or /usr/local/bin (if it's a single binary).

GNOME

GNOME is a pretty opinionated piece of software. The developers' particular set of opinions resonates with me and I've been a very happy user for the past 20 years or so. I prefer vanilla GNOME interface (ie. without Ubuntu tweaks) and only need minimal customizations (mostly a few key shortcuts).

I do tend to only use the basic system and utilities. Not because I don't want to use various GNOME apps, but they just don't fit in my preferred workflow much.

1Password

I keep all my passwords in 1p, so it is the first thing that gets installed on a new machine after the OS is installed. It's very easy to set up – install via their apt repo, scan the QR code on my mobile 1Password app, and it's all there.

I also keep my SSH private keys in 1Password and set it up as the ssh agent. This way, I only need to unlock 1Password to unlock my SSH keys.

Dropbox

The next app is Dropbox. I'm a paid user and keep everything important (documents, company and personal documents, some media files, etc.) there. I also use Dropbox to auto-upload my mobile photos and videos, and symlink Pictures/, Videos/, and Music/ to the respective Dropbox folders. Another useful option I use is to scan documents, bills, etc. with my mobile and have them auto-uploaded to Dropbox. Though the quality is not the same as with a proper scanner, it's good enough for most purposes.

I don't keep my code, or the dotfiles/settings in Dropbox.

Firefox / Firefox Dev Edition

I use Firefox as my main browser, and a separate installation of Firefox Developer Edition for development work. Although Firefox is already available on both Ubuntu (via Snap) and Debian (Firefox ESR), I remove those and install the latest version directly from Mozilla's apt repositories.

I love the Multi-Account Containers feature/extension in Firefox. I also install UBlock Origin (ad/tracking blocker), 1Password (1p integration) and Kagi (search engine) extensions. I use (and pay for) Kagi as my search engine, and I'm very happy with it.

Many of the services I use daily are web-based (Fastmail for private mail, Google apps for work, GitHub,)

Tailscale

I have a personal VPN provisioned with Tailscale. Setup involves installing and enabling tailscale client and logging in with my account. Once enabled, I can connect from my laptop to my desktop from anywhere without punching holes in my router or worrying about security.

Visual Studio Code

I use VSCode as my main editor, mostly for editing Python, JavaScript and Markdown files. I use the official Microsoft binary and immediately turn off all telemetry (hopefully all!).

I heavily use the Remote SSH feature of VSCode: most of my projects are located on my desktop. When on Laptop, I open them via Remote SSH, and since that goes through the Tailscale VPN I can do this anywhere in the world. The SSH latency (for terminal work) can be a bit high when accessing from another continent though.

One thing I was worried when first setting this up is potential conflicts if the same project is opened locally (on desktop) and remotely (from laptop) at the same time, but I haven't had any issues with it.

VSCode has pretty good support for Python (including my linter/formatter of choice, ruff) and JavaScript. I also use GitHub Copilot, mostly as a smart auto-completion tool.

Obsidian

I love Obsidian, but keep it simple. I use it without any extra plugins or customizations – just a bunch of Markdown files in folders. I keep the data in Dropbox to get free sync with Dropsync on my phone.

CLI tools

My terminal app of choice is Tilix. It's fast, has tiling support, and integrates nicely with the rest of the GNOME terminal. I use Bash with minimal customizations (prompt, a few aliases and history settings).

I use vim for quick edits in terminal (no config to speak of – just syntax and smart indentation), ripgrep to search in files and fdfind to search files by name/extension. When connecting to a remote server I prefer screen (shows my age, I guess).

As a Python developer, I love the new ruff (a linter/formatter) and uv (package manager) tools so these get installed immediately (just drop them in /usr/local/bin).

I use git for version control, and my personal and work repos are hosted on GitHub. I don't use their CLI app though.

Media

OBS Studio

For any kind of screen recording or streaming, I use OBS Studio. Pretty vanilla setup, works great out of the box, I barely scratch the surface of its capabilities.

CLI tools

I use the command-line mpv for video playback, ffmpeg and friends for audio/video manipulation in the command line, and yt-dlp for downloading videos from YouTube (hey, that's not piracy, I'm a YT Premium subscriber!).

GIMP

If I need to do some image editing (cropping, resizing, adding text, minimal tweaking) I use GIMP. I'm not a graphic designer or a photographer, so GIMP is more than enough for my needs.

Spotify

I still have an old carefully-curated archive of MP3s somewhere, but these days I just use Spotify across all my devices. After a few years of use I've favorited enough of the songs I like so that it recommendations are mostly on point. Not everything is there, and for that I use YouTube. I also like the fact that I can just download all my liked songs for offline use (eg. when outside wifi and mobile coverage, on a plane, or roaming).

Online meetings

This depends on what I'm working on and with whom, but some combination of Slack, Zoom, Google Meet, and Discord (ideally all in the browser whenever possible).


This list is not exhaustive, but it covers the apps I use (almost) daily and will invariably need on a computer. All of them also have very good alternatives, so the list is highly subjective – it contains the tools I prefer and that work well in my personal workflow. Each time I (re)install a workstation there's some tweaking, but these are the ones that I keep coming back to.

Conventional wisdom these days, especially for startups, is to design your software architecture to be horizontally scalable and highly available. Best practices involve Kubernetes, microservices, multi-availability zones at a hyperscaler, zero-downtime deployments, database sharding and replication, with possibly a bit of serverless thrown in for good measure.

While scalability and high availability are certainly required for some, the solutions are often being recommended as a panacea. Don't you want to be ready for that hockey-stick growth? Do you want to lose millions in sales when your site is down for a few hours? Nobody got fired for choosing Kubernetes¹, so why take the chance?

In ensuing flame wars on Hacker News and elsewhere, this camp and the opposing You Ain't Gonna Need It (YAGNI) folks talk past each other, exchange anecdata and blame resume-padding architecture astronauts for building it wrong.

A hidden assumption in many of these discussions is that higher availability is always better. Is it, though?

Everything else being equal, having a highly-available system is better than having one that has poor reliability, in the same way as having fewer bugs in your code is better than having more bugs in your code. Everything else is not the same, though.

Approaches to increase code quality like PR reviews (maybe from multiple people), high test coverage (maybe with a combination of statement-level and branch-level unit tests, integration tests, functional tests and manual testing) increase cost proportionally to the effort put in. A balance is achieved when the expected cost of a bug (in terms of money, stress, damage, etc.) is equal to the additional cost incurred to try and avoid the bugs.

You can't add a bit of Kubernetes, though. The decisions about horizontal scalability and high availability influence the entire application architecture (whichever way you choose) and are hard to change later. The additional architecture and ops complexity, as well as the additional platform price to support it, goes up much easier than down.

Faced with this dilemma, it pays to first understand how much availability we really need, and how quickly we will need to scale up if needed. This is going to be specific to each project, so let me share two examples from my personal experience:

At my previous startup, AWW, our product was a shared online whiteboard – share a link and draw together. It was used by millions of people worldwide, across all timezones, and the real-time nature meant it had to be up pretty much all the time to be usable. If you have a meeting or a tutoring session at 2PM and are using AWW, it better be working at that time! One stressful episode involved scheduled downtime on an early Sunday morning European time and getting angry emails from paying customers in India who couldn't tutor their students on the Saturday evening.

Clearly, for AWW the higher the availability, the better. During COVID, we also experienced the proverbial hockey stick growth, servers were constantly “on fire” and most of the tech work included keeping up with the demand. A lot of complexity was introduced, a lot of time and money was spent on having a system that's as reliable as it can be, and that we can scale.

On the other hand, at API Bakery, the product is a software scaffolding tool – describe your project, click a few buttons to configure it, get the source code and you're off to the races. It's a low-engagement product with very flexible time limits. If it's down, no biggie, you can always retry a bit later. It's also not such a volume product that we'd lose a bunch of sales if it's down for a few hours. Finally, it's not likely to start growing so fast that it couldn't be scaled up the traditional way (buy a few bigger servers) in a reasonable time frame (days). It would be foolish to spend nearly as much effort or money on making it scale.

When thinking about high availability and scalability needs of a system, I look at three questions (with example answers):

1) How much trouble would you be in if something bad happened:

  • low – nobody would notice
  • minor – mild annoyance to someone, they'd have to retry later; small revenue loss
  • major – pretty annoying to a lot of your users, they're likely to complain or ask for a refund; significant revenue loss
  • critical – everything's on fire, you can't even deal with the torrent of questions or complaints, incurring significant revenue and reputation loss
  • catastrophic – you're fired, your company goes under, or both

2) How often are you prepared to experience these events:

  • low – daily or weekly
  • minor – once per month
  • major – once or twice per year at most
  • critical – hopefully never?
  • catastrophic – definitely never!

3) What's the downtime for each severity:

  • low – 30s/day (AWW – we had auto-recovery so this was mostly invisible to users), 5min/day (API Bakery)
  • minor – 5min/day (AWW), 1h/day (API Bakery)
  • major – 1h/day (AWW), several hours outage (API Bakery)
  • critical – 4h+/day (AWW), several days outage (API Bakery)
  • catastrophic – 2+ days (AWW), a few weeks outage (API Bakery)

These are example answers to give you intuition about thinking in terms of expected cost. In this case, it's obvious that the availability and scalability needs of AWW and API Bakery are wildly different.

Quantifying the costs of (not) implementing some architecture or infrastructure decision is harder, and also depends on the experience and skill set of people involved. Personally, for me it's much easier to whip up a VPS with a Django app, PostgreSQL database server, Caddy web server, with auto-backups, than it is to muck around with Helmfiles, configuring K8s ingress and getting the autoscaling to work, but I know there are people who feel exactly the opposite.

When quantifying the cost, I think about:

  • is this something we already know how to do?
  • if not, does it make sense to try and learn about it (appreciating the fact that we will certainly do a substandard job while we're learning)
  • can we engage outside experts, and will we be dependent on them if we do?
  • what are the infrastructure costs, and how easy is it to scale them up or down?
  • how will the added complexity impact the ongoing development, growth and maintenance/ops of the system?
  • how far can we push current/planned architecture and what would changing the approach entail?

We might not be able to get perfect answers to all these questions, but we will be better informed and base the decision on our specific situation, not cargo-culting “best practices” invented or promoted by organizations in a wildly different position.


¹ I'm not hating on Kubernetes or containers in general here. Those are just currently the most common solutions people recommend for failover and scaling.

The post-pandemic return-to-office (RTO) mandates are still the subjects of intense debate about working in the office or remotely, with two sides often talking past each other. Can there even be a solution that would satisfy both sides? [0]

The setup

Assuming rational knowledge workers Alice and Bob with clear preferences, and a rational manager, Michael, let's think through their preferences, constraints, problems that arise and potential solutions.

Let's assume Alice and Bob both live in the same city (or within daily-commute distance) and work for Acme, Inc., a successful tech company whose CEO Michael wants to do the best for all the stakeholders (the employees, the shareholders, and the consumers). Both Alice and Bob are highly-skilled, passionate about the company's mission, happy with their paychecks and go along well with their coworkers.

Alice prefers working in an office every day. She has short bike commute through a nice neighborhood, works on tasks where constant team collaboration is important, is energized by teamwork and enjoys the company of her colleagues. She lives in a tiny apartment with no room for a nice home-office setup, and has noisy neighbors to boot. Alice loves her work-life balance, is happy that the work is confined to the office, and has no need or desire to read or respond to emails in the evenings.

Bob's got an hour-long commute each way through rush hour traffic. Working in an open-space office, he's constantly interrupted, either explicitly by a coworker or accidentally by someone chatting or just something that catches his eye ever so often. Bob's work requires him to have deep focus for long stretches of time, so interruptions stress him out and make him less productive. He copes by arriving earlier or staying late to avoid rush hour and crowded office and by using noise-cancelling headphones. At home (in a quiet neighborhood), he's set up a cozy home office in the spare bedroom.

Michael knows that the best way for the customers to love Acme Inc products is to have high-quality people like Alice and Bob be happy and productive. He does need to keep an eye on expenses though, and that seldom-used big office space Acme rented on a long lease just before Covid is a constant thorn in the eye of Acme shareholders and board, and they're giving him grief for underutilizing it. Having risen from the trenches, Michael knows that both effective communication and focus time are important for the overall success of Acme, and is aware of the related challenges in both office and remote setups.

How should Michael organize the work so that both Alice and Bob are happy and that Acme continues to be successful?

The first choice that Michael faces is between finding an office/remote balance, or going all-in and convert Acme to a fully remote and geographically distributed company, with no offices and employees across multiple time zones. Aware that brings a whole other sets of challenges[1], Michael leaves that option to a future thought experiments, and chooses to find an hybrid office/remote setup.

(All-in) One way or another

Can Michael just choose one and make the choice palatable to the person preferring the other?

If he chooses full return-to-office, Alice is happy and Bob is miserable. Michael could pay Bob more so he relocates to somewhere closer, but Bob really likes his house. He could provide a separate office, with a real door, and get an earful (or worse!) from the shareholders about the inefficient use of office space. He could try to enforce quiet office (library style) and schedule meetings so that Bob maximizes focus time, but with hundreds of people with calendars to sync he knows that's a losing game. Also, Alice would be less than happy for Bob to get paid more, or have a separate office, or for her calendar to get whacked, just because Bob has different preferences. And Bob might be okay with the arrangement, but he'd still prefer his nice house and cozy office if he had a choice.

If Michael chooses “work from home” for everyone, Bob is happy and Alice is miserable. Michael could pay Alice more to improve her home-office setup or get a bigger apartment, but that might come with a longer commute, worse neighborhood, or both. Now she'd need to wear noise cancelling headphones the entire day and spend hours on Zoom. Worse, since everyone works from home, people naturally take advantage of extreme flexibility of working hours, and as a consequence there's always someone working and pinging her throughout the day[2], wreaking havoc on her work-life balance. In this scenario, Michael also gets pressured into getting out of that expensive office lease, as well as tapping remote talent. In due time Alice is laid off (too expensive! doesn't work well in a remote setting!), workers get hired from remote locations (cost benefits!) and Acme transition into fully distributed company.

Free-for-all

Suppose Michael chooses “work from wherever”. Bob can work fully from home, Alice comes to the office every day, and for a time all seems good.

Yet Alice is frustrated by half her team being away and still needing to hop on Zoom calls multiple times a day. They also now take advantage of their flexible schedule, so some work evenings. As a consequence, communication is worse, slower, and Alice needs to stay in the office longer as some meetings are pushed to later in the day. She tries to schedule “sync” days where everyone would be in the office, but her WFH coworkers have different preferences for when that'd be. And anyways it's a half-measure, because ideally their team should touch base every day.

At first, Bob is happy. However, constant Zoom calls to “sync up” and people trying to get him to come to office for meetings bug him a bit. It looks like his coworkers are unable to properly convey their thoughts in text or other async-friendly way and want to have a “quick call” for things that could have been 2 Slack messages. To make matters worse, the in-office crowd is all in the same room for the conference calls, he can barely hear half of them and has less chance to jump in the discussion due to network lags, so he becomes less involved and more passive on such calls. As time goes on, Bob also has a nagging suspicion that his manager has less insight into his challenges or accomplishments than of his in-office coworkers, and that his recent lack of promotion/rise might be connected. This deadens his passion for the mission of Acme, Inc. – it becomes just another job.

Mandatory office days

Trying to balance flexibility and communication needs, Michael instead chooses “X days in the office, 5-X days work-from-home” policy. There's two ways he can go about doing that.

The first option is to fix the days. For example, everyone must come to the office on Mondays, Wednesdays and Fridays, and has an option to work either in-office or from home on Tuesdays and Thursdays. This allows him to mandate that all meetings must happen on Mon/Wed/Fri, since everyone's physically present on those days.This makes Alice happy because she can continue to go to the office every day, and just needs to schedule her meetings on the “in-office” days. While it's better for Bob than going to the office every day, he's not terribly happy because of the obligations in his private life he'd really prefer having those days more flexible and going to the office on Tue/Thu instead. Also, since there's still the same number of meetings but now two less days to use, Mon/Wed/Fri are more packed, leaving smaller blocks of focus time, making him totally unproductive for focused work and effectively halving his productivity.

Finally, this setup makes the Acme Inc. shareholders angry, since there's no way to downsize the office. It still needs the same capacity but sits half-empty half of the week.

The second option is to have everyone choose the days they'll come to the office. Alice comes in every day, while Bob picks the days he's most comfortable with. The trouble is, having everyone in each team agree on the days is nigh impossible. The office looks half-empty all of the time, every day they come to the office some different parts of their team is missing. As a result they resort to doing the meetings on Zoom (with half of the team from the conference room). At least once a week someone's mic stops working or they're near a construction site, turning meetings into “hello can you hear me now?” quagmire. On top of this, Acme, Inc. downsized the office (to the delight of shareholders who got a fat dividend) and introduced hot desking. Bob and Alice don't actually have a desk any more, and they need to reserve one each time they come to the office. Neither are happy with the situation.

Us vs. Them

Can Michael mandate “X days in the office, 5-X days from home” policy, and let each team fix their in-office days? Maybe.

Each person in each team might still have different preferences for office/home days, but in theory there's more chance that they might be more compatible, or make it easier to compromise. Since Michael can't simply organize Acme teams according to employee's WFH preferences, whether that works will be left to chance. So Alice might get stuck in a team that wants to be as remote as possible, or Bob in a team that's all in-office or everyone prefers exactly the days he doesn't. Furthermore, the communication between teams will be harder, since they'll be more isolated from each other. It becomes easier than ever to blame the other team for delays or misunderstandings. And if Bob or Alice are team leads or managers of their respective teams, they'll likely need to accommodate their schedule to be able to sync with the rest of the company.

If Michael can pull it off, gets lucky with employee preferences within each team, and can still ensure effective cross-team communication, this could work. That's a pretty big “if”, though.

One day to meet them all

Going back to the “X days in office” mandate, what if Michael chooses one day everyone must be in office, and mandates all meetings must be scheduled only on that day?

Alice can still come to the office each day. Since her work requires her to communicate with her team constantly, she's not too happy with one-day-for-meetings mandate. Her Meeting day is now packed full with back-to-back meetings, as is everyone's so there's a bunch of schedule conflicts. The office has nowhere near enough meeting rooms so groups huddle in different corners of bigger offices. Even if Alice loves teamwork, a full day of this is really exhausting and she wonders if the 2nd half was useful at all. Also, since her work requires the communication to happen more often, she must rely on Slack, mail and impromptu 1-1 calls throughout the week, slowing her down and frustrating her.

Bob is happier, since he now gets large uninterrupted blocks of time for the rest of the week. Yeah the full-day meetings are mind-numbing and he powers through, but at least there are no small chunks of time to waste in between. Also, much prefers the once-a-week commute to everyday commute. While it's not his preferred day, it's still better than having to go multiple times a week!

On the other hand, the shareholders are up with pitchforks. Acme. Inc pays through the nose for the office and they only use it one day per week? This puts enormous pressure on Michael to choose a different model. The fact that the office needs to be remodeled with more adequate meeting places adds to the pain. Michael also has a feeling that one touch point a week might be too infrequent, and is slowing his company down.

You choose, you lose

It seems like there's no winning combination. Whatever Michael chooses, someone will be unhappy.

Although Alice and Bob in this thought experiment embody the extremes, in any large company most people will have some mix of their traits and preferences, and there's no one solution that will satisfy everyone. This is before taking into account that the real executives might be less rational and will certainly have more constraints than our hypothetical Michael.

This realization should be the starting point of every RTO/WFH debate. Ideally, every organization would think through a similar thought experiment, identify realistic options and their consequences, taking people preferences and other constraints into accounts, and see how it can optimize its processes to make one of the options work well for it.

Fewer, more impactful meetings combined with more effective async communication might be one thing. Private per-team offices and allowances for setting up home-offices might be another. There are probably many more such tweaks an organization can make to make whichever choice more palatable to everyone.

I admit, in a cutthroat corporate capitalism world, this sounds like a pipe dream. But hey, one can dream, can't they?


[0] Why should knowledge workers deserve such a perk, when blue-collar, emergeny and many other workers working in worse, possibly dangerous, conditions, and for worse pay, can never get the same? As long as WHF for knowledge workers doesn't negatively impact the others, I don't believe that's a valid argument: people in every profession naturally try to get the best possible work conditions. Knowledge workers are no exception. “Why should you have the WFH option when they can't?” doesn't help at all to advance a (valid) quest to live in a more equal world.

[1] Collaboration across time zones, effective communication with team members from different cultures, wages for people in wildly different cost-of-living places, to name a few.

[2] Alice can always choose to ignore any pings after her working hours, but takes more mental effort and stress than not receiving them in the first place, even if nobody expects her to answer (which, if hours are entirely flexible for everyone else, is by itself a tall order). And that's even before we consider weekends.

Long time ago, information used to be hard to find. You had to go out there and look for it. Nowadays, a lot of information is at our fingertips at any moment.

Now we have the opposite problem: too much information. The challenge today is curation: we want to consume a reasonable amount of information that's relevant to us.

What's “reasonable” and what's “relevant” is tough to define as it is person-specific. What's reasonable and relevant to you may not be the same as what's reasonable and relevant to me.

From curation ...

Curation of information is nothing new: newspapers, radio and TV have done it since the beginning. We relied on the media to pick what's most important and only kept the choice of a source we trusted or liked.

As the Internet grew, the amount of information grew exponentially. One of the reasons for Google's early success is that it was very good at curating this information. While the other search engines could also find 10,000 pages matching your search, Google was the best at picking what was most useful, informative and relevant.

Other services were also trying their best at curation: Netflix had a great recommendation algorithm, and have for years organized a public competition to optimize it and provide even better recommendations.

.. to filter bubbles

Over the years, as the various algorithms got improved and optimized, they got subjectively worse for us, the users. Google is in arms-race with SEO folk, Facebook prefers to serve you content that will rile you up, Netflix' algorithm seems braindead and the less we talk about YouTube recommendations the better.

Why is this? Turns out that, at some point in their lifecycle, big companies had to choose between optimizing for the user experience and optimizing for revenue. The algorithms improve allright ... but using a different metric.

Google now hyper-optimizes for what it thinks you should see (filter bubble), Facebook serves you whatever will keep you on the site for longer (mind-numbing memes interspersed with viral outrages), and Netflix brings up and center its own content that it would like you to get hooked at (and not cancel the service), not the things you might actually want to see.

Curation for the benefit of the user, has turned to serving targeted content, for the benefit of the company.

Enshittification

Enshittification is an ugly word to describe an ugly thing. Basically, it is a strategy change in an online platform where it switches from user growth (where optimizing user experience is paramount) to revenue growth (where you need to squeeze maximum income from each user).

Enshittification, or platform decay, is not limited only to Internet media companies. But it is especially visible in online startups that grew big by giving away their product or undercharging for it, in order to grow as fast and big as they can.

At some point you got to earn money, and you have to earn as much money as possible, and since you have a mostly-captive audience (there is only one Google, Facebook, YouTube or Twitter), they won't leave if the user experience is marginally worse.

And so the UX frog gets slowly cooked. People today are hooked on Facebook scrolling, Twitter mob rages, memes everywhere and 10-second dopamine hits on TikTok or YouTube Shorts. It's the sugar thing all over again.

Information detox

Faced with this, some people try limiting their consumption of this kind of content (me included, see my Junk Social and Digital hygiene posts).

This can work if you're willing to limit your information access, don't suffer from the fear of missing out (FOMO) and have the mental strength to not succumb when you're tired, bored or just open up your favorite social media app on autopilot because it's on your mobile phone home screen.

But it's hard work: you're basically building and maintaining a Great Wall between yourself and most of the modern Internet.

Curation on our terms

Can we do it better? Is there a way to again outsource curation of content that is optimized for us, the users?

Curation is hard work, whether you want to build and maintain sophisticated algorithm or AI to do it, or if you have actual people doing the work. And it's not entirely obvious how that would work without degenerating into the filter bubbles and social media we have today – that's how some of those companies started, after all.

As a technologist I do believe a solution is within a realm of possiblity. As someone who's watched Internet grow from an academia and hobbyst garden into what it is today, I am skeptical we'll reach that solution (and if you really want to be scared about the prospects of it, go read The Master Switch).

A creative robot

Any sufficiently advanced technology is indistinguishable from magic.

Modern AIs and ChatGPT in particular look like magic to many people. This can lead to misunderstanding about their strengths and weaknesses, and a lot of unsubstantiated hype.

Learning about a technology is the best antidote. For curious computer scientists, software engineers and anyone else who isn't afraid of digging a bit deeper, I've compiled a list of useful resources on the topic.

This is basically my reading/watching list, organized from more fundamental or beginner friendly to the latest advances in the field. It's not exhaustive, but should give you (and me) enough knowledge to continue exploring and experimenting on your own.

General overviews

If you don't have a lot of time or don't know if you want to dedicate effort in learning the ins-and-outs of modern AIs, watch these first to give you a general overview:

Fundamentals of neural networks

The videos here provide both teorethical and hands-on introduction to the fundamentals of neural networks.

MIT Introduction to Deep Learning

A good theoretical intro is MIT's 6.S191 class lectures, especially the Introduction to Deep Learning and Recurrent Neural Networks, Transformers and Attention.

These overview lectures briefly introduce all the major elements and algorithms involved in creating and training neural networks. I don't think they works on their own (unless you're a student there, do all the in-class excercises, etc), but it's a good place to start with.

The topics discussed here will probably make your head spin and it won't be clear at all how to apply them in real life, but this will give you the lay of the land and prepare you for practical dive-in with, for example, Andrej's “Zero to Hero”.

The example code slides use TensorFlow. Since Andrej's course uses PyTorch, going through both sets of lectures will expose you to two most popular deep learning libraries.

Neural Networks: Zero to Hero

An awesome practical intro is the Neural Networks: Zero to Hero course by Andrej Karpathy (he also did the busy person's intro to LLMs linked above).

Andrej starts out slowly, by spelling out the computation involved in forward and backward passes of the neural network, and then gradually builds up to a single neuron, a single-layer network, multi-layer perceptron, deep networks and finally transformers (like GPT).

Throughout this, he introduces and uses tools like PyTorch (library for writing neural networks), and Jupyter Notebook, and Google Collab. Importantly, he first introduces and implements a concept manually, and only later switches to a PyTorch API that provides the same thing.

The only part where things look a bit rushed is the (currently) last – “Let's build GPT from scratch”. There's so much ground to cover there that Andrej skips over some parts (like the Adam optimization algorithm) and quickly goes over the others (self-attention, cross-attention).

The latest video in the series (not yet on the site as of this writing) is Let's Reproduce GPT-2, which can be also viewed standalone. In the video, he implements the full GPT-2 model (as described in the original paper), using PyTorch. By training on a newer, higher quality dataset, the model even approaches GPT-3 level of intelligence!

Overall a great guide. You only need to know the basics of Python, not be afraid of math (the heaviest of which is matrix multiplication which is spelled out), and do the excercises (code along the videos) without skipping the videos that don't seem exciting.

Understanding embeddings

Both the MIT and Andrej's lectures touch on embeddings (the way to turn words into numbers that a neural net can use) only lightly. To deepen your understanding, What Are Embeddings by Vicki Boykis will teach you everyhing (and I mean everything) about embeddings.

If you don't want to read an entire book but still dive deep, Illustrated word2vec article explains word2vec, a popular word embedding algorithm, step by step. It also features a video explanation for those that prefer it to text.

Another good lecture on the topic is Understanding Word2vec.

CNNs, autoencoders and GANs

The MIT lectures mention earlier also contain lessons on Convolutional Neural Networks, autoencoders and GANs, which are important building blocks in neural networks used in vision.

Again, these are high level overviews and although formulas are present, the lectures more give an overview of the algorithms without going into too much detail. That makes them ideal prequel to the Practical Deep Learning course by Fast.ai.

Diffusion models

Diffusion models build on top of CNNs to create image-generating and manipulating AI models. Beyond the general overview linked earlier, the Introduction to Diffusion Models for Machine Learning is a deep dive into exactly how they work.

Coding Stable Diffusion from scratch in PyTorch is a hands-on implementation of the stable diffusion paper, similar in style to Karpathy's.

Practical Deep Learning

Practical Deep Learning is a free course by Fast.ai that has (current count) 25 lectures covering both high-level practical parts of neural networks and the underlying fundamentals.

In particular in Part 2 they cover “zero to hero” on Stable Diffusion, a powerful image-generation AI model.

Large Language Models

These resources go in-depth about constructing and using large language models (like GPT):

Transformers

Andrej's course goes over the transformer (building blocks of GPT) architecture, but the complexity makes it easy to get lost at first pass. To solidify your understanding of the topic, these two are super useful:

The Illustrated Transformer describes the transformer (building blocks of GPT) in detail while avoiding tedious math or programming details. It provides a good intuition into what's going on (and there's even an accompanying video you may want to watch as a gentler intro).

Follow that up with The Annotated Transformer which describes the scientific paper that introduced Transformers and implements it in PyTorch. Since it's 1:1 annotation of the paper, you need a lot of understanding already so only attempt going through this once you've watched both Andrej's course and once you've read and understood the Illustrated Transformer.

Reinforcement Learning through Human Feedback

Language models are good at predicting and generating text, which is different from answering questions or having a conversation. RLHF is used to fine tune the models to be able to communicate in this way.

Illustrating Reinforcement Learning through Human Feedback from folks at HuggingFace (an open source AI community) provides a good overview of RLHF. They also did a webinar based on it (video is the complete webinar, link jumps directly to start of RLHF description) based on the blog post.

If you want to dive deeper, here's the InstructGPT paper from OpenAI, which basically describes the method they used to create ChatGPT out of GPT3 (InstructGPT was a research precursor to ChatGPT).

Fine-tuning

Fine-tuning allows us to refine or specialize an already (pre)-trained LLM to be better at a specific task (RLHF is one example).

Sebastian Rashka's Finetuning Large Language Models explains a few common aproaches to finetuning, with code examples using PyTorch. He follows that up with Understanding Parameter-Efficient LLM Finetuning, a blog post discussing ways to lower the number of parameters required, and an in-depth article about Parameter-Efficient LLM Finetuning with Low-Rank Adaptation (LoRA).

While fine-tuning excels at making the model behave differently, RAG (Retrieval-Augumented Generation) is often a better choice for giving additional information or context to the LLM. A good overview of use cases for both technologies is A Survey of Techniques for Maximizing LLM Performance by OpenAI.

Agents

Agents are LLM-driven components that can use tools (for example access the web) and communicate with other agents in the system to collaboratively solve a complex problem. Here's a good overview of agentic workflows by Andrew Ng, followed by a deeper dive by Harrison Chase (both videos from the Sequoia “AI Ascent” conference).

Full courses

If you want a really deep dive (undergrad or higher level), follow these courses including doing the excercises / playing around with the code:


This is a living document (ie. it's a work in progress and always will be). Come back in a few weeks and check if there's anything new.

ChatGPT and other AIs are all the rage and I see many (mostly junior) programmers worrying if the market for devs is going to dry up.

You don't need to worry, and here's why.

ChatGPT is scarily good. It really is. No it's not better at web search than Google (yet), and it's nowhere near being sentient. But for the tasks where it makes sense, it's very good. So I'm not going to tell you that you don't need to worry because it's a useless tool.

ChatGPT (and other AIs) is a tool, in the same way a calculator is a tool or a compiler is a tool. The word “calculator” used to refer to people doing number crunching. My first calculator was a handheld device. Now it's just an app. Yes, calculators, the people, lost their “job” doing mind-numbing number crunching, but they were able to work on more interesting problems in math, physics, or what have you.

Same with compilers. When compilers were invented, people were furious that someone thought a machine could do a better job than an expert programmer at crafting assembly code. Today almost nobody writes directly in assembly, except in rare cases where that makes sense. But “assembler programmers” didn't lose their jobs, they just became “C programmers” or “Lisp programmers”.

It is the same with the new AI models. They are very effective on a whole other level than just number crunching or compiling software, but at the end of the day they're just tools, like programming languages, libraries or APIs.

If you've been a programmer for more than a few years, you know you always need to learn new stuff and stay on top. You'll need to invest some time to learn what ChatGPT and others can do, or can't. What you don't want to do is completely ignore the trend, or (equally bad) think it'll solve all your problems (or put you out of business). And as John Carmack said in a recent tweet keep your eyes on the delivered value and don't over focus on the specifics of the tools.

I've been a programmer for some 30 years (20 or so professionally) and my usual reaction to new stuff is “oh, so they're reinventing that particular wheel again”. Yet the current crop of AIs gets me really excited, like I was a kid again uncovering the vast potential of what you can do with a machine that you can order around! I've been playing with ChatGPT and it makes me faster and it makes programming (more) fun!

Not by writing my code – I don't use it like that because it does produce bugs and it can hallucinate stuff. I use it to explore (how I might go about doing X), and to quickly recall something I forgot (how a particular API or library function is used, for example). As Simon Willison (of Django and Datasette fame) puts it AI-enhanced development makes me more ambitious with my projects.

This is just a beginning, and we're just seeing a boom in integrating these AIs into everything else. Copilot, Bing search, are the big names but people are experimenting with integrations with anything under the sun (I made a service that creates API backends based on your project description, for example). Time will tell which of these will be truly useful, but I have no doubt there will be a lot of them.

AI is a tool, with limitations, but with a lot of potential. It would be a shame not to use it effectively. It won't put you out of a job, it will make your job better.

As a long-time Hacker News reader, I've seen trends come and go. One of those trends is popularity of programming languages, manifested through the number of stories referencing a language.

Examples include “Building a Cache in Elixir”, “Helix: Neovim inspired editor, written in Rust” or “Text formatting in C++ using Libc++”.

Since these “in $LANGUAGE” articles come up fairly often, I was curious to see what we can glimpse about programming language popularity just by counting them. This is easy to do using HN Search API provided by Algolia.

Photo by Thomas Tastet @ Unsplash

Methodology

I used it to search for stories in the past year, where the phrase “in $LANGUAGE” appears in the story title. The search included C, C++, C#, Clojure, Dart, Elixir, Erlang, F#, Go, Haskell, Java, JavaScript, Kotlin, Lisp, Lua, PHP, Python, Ruby, Rust, Scala, Scheme, Swift, TypeScript, and Zig languages.

I also included Racket Scheme and added its numbers to Scheme. I'd have included Common Lisp as well, but there were no stories mentioning it explicitly in the title in the past year. Also, since C would match both C++ and C# results in this simple string search, I deduplicated the results to get the correct count.

I ranked the languages in three ways: by the number of stories, number of comments, and sum of points. The results were fairly consistent across all three metrics.

If you're interested in running this yourself, this is the script I used (you'll need the requests package to run it). It downloads and caches search results as JSON in /tmp/hn, sorts the languages, and prints the results.

I ran the script on October 12th, 2022.

And the winner is ...

... Rust, by a wide margin. Here are the full results:

By the number of stories:

  1. Rust (573 stories)
  2. Python (375 stories)
  3. Go (332 stories)
  4. JavaScript (200 stories)
  5. C (143 stories)
  6. C++ (136 stories)
  7. TypeScript (79 stories)
  8. Java (76 stories)
  9. Ruby (57 stories)
  10. Elixir (49 stories)
  11. Haskell (48 stories)
  12. Swift (41 stories)
  13. C# (32 stories)
  14. PHP (31 stories)
  15. Zig (27 stories)
  16. Clojure (23 stories)
  17. Scheme (13 stories)
  18. F# (12 stories)
  19. Kotlin (9 stories)
  20. Erlang (8 stories)
  21. Lisp (8 stories)
  22. Lua (8 stories)
  23. Dart (4 stories)
  24. Scala (4 stories)

By the number of comments:

  1. Rust (6384 comments)
  2. Go (3394 comments)
  3. C (3027 comments)
  4. Python (2783 comments)
  5. JavaScript (1583 comments)
  6. C++ (771 comments)
  7. Zig (735 comments)
  8. Java (615 comments)
  9. Ruby (396 comments)
  10. TypeScript (341 comments)
  11. Haskell (301 comments)
  12. PHP (223 comments)
  13. Scheme (177 comments)
  14. Swift (165 comments)
  15. C# (151 comments)
  16. Elixir (133 comments)
  17. Clojure (82 comments)
  18. Lua (75 comments)
  19. Lisp (53 comments)
  20. F# (45 comments)
  21. Erlang (39 comments)
  22. Kotlin (7 comments)
  23. Dart (1 comments)
  24. Scala (0 comments)

By points:

  1. Rust (16887 points)
  2. Go (7144 points)
  3. Python (7048 points)
  4. C (5422 points)
  5. JavaScript (3644 points)
  6. Zig (2427 points)
  7. C++ (1846 points)
  8. Java (1595 points)
  9. Ruby (1044 points)
  10. TypeScript (998 points)
  11. Haskell (846 points)
  12. Elixir (751 points)
  13. Scheme (549 points)
  14. Swift (447 points)
  15. PHP (402 points)
  16. Clojure (365 points)
  17. C# (334 points)
  18. F# (331 points)
  19. Erlang (322 points)
  20. Lua (238 points)
  21. Lisp (154 points)
  22. Kotlin (34 points)
  23. Dart (14 points)
  24. Scala (7 points)

Hacker News loves Rust

Go, Python and JavaScript are quite popular, but Rust absolutely blows the competition out of the water. It had almost as much stories and comments as the languages in second and third place combined, and was awarded even more points.

If you're a regular HN reader, this isn't surprising. As a modern language with passionate community, Rust is invariably mentioned in most conversations about languages in general, and stories about using Rust for a new project or in a new are always exciting.

And I'm not even counting the “Rust in the Linux kernel” theme, which alone got around 600 points across various stories in the past year.

The new kid on the block is ... Zig

What did surprise me is the relative popularity of Zig, a fairly young language with still a relatively small comunities – I'll be sure to keep an eye on it.

(comment on Hacker News) | (follow me on Twitter)

As a software developer with many years of remote and in-office experience, I am excited by the current shift that's making remote and hybrid work commonplace.

The pandemic was a global tragedy and a big disruptor to every aspect of the lives of people everywhere. One of the few silver linings was that it forced a rethinking of the importance of location for the knowledge worker and what “office” actually means.

In my two decades of professional experience, I spent about half of it working remotely. The other half we had a small office (10 to 15 people), but mostly worked with clients overseas. In my time as both an individual contributor, and manager of dev teams, I had to handle the challenges arising from, and enjoy the advantages of, remote work.

a digital nomad in its natural habitat

My view is naturally colored by my experience. I am a software engineer and have worked in IT my entire career. While I believe most of what I say here can be generalized, there are nuances for every profession.

Location, location, location

Let's get obvious out of the way. Not all work is – or will be – possible to do remotely.

Although technology will continue to push the limits (remote truck driving, anyone?), many jobs, from surgeons to shopkeepers to street sweepers, still require people to be in a certain location to do their job. This does not mean those jobs are lesser, or that they will lose out in the “remote work revolution” – quite the opposite. Fewer people commuting to work and crowding in hot locations means less traffic, less stress and more room for those that choose to – or have to – be there.

Bare necessities

My job as a software developer consists mostly of reading from a computer screen and typing on a keyboard, with the occasional use of a mouse. I can do that anywhere on the planet, provided there is electricity and an adequate internet connection. In reality, what I consume (as a work input) and produce (work output) is information, and that's easily sent across the world.

The same holds mostly true for most knowledge workers. When push came to shove (the pandemic lockdowns), a lot of people realized the work they do can be performed remotely.

Hierarchy of needs

Of course, that's not all there is. We still need information, often implicit, from our coworkers, about the problem we're solving, and the wider context our company, customers, and clients are operating on. Much of this is recorded poorly, or not at all. Remote interactions are harder than face-to-face and Zoom fatigue is real. And sometimes we need specialized office or lab equipment.

And as much as we love working in our pajamas, we might miss office chit-chat or getting that energized feeling from being a member of a well-oiled team of awesome people doing great work together.

Back to life, back to reality, back to the here and now

Now that things are normalizing a bit (I know it's a stretch to use that word in the middle of 2022), companies are starting to mandate back-to-office or hybrid work. Conspiracy theories about reasons are many, from wanting to micromanage workers better, to making sure expensive office leases are utilized, to being a sneaky way to lay people off without actually giving them the pink slip.

However this is resolved, the genie is out of the bottle. Remote work has previously been regarded as a unique perk or very specific work arrangement. Now, wherever it's allowed or not in a particular company, it's normal. I think that's a good thing.

Checking our privilege

Us IT folks have it easy. The job market is hot and there are plenty of opportunities for good engineers. Yes, there are hiring freezes and some companies are laying off people. In most of the cases I heard of, people were quickly snatched by the competitors.

This puts us in a position to demand the ability to work remotely, and if needed, to quit and join a company that will allow that. Not everyone is so lucky and there are legions of knowledge workers who have returned (or will have to return) to the office just because the bosses decreed it.

In IT, and especially in software development, the shift is real, and companies need to adapt.

In-office vs remote work

There is a fundamental divide between in-office and remote work, and that is how the information within the company flows. Companies where in-office work is the norm can get away with much larger implicit context and rely on employees communicating directly ad-hoc, as needed.

To take full advantage, such companies maximize the overlap between the employees' working hours (9 to 5 everyone!), and try to cluster employees together (if not all the company can be in the same location, at least the teams working together) and design office space to optimize for interaction (leading to unfortunate things like open offices).

Remote-first companies focus on async work (work on your own time, communicate asynchronously with coworkers) making it possible to hire globally. Instead of spending on office space, they give workers budgets to improve their workspace or equipment. But most of all, remote-first companies take communication at all levels seriously. From office banter to all-hands meetings, stuff is happening online and is often saved or recorded for the benefit of people who are not online at that very moment.

It's easy to contrast these two extremes, but many companies will probably lie somewhere in between.

Old-style hybrid work

“Hybrid” is a word that describes two different work arrangements, depending on where the company started from.

For traditional office-based companies that are introducing hybrid work, this usually amounts to a perk allowing employees to work from home a few days per week. Often, at least two days are fixed for meetings and any other work that must be performed on-site. This type of hybrid office can easily slip back into the “office is king” mentality, where the important things still happen in the office, not everything is shared with remote workers (not necessarily due to any malicious intent), and the more you're present, the faster you'll career trajectory will be.

This was mostly the status quo for remote workers before the pandemic. Except in rare enlightened companies, remote workers were de-facto second-class citizens. The more a company now stresses that being in-office is important, the more likely it'll fall back on this pre-pandemic default.

I believe that companies that are now requiring employees to be present at least three days a week will most likely fall back to this mentality.

If they combine this with hot-desking, where employees must find themselves a (possibly different) desk each day, it will only make things worse. I fear this may be a real possibility for many, to try to have the workers back but still save on office space.

Remote-first hybrid work

Another type of hybrid-location company is remote-first which recognizes there is benefit in getting people together or making it easy for people who want to work from the office to do so. These companies understand office is just another location. Hopefully, the environment is optimized for work (quiet environment, good equipment, etc) but in terms of work communication, it holds no special place.

A company I currently work with has such an approach. They're NYC-based, but have employees throughout the US, as well as people in Latin America and Europe. The main communication channels are all digital – Slack, Zoom, Notion, and email. Scheduling Zoom meetings across time zones is a bit challenging, but they manage and are looking to improve as they grow.

From remote-as-perk to remote-first hybrid

There's a tremendous opportunity for companies that are willing to switch from the office-based (or remote-as-perk a few days per week) model to the remote-first hybrid model. They can still make use of their existing office space (possibly downsized to reap financial benefits), reconfiguring it so it is support for people who want to work from the office, or who need to come to the office.

Essentially, they can turn their offices into a sort of internal coworking space. Employees that don't want or can't work from home can still come to the office. Coworkers or entire teams can independently or ad-hoc agree to come to the office for a day or a week for brainstorming sessions, project kick-offs, or just to socialize from time to time.

Remote-first, but not remote-always

In a few places I worked as a remote developer, companies had policies to bring everyone on the team together for a few days. This was sometimes combined with everyone going to a conference but hanging out in the off-times, or for everyone to be in the same place to start a new initiative. This was always a fun experience (even when paired with hard work) and I felt energized and ready to take on new challenges.

I believe fully remote companies should try to bring their people together at least once or twice per year. This can be company-wide (for smaller organizations) or per-team (easier to do with larger companies). If the company is fully remote and doesn't have an HQ, doesn't matter: pick a nice location and if there's a conference or another interesting event, that's even better.

Remote vs overseas

A lot of companies these days say they hire remotely but within a country. For example, a job listing may say “remote, US-only”.

This is understandable: it's legally easier to hire within the country you're incorporated in, the cultural differences are smaller, and time zone difference is more manageable. it can also be harder to find remote talent since you don't know where or how to look. The stigma of hiring overseas developers as “outsourcing to cheap labor” doesn't help.

Cultural differences are real and should not be ignored. Being cognizant of the different ways people communicate (or avoid difficult subjects) is important. I've found that (respectfully) overcommunicating helps, as it reduces the potential for confusion. Time zone differences can have a larger or smaller impact, depending on whether your team's communication is more synchronous or async. If you have daily Zoom meetings and your team is spread over Australia, Europe, and the US, you'll have a lot of unhappy people. Somewhat localizing the teams, and making most communication asynchronous help.

Remote doesn't mean cheap labor

If you pay peanuts, you'll get (code) monkeys. Good developers anywhere know what they're worth. That said, the differences in cost of living are huge, and people do take into account the (in)convenience of living anywhere. $100K means a very different lifestyle to someone in San Francisco than to someone in Vienna or Manila.

Developers are ambivalent toward location-based salaries – on the one hand, we'd prefer to be paid what we're worth (based on the value we provide), not based on where we live. On the other hand, we don't accept that there are talented and skilled people that can provide the same value for a slightly (or more than slightly) lower price, in lower-income countries.

If companies were completely oblivious to someone's location, it would be natural for them to want to pay a lower price for the same value, so they'd hire in lower-income locations (within a country or overseas). But they're not. They do prefer local (or nearby) employees, and they are willing to pay a higher price for that. A direct consequence of this is that at least some portion of the salary is going to be location-based.

As knowledge workers, we should get to terms with that fact. We are paid based on the value we provide, but not in isolation, and the job market will always have an impact.

The future is already here – it's just not evenly distributed

The pandemic turned remote work from a fringe benefit to a normal, accepted, way to do your job, in cases, companies and jobs where that makes sense. The technology will keep pushing the boundaries where that's possible, and the knowledge workers will keep pushing to have that option.

Companies will have to adapt: by doubling down on office (and paying extra for it), embracing the new opportunity to broaden the talent pool, or finding a hybrid sweet spot that makes sense for both them and their employees.

There's no going back.

Last month, after successful transition of our users to Miro, I shut down the public servers of A Web Whiteboard (AWW App), marking the end chapter of a decade long story.

Throughout, millions of users have collectively spent a millennia drawing, learning, collaborating, and just having fun with AWW. It's been used in schools, FAANGs and everywhere in between. Early in 2021 we joined forces with Miro, the world's leading online whiteboard platform.

The start of the story is much humbler. I never intended or even imagined it would be a success it has been.

Origins

Ten years ago, back in 2011, netbooks were all the craze. What started some six years earlier with EeePC turned into an entire product category with Cambrian explosion of options. I got my hands on one with a curious feature – a touchscreen. At the time, system UI wasn't at all suited for clumsy fingers instead of precision controlled mouse pointer.

So there I was, back in 2011, trying to find a simple, nice, drawing program for Linux that would work well, UI-wise, with touchscreens. I didn't find any so decided to write my own. I took this as an opportunity to learn HTML <canvas> and write it as a web app.

It was all pretty rudimentary. You had a pencil in a few colors, eraser, and ability to clean the entire canvas. This first version even lacked the ability to save the result – you could always screenshot the page if you wanted to save.

This wasn't even a side project at the time – just a fun experiment. For kicks, I decided to put it up online and, after thinking for a whole two seconds, named it A Web Whiteboard. Since aww.com was taken, I used awwapp.com for the domain (nameapp.com was a popular way of getting around the domain name squatters at the time).

Sharing

A few months in, someone mentioned it'd be cool if you could actually have multiple people drawing at the same time. I took the suggestion an opportunity to again dabble in new tech. At the time, websockets were just being entering mainstream use and Node was starting its meteoric rise, so I selected it for the synchronization backend and now all-but-forgotten RPC-over-sockets library called nowjs. Being primarily a Python developer, I kept to at least something familiar, using Flask for saving the board contents.

This is how it looked like:

First public version

I spread the word around, got some mention in the press and received positive feedback, including on Hacker News and Reddit. There weren't many users back then (the whole thing comfortably fit in RAM of a rather small VPS), but each time users gathered to draw together, or shared a saved drawing on social media, more people heard about AWW.

After the launch, I maintained AWW but treated it as a side hobby, and for a couple of years most of my focus was on other things.

Embedding

One thing I did from the start is organize AWW as whiteboard widget that could be embedded into other sites, with awwapp.com being just one of the sites that embedded it.

When I started getting requests to let people add it to their sites, it was easy to do. I set up a PayPal subscribe button and wrote up a simple doc on how to embed the widget. When someone subscribed, I'd add them to the accounts “database” (actually a JSON file) and gave them an unique API key to embed with.

To add whiteboard to their sites, my customers would include jQuery and add the following snippet:

<div id="wrapper" style="width: 800px; height: 600px;"></div>
<script src="http://static.awwapp.com/plugin/1.0/aww.min.js"></script>
<script>
    $('#wrapper').awwCanvas({apiKey: YOUR_API_KEY});
</script>

After a time, this netted me a few hundred bucks a month, which was nice but nothing I'd base my business on – or so I thought.

I also started getting more and more serious questions about using the whiteboard on awwapp.com for business, tutoring, etc, as well as feature requests. Most of my customers were tutors using it together with Skype to provide online lessons to their students. They often didn't have a site to embed the whiteboard on, were happy to use awwapp.com version and willing to pay for it.

The rewrite

As I started treating AWW as a viable business in its own right, I started thinking about rewriting it.

The old JSON file approach was getting unwieldy, as was the fact that all active board were being kept in memory, limiting growth. Finally, the old nowjs framework was unmaintained meaning we were stuck on an extremely old Node version (0.6 if I remember correctly).

I decided on a full rewrite with redesigned architecture and using modern (for the time!) JavaScript features – stuff like promises, the socket.io library and using code bundler tools. For database we selected MongoDB, primarily for ease of use and ease of clustering/failover. We also switched from a single server (and thus single-point-of-failure) to a load-balanced system with full failover (using haproxy for web server load balancing).

This was an explosion in ops complexity. Mostly warranted, but I definitely over-engineered some things, and hit many of the novice mistakes when scaling out.

We started the rewrite in the summer of 2014 and deployed the new version in spring of 2015. I say “we” because I had help – this was now a proper project internal to my dev agency with more people involved. However it still wasn't a priority and was done at off-times and between client projects.

The rewrite looked similar to the original

With the new version, we were finally able to sign up users directly on awwapp.com, again using PayPal subscriptions. We followed the freemium model, in which most of the functionality was free, but you could pay for some additionally useful features.

Ease of use

Throughout entire lifetime of AWW, I had one major focus, sometimes to the detriment of other aspects of the app: it had to be easy to use. Really easy to use. So easy a child could use it.

We did that successfully – in fact, many children did use it in kindergarten and preschool. It was great seeing pictures teachers shared of kids doing a collaborative class, or exploring maths using our whiteboard. “This is why I'm doing this”, I'd say.

For example, to start drawing on AWW you only had to click one button, labelled Start drawing, and less than a second later, you're drawing. And the only reason there was some text and a button to click first, was so people coming first time to AWW knew what the site was about! Before this, people coming up on an empty board were sometimes confusing what that was for.

Word of mouth

One thing we didn't do (except in a small experiment at the start with StumbleUpon) until much later, is promote or advertise AWW. In fact, I did hardly any outreach at all.

Virtually all of the (slow and steady) growth was entirely through word of mouth – our users telling others about us. This was helped by the fact that AWW is a collaboration platform, so users naturally wanted to share the drawing activity or results with others.

Combined with the ease of use and the fact most of the features were free, this made AWW a favorite whiteboard for many teachers and tutors. They were telling each other about it, featuring it in teacher seminars and doing video tutorials on it! This showed me how important and useful AWW was for a growing number of people and really warmed my heart.

Spinning off into a separate startup

Usage slowly but steadily grew over the years, to the point that there was simply too much to do on maintenance, customer support, inbound sales and further development, to treat AWW as a side-project.

I also realized that, as a technical person, I should get new people on board that will have “skin in the game”, a fresh perspective and complementary set of skills such as business development, sales.

I met Zvonimir at local startup event where he was just describing the challenges of his previous startup attempt. Zvonimir had tech chops and biz acumen in equal amount, combined with startup experience, but what I most liked was the levelheadedness when thinking about the realities of building (and selling!) a product in an uncertain environment.

We hit it off immediately and after some time I managed to convince him to join me as a cofounder. I consider this to be the best decision (and best sale!) I made throughout the development on AWW.

We spun off AWW as a separate startup in late 2016, incorporated it in the US using Stripe Atlas, brought several more people on board (a developer, a designer, and a marketing/support person) and set to work. We had three priorities:

The plan

First, to figure out who our users were, why they used us and what we can do to make us more useful to them. This involved identifying different markets we were playing in, such as EdTech, enterprise collaboration, etc. Up until that point we “lucked out” in that our product was useful – now we set to deliberately discover our customers and optimize our approach (feature set and messaging) towards them.

Secondly, to improve our user experience. AWW was simple and easy to use, but not so simple to figure out all the capabilities it had, or indeed why you'd want to pay for it. The more advanced “premium” functionality was hidden away unless you were a premium user, so many of our free users didn't even realize there was more to it.

The new UI

Also, as our features and supported use cases grew (and were expected to grow much further), bolting them onto the old interface made it more and more clunky over time. We set out to rethink the UI from scratch, still respecting our old imperatives (simplicity) but incorporating the lessons learned.

Third priority was to start working on our marketing, sales and business development, and to improve our customer support.

Up until then, customer support meant me answering email questions with a few days' delay. We aimed to lower the reply time to within a business day and to be active on more channels, such as Twitter and Facebook.

The online presence tied in to marketing. We didn't do any advertising, aside from a few experiments to test the waters and measure potential interest for various whiteboarding use cases. However, we did want to be present on Twitter, Facebook, LinkedIn, reaching out to blogs reviewing us, talk with teachers recommending us in workshops, and this tied in nicely with the customer relations part of the plan.

The funding

Now we were in business. We had several people and everyone worked exclusively on AWW, except me. I spent half my time on AWW and half managing my software development agency GoodCode.

Even though Croatian salaries are nowhere near US (and especially Silicon Valley) ones, having more people still meant more expenses. What was previously a profitable side project now turned into a startup that wasn't yet ramen profitable. I was funding the startup from the profits from my agency so we didn't need to worry about runway for the time being, but were still open to investments.

Our rough plan was to ramp up the product and get to profitability within twelwe months. Of course we hoped for much more – our valiant efforts will result in a hockey-stick growth and back to black in three months and then sky's the limit!

That, of course, didn't happen.

YC School

At that time we applied to several accelerators, including YC, but didn't get in. Though one of the reasons was for funding, our primary reason was the advice, learning experience, networking and fresh ideas.

We also applied to the first YC Startup School in 2017 and did get in! The online MOOC-style course combining online video lectures, weekly check-ins, discussions with mentors and our fellow founders were phenomenal. I imagine the intensity is nowhere near a “full” YC experience, but it was tremendously helpful for us.

YC Startup School Completion Certificate

First, the weekly check-ins did wonders for our accountability. We had metrics before and we did look at what we've achieved (or not) previously, but not in as critical and clear way we had to for Startup School. We soon found out our old non-actionable vanity metrics were pretty much worthless and that we didn't even have an easy way to get to the right numbers. Focusing on the real was like wiping your glasses (or the window) and actually seeing what was going on.

Weekly discussions were as valuable. We received (and gave out) thoughtful and probing questions which made us reflect on our implicit assumptions and ways of thinking. Here's an example: late in the batch, we were talking about monetization problem since the vast majority of our users were free users. We didn't want to turn cripple AWW free functionality but couldn't find a compelling reason for many of them to upgrade. Our mentor asked us point-blank: “I know you're against putting ads there, but why not give it a shot and see what happens?”

The ads

This wasn't new idea. This was pretty old idea that I shot down time and again because as a consumer I dislike ads, especially the creepy follow-you-around targeted kind, and I subscribe to the “if you're not paying for the product, you're the product” line of thought. I also thought ads would clutter useful screen estate – on a whiteboard, you want the whiteboard area to be as large as possible. So I always dismissed this idea without a second though, and even my cofounder didn't convince me otherwise.

But this question at the Startup School office hours got us thinking. Can we do it in a way that wouldn't screw up the experience? Perhaps. We didn't jump to it straight away but the idea kept simmering in the back of our minds.

Finally, in the spring of 2018, we gave it a go. The idea was simple: free users get ads. Subscribe and besides getting all the premium features, you also get rid of ads. The rationale was that a lot of people don't actually mind ads for free stuff and for those that do, well, there's an easy way to remove them. This sounds like such a simple concept but it took us a lot of time to internalize all the rationale and the consequences of this decision.

We implemented Google Adsense in early 2018 and after a few weeks of wildly erratic earnings, it settled down to a pretty low value, with a lot of traffic showing as invalid. To this day I'm not exactly sure what was the reason.

I suspect it was some combination of us testing the ads on only 10% of users to begin with, screwing up the ad placement implementation (there were initial app modal dialogs that were showing above the ads in the first few seconds), and ad rotation (our users were on the same page for extended period of time and showing the same ad for 15 minutes makes no sense).

One by one we fixed these perceived problems. It took very long time for the invalid traffic ratio to lower down (from 50% of ads shown to some 10% of total).

In the end, ads were bringing in about a fifth of our revenue, but they also increased our conversion rate (free to paying users), and haven't materially dented our growth (users leaving in frustration because of ads was my biggest fear).

Chasm

Late 2018 was a rough time for AWW. About two years of talking to customers, building, measuring, revamping the product and the UI didn't produce the hockey-stick we were aiming for. Not even the “reasonable” 4x-5x growth we expected at the least. We were growing – but very slowly.

What did we do wrong? Maybe we didn't focus narrowly enough (we still catered to EdTech, small business users, other apps embedding us and even had an on-premises enterprise offering). Maybe we weren't aggressive enough in customer acquisition (inbound sales only, no advertising). Me spending just half of my time on the startup certainly didn't help.

We had some tough decisions to make. After a lot of soul searching we decided to trim down, letting everyone go, with Zvonimir and me staying to work on it part-time. Most of our time we spent on customer relations, inbound sales and maintaining the service.

Without our noticing it, the service still grew, though. In retrospect, we did good work on a number of things back in 2017, it just took a lot more time for the effects to become visible. By the end of 2019 we approached triple the revenue numbers from 2017 (on a slower user growth).

Preparing to sell

Okay so it wasn't a rocket to the moon we hoped for, but AWW was still a nice little side business. Trouble was, both of us also had other businesses to attend to and couldn't keep up focus both on the new ventures and AWW indefinitely – something had to go. We also didn't want to shut it down, so we started thinking about selling it.

We researched and interviewed several online business brokers and decided to go with FE International, partly on the strength of patio11's recommendations.

Preparations for going to market involved us compiling detailed financials for the previous few years, as well as answering a questionnaire about the business itself, the customers, competition, reasons for selling, and so on.

After we provided this information, FEI prepared a go-to-market prospectus (a sales brochure), we okayed it and they started reaching to their network. This preparation took several months, so by the start of March 2020. we were on the market.

In two weeks, everything changed.

The world moves online

With the pandemic surging everywhere, much of the western world started shutting down or going online. Schools started adapting their curriculum for online, businesses went full work-from-home. Everyone was on Zoom, Teams, and Meet.

Every online productivity and collaboration tool imaginable started seeing their usage numbers surge. This also happened to us. Teachers, school district IT departments, businesses, started signing up in droves.

Our user numbers, customer numbers, number of shared whiteboards, revenue – basically every important metric – skyrocketed. We were up 6x (that's 600%) by end of April compared to mid-March, and the growth didn't stop there.

We were profiting from the situation, but also wanted to help. We set up a policy to give free premium access to any educational institution until the end of the school year. Besides being a decent thing to do, we figured (correctly) it would convince many schools to stay with us the next year.

The mad IT scramble

Our systems were woefully unprovisioned for the sudden increase in load. We started provisioning new servers left and right but soon found out there were bottlenecks in how the system was organized that prevented taking full advantage of horizontal scaling.

Examples were having many smaller (unsharded) MongoDB replicas in a set instead of using fewer larger-capacity servers, having database interconnect latencies that caused MongoDB queries to be dispatched unevenly across the servers.

Then there were the sloppy things that went unnoticed for years on a smaller load, such as not cleaning up Redis temporary keys quickly enough (leading to our Redis servers quickly using up any amount of memory we threw at them), using a few database queries where a single (complex) one might do the trick, or our indexes not matching the actual usage patterns.

And then there were the subtle bugs and edge cases. As the saying goes, if you have a million users, million-in-one bugs are a daily occurrence. Although AWW is designed to gracefully withstand shorter connection problems, this didn't always work flawlessly and sometimes even increased the problem by hammering the servers too much when they were already under heavy load.

Our platform providers had their own woes. With everyone buying or renting any capacity they could get their hands on, the platforms themselves become less stable. We were hosted on Digital Ocean and although they're in general great as a platform (and I have virtually all my other stuff there), we were plagued for months by strange intermittent issues where a system of ours would just lock up under moderate load.

Lots of investigation, together with the engineering team on Digital Ocean's side, yielded no specific results, except that issues were related to the “shared” type droplets (servers) we were mostly using at the time. Switching to “dedicated” droplets helped a bit, but didn't eliminate the issue. Whether it was something we did, or there was a just a capacity and noisy neighbor problem, we'll never know. In the end, we resulted to rebooting any droplets showing signs of being locked up.

Droplet rebooting

Growing the team

Everyone has their “what did you do when the pandemic hit” story. Mine was holed up in the attic-turned-home-office in front of a computer, madly trying to keep the system running, both excited and terrified. Days were a blur – I don't remember much of anything else until some time at the end of April.

When the pandemic hit, both Zvonimir and me had other work obligations. We couldn't just drop those at a moment's notice, so for weeks we put in required hours on those obligations and every other waking moment on AWW.

It still wasn't enough. Especially the support tickets (combining the questions about our service and signing up, and problem reports) were overwhelming. I remember powering through them late at night, going to sleep, waking up early morning (usually by some server alarm going off) and then seeing another few hours' worth of tickets already waiting.

It took us about a week to decide to grow the team, and then find, hire and onboard a dedicated support person. She was able to handle most of the questions from users while escalating to us stuff like bug reports, bizdev/sales questions and other more complex inquiries.

This finally gave us some breathing room and ability to focus on the business (Zvonimir) and tech (myself) matters. In the coming months we hired another support person, two developers and a part-time designer.

Growing the business

While I was busy scaling up our tech, Zvonimir spent most of his time on Zoom calls, talking to our customers and prospective customers. Whereas previously most of our users and customers were individuals, now organizations of varying sizes were knocking on our door.

We tweaked our bigger pricing plans to better match what people were asking us for. With bigger enterprises now knocking on our door, things like service-level agreements (SLAs) and standard operating procedure (SOP) documentation needed to be set up and addressed before bigger companies could roll us out to their teams.

All of this took enormous amount of time. Deals still took weeks or months from initial contact to implementation. All of it was inbound interest but there was a lot of work for every bigger deal. Had we had a dedicated smooth-running biz team, I'm sure we would have grown twice as fast or more.

No matter. Even with a small team, by end of 2020 we've grown 10x, and that includes lull summer months when most of the EdTech activity is subdued.

To sell or not to sell?

Our explosive growth started just as were about to put ourselves on the market. Almost immediately, the old numbers didn't make sense. Users, revenue, growth numbers were nothing like the ones used to calculate a (target) sale price.

In the next several weeks we did receive several offers that would've been interesting before the growth started, but didn't make sense in the new environment. We decided to put the sale on pause until the numbers became more predictable and for the next few months didn't think about it.

By mid-summer the growth stabilized – we were still growing, but at a bit more predictable pace – so we revisited the idea to sell, as we still had a (paused) contract with our brokers.

Based on our experience, selling via brokers is easier if there's a flat or consistent small growth where future can be easily extrapolated. Coming up with the price then involves tallying up income and expenses and multiplying with a factor you come up from experience and rules of thumb.

In our case, growth was triple-digit, virtually all of the expenses were investments into growth so this pricing model breaks down. We adapted the model, updated the numbers and went to market again. Zvonimir had by that time learned the nitty-gritty details of company valuation and had actually provided the final calculation himself.

Ultimately, we haven't received any credible offers for the updated price and since we weren't under any pressure to sell, we decided not to.

Miro

Earlier in 2020, Miro had raised $50M in a Series B funding round to fuel their growth. At some point their reached out to us to see if we could cooperate and by the autumn the topic turned to the possibility of them acquiring us.

In contrast to earlier potential buyers, Miro was in the same niche as we were, with a similar product and was also a technological company. We liked the team and their product and felt Miro could be a good home for AWW. Also, the price was right based on our revenue, users and growth numbers.

It took some time to negotiate the details of the deal and go through due dilligence process. We went over dozens of drafts for both LOI (Letter of Intent) and the actual purchase agreement – although we had a “gentleman's agreement” from the business perspective, there were still a lot of details to iron out.

While we obviously had our own lawyers, tax advisors and accountants (on both Croatian and US side!), we wanted to be hands-on and understand everything, so by the end of it both Zvonimir and me read cumulatively hundreds of pages of US contract legalese.

Due dilligence also got complicated: although AWW, Inc. was a US company, both Zvonimir and me were residents of Croatia and we had contractors from all over the place, not to mention the original intellectual property (IP) traced back to my development agency (Good Code) from the times it was just another project there. Although we had IP purchase paper trail, some of it was in Croatian and had to be translated, and the lawyers wanted to triple-check the rest.

Finally, the deal closed and on February 22nd, 2021. we publicly announced that AWW is joining Miro!

AWW joins Miro

Since both AWW and Miro use custom in-house built whiteboarding functionality, there wasn't much sense in continuing to develop both products, and Miro decided to wind the technology powering AWW down.

So we set out to migrate existing customers: for enterprise customers there was more work involved on our side, while smaller users received 1 year free equivalent subscription to Miro and guidance on how to export their AWW whiteboards and upload them to Miro. All users were given 5 months to do the transition, and we timed it such that education organizations were able to finish their school year on AWW.

Miro also took our experience and lessons learnt from AWW in designing and developing a free version, so although no AWW code is directly used, our focus on simplicity, immediacy and ease of use continues.

Looking back

If a time-traveler told me ten years ago how the story would unfold, I would laugh them off. What started as a hobby experiment evolved into one of the global leaders in its niche, was used by millions and changed our lives forever.

I like to think that in a tiny way we improved the lives of all our users – be it a kid doodling something in a collaborative drawing class, a roofing company making it easier for their customers to sketch what they need, or a developer being able to discuss software architecture on a confcall – we made those moments easier and more fun.

And hey, what more can you ask for.

Is software development a proper engineering discipline? On one hand, it looks like it should be – computer science is a rigorous, hard, math-based science. The hardware is deterministic.

As anyone involved with the discipline will attest, though, it's sorely lacking. Just Google for “if programmers made ...” or “if software engineers built ...” for the cheeky, but not entirely untrue, comparisons.

Why are we so bad at it? Is it because we're really struggling to be something we're not? Or are there other reasons we're failing?

Let's do a cheeky comparison of our own.

Bridging the gap

Suppose you want to build a bridge. Right off the bat, you know a few things:

  • which two places across the water you want to connect
  • how many lanes and of which type (car, train, bike, pedestrian) you want to support
  • terrain on both sides and the weather patterns
  • typical building materials for the bridge and their properties

Based on this, you can design the bridge architecture. It'll probably go through a few iterations – maybe number of lanes will change based on the cost projections.

During the design, you'll never:

  • cheap out on the architects – it's vital you get this right and the design phase cost is small compared to the overall project budget
  • rush the design phase – most of the calendar time is spent chasing around stakeholders (getting people to decide on building a bridge in the first place, chasing permits, etc) so there's no need to rush the architects

Once everyone signs off on the plans, you know exactly how and what to build, down to the bill of materials. Then you go and build the bridge.

Once it is built, it is done, except for regular maintenance. The terrain doesn't change, the materials don't change their properties, and you definitely don't attempt to move the bridge or add another lane in a year.

You also don't expect terrorists to be attacking the bridge, and you usually don't account for once-in-a-thousand-years natural disasters. If you try to, you'll be accused (correctly) of over-engineering and being insecure. There are standard safety factors and you don't need to go over them except for a very good reason.

I'm not a civil engineering myself (I have a bit of background in mechanical engineering), but I don't think any would find the above description and assumptions wrong. Simplistic, yes, wrong, probably not.

In contrast, in software development you often:

  • don't know the exact requirements, or they will change throughout the development process
  • want to keep the software as configurable and adaptable as possible (no hard limits on the number of users or amount of data, for example)
  • can't account for all the differences in the platforms your software will run in (either native desktop/mobile, or across web browsers)
  • deal with platforms or frameworks that change rapidly and in a ways that make you rethink the design of your software
  • need to interact with poorly specified and/or poorly working 3rd party components (even if they work great, network stability can always be a problem)

What's more, with software:

  • we expect to need it to change – either to evolving business realities or to keep up with the underlying platform changes
  • we expect it to be reasonably robust against malicious attacks

In contrast to civil or mechanical engineering, when we talk about “designing the solution” in software development, we talk about having a high-level architecture with roughly fleshed-out components (what they are, what they need to interact with). It's not even close to the level of detail the architecture plans give the bridge builders.

This means that the design phase in civil engineering doesn't map to solution design in software development – it actually maps to the entire programming effort, which is the bulk of the work, cost and time in any software development project.

        +------------+-----------------------------------+
BRIDGE  |   Design   |              Building             |
        +------------+------------------------+----------+
 APP    |             Programming             |   Ops    |
        +-------------------------------------+----------+

Copying bridges

Ok, you build one bridge. How hard is to build another just like it? Well, you repeat the whole process again in its entirety. Hopefully you do reap some savings due to the fact that you're now more experienced and have established supply chain connections, but that's it. The majority of work is still there and if everything else stays the same, the budget won't be very different.

In software development that's – you just make (or deploy) a new copy. Done!

What does this mean for builders themselves? This means in civil engineering you get more money for building (executing on the plan), while in software development you get more money for developing the solution (the design part). You just don't hear about multinational mega-corporations of architects on a level with FAANG.

Interestingly, the multinational mega-corporations handling ops do exist (PaaS, IaaS, app stores). They combine infrastructure, distribution and operations – in our comparison, in the civil engineering world they'd be an amalgam of builders, utilities and shipping companies all fused into one.

Cost of failure

If a bridge collapses, that's a huge problem, even if noone gets hurt. If software malfunctions, in most cases it's not a big deal (sometimes it is and then it's newsworthy). More often, if it works but badly, people scrape by.

Nowadays you can even fix the bug and push an over-the-air update – no need to recall all the cars of planes you've built with the defect.

This means the cost of failure (either catastrophic, or failure to live up to the expectations) is much lower in software than in either civil or mechanical engineering. That again means the return on investment (ROI) on any effort mitigating that cost is small, discouraging spending time and money on it.

So software today is buggy becase it can be. It's slow because it can be. It's insecure because it can be. It's not due to any lazyness of programmers, incompetence of managers or malice of big bosses. It's how the world works – if you spend too much on quality where not required, your competitor can spend the same money to grow faster with no downside.

Yet it is not obvious that it is unequivocally a bad thing. Consider the failures of famous projects with (near) unlimited budgets and perfectionism, such as Duke Nukem Forever. It makes as little sense to overengineer a software product as it would be for a building. Tech moves at breathtaking speed, and we've come so far in just a few decades, because it often can “move fast and break things”. Software can be more adaptable and quicker to the changing needs.

As software becomes a vital part of more and more things in the real world, the costs of failure rise. Whereas you might be only slightly annoyed at having to restart your Facebook app, you really don't want any trouble from the software controling your two-tonne car while you're doing 80 on a highway.

If that means more software will need to move slowly and avoid breaking things. And maybe we'll be more like the bridge builders.