Preventing (or Invoking) Duplicate Work Across an Organization

THREE POINTS
Henry Moore

Back when this blog was but a twinkle in my eye, I gratefully received some topics to write about on Twitter. One such:

See more

Duplicate work across an org.

— Jeff Bean (@jwfbean) February 24, 2022

This reminds me of the Fifth Normal Form in database modeling: every piece of data is expressed uniquely. Something that many Wikipedia articles don’t give you is tradeoffs, opting to only describe what something is rather than what would I use it for or better: what shouldn’t I use this for; the same critique I have of basically all AWS documentation^[1]future post: The dum dums guide to AWS services because I just, arghhhhh. I’ll get to that in a moment.

There are tradeoffs inherent in any sort of socio-technical system, mostly having to do with communication patterns and incentives.

In the Fifth Normal Form, you are trading off a sort of purity or simplicity of modeling your data so that when you need to update or change it, you only need to do this once. The tradeoff is in accessing it^[2]this also absolutely happens in warehouse storage by the way. If you store things one way, you can cause hot-spots when people go to retrieve since all the bread is in exactly one spot; stored … Continue reading: many real-world use cases of displaying this data (unless it’s a simple admin interface) will have you joining across many many tables.

Think of a dashboard for example: who is this person logged in as? What can they access based off of their roles and permissions? What are their preferences? Off of that, what graphs to show? What data for the graphs? Any correlations to show? Most dashboards are just reaching for disparate data to show patterns and trends, and given a 5NF, this means lots of joins.

In the small, joins are very fast. If you can load the entire database into memory^[3]and I’m sorry, if you can load your database into memory, it is absolutely not Big Data, and doesn’t require hadoop or whatever, everything is fast, even if you don’t have indexes. In the large, this pattern will absolutely kill performance because joins become prohibitively expensive and don’t you really just need essentially every page to be as simple as select * from dashboard? You can get there…. with caching^[4]future post: the two hardest things about computer science are remembering which two, so, cache invalidation and naming things.

In the very, very large, you would end up doing in-application joins across the network because each table is now a service, or several. Eep.

Back to the issue: In the small, everyone knows everything. You can somewhat fit everyone’s work into your head, and it only requires light updates from them to synchronize your mental model, usually over lunch. In the large, you can have this high level of fidelity about your most immediate team, but even for sibling teams in the same org, it can be difficult to know what they’re working on without a different way of working. Which is where *tada*^[5]imagine I’m doing jazz hands managers come in.

Prevention

Duplicate work is a discovery problem. If person A way over here knew that person B way over there was doing sort of the same thing, would they still be doing it?

Something I’ve seen effective managers do is act as information collectors and radiators. They absorb as much inter-org data as they can, and then share it with relevant people in meetings and 1:1s, and broadcast via messages and mailing lists and presentations. As they are radiating, it is helping ICs and managers to think about related topics, perhaps even duplicate work, which in this twisted analogy is sort of like a join if you squint.

In a startup, this problem solves itself. It would be pretty difficult^[6]read: willful ignorance to accidentally have more than one person working on the same problem at the same time. Often at the very beginning, it is one project per person.

But it’s a different game entirely when you have teams of teams of teams of teams. What can you do?

Every few years I read Steve Yegge’s Google Platforms Rant. I think it’s instructive in a few ways.

1. Working for Amazon or Google sounds pretty difficult^[7]And no disrespect to those who work there, either; I really admire y’all, it’s just not for me.
2. If you think of your teams as a “platforms” themselves with an API, and if you are able to discover and use that API effectively, then it will likely solve this problem

The if at the end there is a giggggaaannnttticccc if, and points to some truer causes of the problem if we apply some analysis to it:

• Why didn’t you just use Service B?
  • Because I didn’t know about it
• Why didn’t you know about it?
  • I .. just didn’t? I know about Service C and D but they don’t fit my needs
• How did you know about C and D?
  • We use them already in our project
• Are there any other services you know about here? How do you know about them?
  • Well my friend told me about Z and Y; I stumbled across M and N in our wiki, but when I went to look at their pages, they hadn’t been updated in years.

I think you can see where this is heading: It’s a knowledge management problem. If only there were a discipline devoted to the efficient and accurate retrieval of knowledge^[8]future blog post: the future of engineering is hiring librarians and people who write good… /s. It’s library science.

Invocation

Is duplicate work all bad…? No.

Against Failure

Let’s say that you’re sending a satellite into space and you know that the processor and all electronics will be bombarded with radiation and so the possibility that bits will flip is inevitable rather than remote^[9]you know because of the giant magnet we sit on and the atmosphere; also ECC ram. What can we do?

Well the chances of two processors having bits flipped at the same time is probably pretty low; or three is even lower. So why don’t we pack some more processors in there? And so they do to guard against failure.

I’ve also heard reports of companies dispatching two teams to build the exact same thing as a sort of competition to see which one will figure out the problem first. The chances of both failing are low^[10]w.r.t. morale, not so sure.

Against Context Switching, Dependency, Bloat

In a similar way, if there are two independent teams building something purpose-built for their system, there are some real benefits to doing it.

1. They can tailor it to their specific needs
2. They don’t have to wait on another team to fix bugs or implement features they want (or the reverse: they don’t need to interrupt to fix things for other teams)

This is very close to the buy vs. build equation^[11]future post: why does everyone build stuff rather than buy it, or buy the wrong thing.

Against the Unknown

And a personal anecdote since it’s been a few years: In mid-2020 when the Paycheck Protection Program and stimulus checks came out in the US, the whole company^[12]Block née Square sprinted and made tiger teams^[13]sort of like a supergroup made up of engineers from different teams but all working on a special project together and then disbanding afterwards to specifically try a lot of things all at the same time to see what would pan out. The directive from the top was literally “pause all roadmaps to focus on helping our customers.”

One effort I was involved in was how to get ACH account and routing numbers to the IRS so that the stimulus check could be directly deposited into Cash App accounts, so that people could get their money instantly.

One such effort was to file a shortened 1040 which, since our team files things with the IRS all the time^[14]future post: the world is largely built on mainframes and CSVs, we could advise. Ultimately it didn’t work out because the IRS requires a lot of testing to make sure your systems can interoperate, and it was going to take too long.

This was GOOD! To go very quickly you need to have divergence and creative options, particularly in an unknown solution space, which then you converge to solutions that will pan out. The fact that you pruned things that didn’t work out doesn’t mean it was wrong to have spent that time–y’all probably learned a lot and covered a lot of ground. But, it is very expensive.

Future Work

What I will leave you with are things I’ve seen in the wild that attempt to address this problem, but unfortunately without any Hard Won Knowledge from me about how to fix it forever because honestly, I haven’t, and every organization is different. I would love to hear from you your horror stories (or happy stories?) of duplicate work across organizations.

Serendipity

Often it’s an accidental meeting that leads to sharing knowledge about similar projects. Then some conversations happen to determine whether both need to exist or just one (with modifications to support the other use cases^[15]which also sometimes results in weird long-term consequences like the original system is used at HUGE scale and your system needs like a millionth of its capacity, which also means that when you make … Continue reading). But as far as strategy, I can’t recommend it^[16]Though if you read Delivering Happiness by Tony Hsieh, Zappos started an impressive experiment to move the whole company and their employees to Las Vegas to increase serendipity; not sure how that … Continue reading.

What if it’s all, like, an API, man?

Related to the Steve Yegge rant above: A large company we integrated with once had completed a transition to wrapping every single API into GraphQL. Integrating with it was awful. But on the other hand, we had access to any system’s data and could stitch it together. But on the other other hand, there were byzantine data dependencies between systems, and so you couldn’t just, say, create a new record. We ended up sort of crawling across many nodes and reverse engineering because again, documentation is hard and no one wants to be good at it nor hire specifically for the skill of making it (unless apparently it’s a core part of making money because you’re selling an API).

Product -> Platform

One company is mid-way through flipping most of its external and internal products into platforms available via API^[17]a little different from what Amazon did which turned each team into an API.

There is a central registry of services so that theoretically you could discover ones that would be useful to you, but not many have spent the time to sell or pitch them. Many are named not what they do, but are puns or inside jokes. They are essentially undiscoverable unless you know someone or take the time to read all their documentation or code or readmes (also of varying quality).

Broadcast channels

Speaking of pitching: If your company has gatherings of engineers across various dimensions (like Special Interest Groups in frontends, observability, databases, general show and tell, etc), I have seen this work as a broadcast medium for up and coming projects or new APIs that people can use. Anything that people have to prepare for will be less noisy than, say, email.

Thank you

Thank you so much for reading. For more up-to-date thoughts you can follow me on Twitter here: Follow @perplexes