02/06/2025 | By Sasha Dzeletovic
In large digital systems, countless technologies, teams, contracts, vendors and dependencies make it overwhelming to grasp every aspect at once. How can you possibly make informed decisions without awareness of how your business's data is acquired, transformed, and delivered? So, what's most important? The flow of data. Here’s why.
By creating a high level map of your entire business data flow, you can confidently grasp what's happening across the board without having to know every detail, especially technical. There’s a way to do this which is not overwhelming, doesn't take forever and doesn't cost a fortune.
In large organizations, data and its transformations happen everywhere, often between teams that barely interact yet rely on each other's outputs. The first step is to map all the standing data and the moving data across the company, high level. This becomes the anchor for all other conversations. Without it, you can give resources to the wrong concerns, which is worse that doing nothing.
You build on that base-map by adding more detail as needed and as it becomes available. Now every other conversation has a place where the significant information can accumulate and add to the big picture. It serves as a scaffolding to connect the dots.
This is the simplest way to capture the most significant information about your system.
This approach applies to all parts of the system: data acquisition, cleaning, normalization, data lakes, data marts, APIs, and user applications for mobile, desktop, and web. It's inherent to digital technology, which primarily manages records of reality.
You can stretch this concept even further. An opinion about an event is also a data record. For example, an email expressing an opinion isn't necessarily reality, but it records that a person held a certain view at a certain time.
This is as true today in the time of Artificial Intelligence as it was when the Sumerians were recording harvest yields on clay tablets.
Since large organizations often lack awareness of all their parts, three common symptoms occur.
Duplication of work which leads to exponential complexity, and lots of wasted energy.
“We didn't know that nobody was covering that post”, which leads to disappointing customers and legal liability.
Missing an upstream or downstream dependency, which nukes expectations, prevents delivery of value to market, and saddles you with lots of extra work. Event to remove it!
If you want to run a tidy ship, overall data flow map is indispensable.
Such a map, void of technical details, is a great tool for cross-functional discussion so everyone can be on the same page.
With this map, you can make high-quality decisions, closer to reality, about what to optimize, what to leave alone, what to dismantle, and where to invest heavily. It allows you to know which risks are taking.
Beauty of getting this map lies in the simple visual language it uses, and the ease of working with it.
Squares represent Standing Data, circles represent Moving Data including data transformations. Arrows represent flow. That’s it. It’s easy to learn.
Name squares with nouns of data they represent, and circles with verbs of what they do.
You can create it on parchment paper with an ink quill, or in any diagramming software. I do recommend the latter. Diagramming software, like Lucid, allows you to add custom metadata to every node that’s relevant to your organization.
Metadata allows you to derive further relevant information. For example, infrastructure cost or size of the team managing it.
This map shouldn’t change radically and often, because systems do not change radically and often. This makes it easy and cost-effective to maintain.
This approach is applicable to entering a new organization and being clueless about the system, when you need to fix an unknown, when you need to adjust fast without breaking anything, when you need to optimize, or when you need to act on a new market opportunity.
This is why you start here.
