Done... until it isn't: how most data engineers deal with tasks

Most engineers know this moment. You finish a task, it works, and the ticket moves to Done. You push the code and move on to the next thing.

Then, a few weeks or months later, something changes. A requirement evolves, someone needs to extend what you built, an issue shows up in production, or you simply open your own code again. And you think:

What on earth was I thinking?

…or something slightly less polite. In my experience, the problem is rarely that we didn't know how to solve it. Looking back, we can usually spot one or two things we'd do differently. We haven't suddenly become better engineers; we just stopped thinking as soon as the solution worked.

‍

We naturally optimize for "it works"

We all work with capable people who solve complex problems every day. Knowing how to make things work is rarely what's missing, and it's not that people don't care. In consulting we're constantly balancing quality, deadlines, client expectations and new requests, and there's always another project waiting.

So it's natural to optimize for “it works”, and that's the right starting point. The problem is when “it works” quietly becomes our definition of “done”.

What we build keeps evolving after we deliver it: requirements change, data changes, volumes grow, and someone else will need to understand what we did. Quite often, that someone is future us.

There's a difference between finishing a task and finishing a solution. Sometimes the task is complete, but the solution deserves another five minutes of thought.

Five questions before calling it done in data

I usually try to ask myself a few questions before I consider something finished. It's not a formal checklist, and I don't get it right every time. The questions just help me step out of the task for a moment and look at the solution from a different angle.

‍

What could fail?

We naturally design for the happy path, and production has a way of finding the one scenario we didn't consider. A load runs twice and a report suddenly shows double the revenue. A source column gets renamed overnight. A file arrives with NULLs where nobody expected them, or an API simply times out. None of this is exotic.

So beyond “does it work?”, it's worth asking: “if it doesn't, will we know what happened?”. A useful log or a simple validation is often the difference between a painful incident and a quick fix.

What will probably change?

Trying to predict every possible future is a great way to overengineer anything. But we can usually spot what's likely to evolve: a hardcoded date or ID, a business rule buried in a query, a source schema, a volume assumption that holds for today's ten thousand rows but not for next year's ten million.

Leaving these things simple is often the right call. What matters is making that decision consciously.

Do we trust this in production?

The question isn't whether it can technically run in production, but whether we'd be comfortable knowing it's running tonight with nobody watching.

If that makes us hesitate, there's usually something worth looking at: validation, logging, error handling, what happens on a rerun, or that one fragile part we were hoping wouldn't become a problem. We've all done that one.

A simple test: imagine the job failing at 3 a.m. Who finds out, and how? If the honest answer is “the client, on Monday morning”, it's worth another look.

Would we be comfortable maintaining this?

We often talk about “the next developer”, but quite often that's us, six months later, and far less forgiving. Will we understand why a decision was made? Will debugging be easy? Will changing one small thing mean changing five others?

In data work this shows up quickly: a transformation with five nested CTEs and no comments, a column called flag2, or a pipeline where one small change means updating three notebooks and a dashboard. Often a clear name, a short comment explaining why, or splitting one giant query into steps is all it takes. It feels theoretical until the day we're the ones maintaining what we built.

Would we confidently present this to a client?

This is probably the one I value most. Imagine the client asks: “Why did you build it this way?”, “What happens if this fails?” or “How hard would it be to change this later?” Would we be comfortable answering?

In data projects, these questions come up more often than we'd expect. Why doesn't this number match the one in finance's spreadsheet? What happens if last night's load fails? Can we add a new source without rebuilding everything? A clear answer, even if it's “we accepted this trade-off on purpose”, builds far more trust than a polished solution nobody can explain.

The solution doesn't need to be perfect, and the answer doesn't need to be impressive. But we should understand and stand behind the decisions we made. If I wouldn't be comfortable explaining a decision to a client, that's a sign to think about it once more.

This isn't about perfection: it is about the quality of our work

None of this means building perfect software, adding more process or turning every task into an architecture workshop. The simplest solution is often the right one. A hardcoded value can be fine, and a temporary shortcut can be justified when the context calls for it.

The difference is whether that simplicity was a conscious decision... or simply the first thing that worked.

This matters because what we build in consulting gets used, often for years. A teammate, the client or we ourselves will maintain it, extend it and depend on it. It's worth being a little more thoughtful about what we put out there.

When the extra 5 minutes pay off

Not every task deserves the same level of scrutiny. A one-off analysis or a quick prototype can live with a few shortcuts. Over time, I've noticed some signs that a task deserves a second look:

• It runs without anyone watching. Scheduled jobs, overnight loads, anything that can fail silently.
• It produces numbers people will trust. Reports, dashboards, anything that ends up in a client's decision.
• Others will build on top of it. A shared table, a common function, a model other teams will reuse.
• It's hard to undo. Deleting data, migrating a schema, changing something other systems depend on.

When a task ticks one of these boxes, those five minutes are usually the cheapest time I'll spend on it.

One question I'm trying to ask more often

I still stop too early sometimes, and I still open old code and wonder what I was thinking. But I've started asking myself one extra question before I close a task:

Have I really thought this through?

Sometimes the answer is yes. Other times it sends me back to add a log, remove a hardcoded value, add a data quality check or test one more scenario. Nothing revolutionary, but it helps more often than not.

And, hopefully, the next time I open my own code, I'll swear a little less.