Balancing Perfectionism, Deadlines, and Budget in Software Development

From an experienced developer’s perspective, the ideal order of importance in a project usually looks something like this:

• Well-Architected Code
• Security
• Clean & Readable Code
• Functional Code
• On-Time
• Looks Good
• On-Budget

From a management or client perspective, you probably see the flaws with this list.

Developers tend to be fairly opinionated people, and many of us are not highly business-minded people. As such we tend to be perfectionists, or at least highly critical thinkers, who enjoy the process of analyzing, engineering, and refactoring our code for security, performance, scalability, you-name-it.

The issue is that projects cost money, and more time spent means higher cost. This problem compounds when you move from a FAANG style big-tech company into a smaller product-based startup, where those costs have a real impact on the company’s future. And they compound further when you get into the agency world where customers are paying the agency for services.

In the real world, we often find ourselves with projects that look more like this:

• On-Budget
• On-Time
• Looks Good
• No obvious bugs

A naive project manager would think that list looks perfect but, as with most things, there are trade-offs. Say that client reaches out 2 months later with an update that requires major re-tooling because there wasn’t enough time/budget in the original project to properly architect the app. Now you have to explain to them why you didn’t do it right to begin with, and you have to convince them why they need to pay extra now.

I believe there is a happy middle ground in which developers and PMs can both be happy. Of course, there are still trade-offs. And no, Agile is not the solution for your small-mid-size agency no matter how many JIRA ads you see.

Below is a detailed outline of what I have seen work with fairly regular success for digital agencies with small to medium size dev teams.

Managers & Sales

Include paid discovery meetings in the initial scoping process

• Include a mid/senior-level developer, someone from UX/Design, or at the very least a sales engineer on these discovery meetings to get a technical and design perspective

Set a realistic budget for a project, and make sure that your scope clearly defines what needs to be done

• Involve your designers and developers as early as possible and listen to their input
  • You may not always like to hear that something is going to be complicated, but it’s better to hear it up front than in the middle of development.
• Clients always want as much as they can get for as little money as they can get it for. Make sure you keep things realistic, and educate them on the trade-offs.
  • If a client wants more features, you can explain that it will cost them more and you can brainstorm with them on what compromises can be made to keep costs low.
  • If you have a client who has a hard cap on their budget, you can help them to identify the truly important functionality that will fit within their budget.
• I have seen scope docs that had line items like “Build a Website”. This will absolutely come back to bite you, and your production team will pay the price.

Estimates & Budgeting

Budgets are always too low and Estimates are always wrong – end of lesson.

Seriously though, an Estimate is a gut check at best. The larger the scope of the project, the more inaccurate the estimate is likely to be. An experienced developer/project manager is likely to be able to get a good gut feel, but make sure you are building in padding for the possibility that things don’t go according to plan.

I touched on this above but if you are working with a client who has a hard cap on their budget, your job is to work with their budget and try to satisfy their needs. In most cases, you can work with a client to reduce hours and reach a compromise. If a client’s budget is simply too low for their expectations, you can always politely point out that you have reduced as much as you are comfortable with, decline, and move on.

The Importance of Scope

It really can’t be stated enough, scope definition will make or break an entire project. If a project does not have a well-defined scope from the start it will result in inaccurate estimates, missed deadlines, feature creep, miscommunicated deliverables, frustrated clients, and burned-out designers and developers.

So, how do you deal with all that?

• Bring in the design and development team to weigh in for sales and discovery calls
• Sell the client on a bucket of hours for discovery, to ensure project success
• Make sure the client elaborates in great detail about all of the desired features
• Make sure that your proposals and project timelines include very specific line items detailing exactly what the deliverables are

Developers

Be pragmatic, not perfect

Realize what can be compromised, and when it makes sense to do so

• HIPAA compliance is important, and not something to compromise on. Can you really say the same thing about that overly complex caching system you are building for a website that gets 1000 hits in a month?
• Sometimes clients really do want the cheapest option and would rather take the tradeoffs. As long as they actually understand what those tradeoffs are (and they are not security-related), sometimes it’s okay to build a less flexible, less scalable, or less performant solution.
  

Have empathy for the managers/clients you are working with

• It’s easy to forget that sometimes the PM had little control over the Scope they got stuck with, or that the client might be a struggling startup
  • I personally find it helpful to communicate with clients and get to know them and their businesses. That helps me to make sure that what I am suggesting to them is what I truly believe is the right option for them.
• Some companies choose to totally separate their developers from client relationships. Personally, I think this is a bad idea. Developers are people like anyone else, and having some skin in the game can go a long way. On the flip side, don’t give clients direct access to developers if you have a client relationship manager who can handle ad-hoc requests to balance what the dev team has to focus on.

Refactor proactively

• If you find yourself thinking “I will eventually block out some time to refactor this ugly code”, you probably need a change in perspective. The thing with refactoring is that clients and managers often don’t understand it, and waiting for approval to do it usually goes nowhere.
• If the change is small enough as to not incur any major overhead, just do it while you are working around it and make sure it gets tracked for review/QA.
• If the change has a wide scope, or the entire codebase is a mess, you are probably better off waiting until there is an opportunity to pitch a rebuild as a feature improvement. Another thing I have done in this situation is pitch a rebuild when it would actually take more time to work around the messy piece of functionality.

Build proactively

• Use starter code that addresses common concerns with architecture, security, optimization, etc. This will help you build faster while compromising less.
  • I am a fan of custom pre-built component/module systems in starter apps – If you architect your starter app well you can customize and extend easily for more complex apps, while also having a solid base to easily stand up faster/cheaper projects
• Working on something that can be re-used? Put it in your starter app.
• Use gists, snippets, notepad, hell even sticky notes – Use whatever works best for you, but write down a solution to any problem that you face more than once, and comment the shit out of your code so you don’t have to waste brain power on solving the same problems over and over

Let it Go

• Sometimes we have to write bad, ugly, or otherwise unsatisfactory code. There are endless reasons. As long as you are not compromising privacy, security, or legal obligations, it’s probably fine. Sure, in 5 years someone is going to look at it and think “What absolute psychopath wrote this?” but, at that point, you’ll probably be doing the same with someone else’s spaghetti code.

Clean Code, Quick Hacks, & Compromise

I like a well-maintained and organized codebase as much as the next software developer. I think that’s something most professionals can agree on, regardless of whether you prefer vim or emacs.

Unfortunately, the reality is that, whether the constraint is time, budget, or something else, sometimes quick fixes, hacks, and ugly solutions are necessary evils.

In these situations, it is important to keep balance in mind. There may be times when you need to sacrifice readability for performance, or vice-versa. There may be times when you need to sacrifice robustness for a quick solution.

As long as you aren’t compromising security/privacy/compliance, it’s usually okay to make a compromise but when you do you should still be thinking about how to make the best with what you have to work with. Just because you are building a quick and dirty version doesn’t mean you can’t write a function to avoid copy/pasting the same code 5 times.

Also of note, there is a difference between writing less-than-optimal code and writing code that is an unmaintainable mess. Sometimes code really doesn’t need to scale, but that doesn’t mean it doesn’t need to be robust and flexible.

Finally, I think we are probably all guilty of inheriting a poorly built, poorly maintained codebase at some point and adding to the mess. Inheriting legacy code is part of the job, and is something that all developers have to come to terms with.

The best thing you can do when you find yourself surrounded by miles of spaghetti code is to take a breath and try to make sure that anything that you add improves the code. If you’re able to improve existing code while you are in there, even better. That said, sometimes there are cases where you get handed a complete shitshow and it seems like everything you touch breaks something and it just isn’t worth touching the existing code. In those cases, you can still think about your impact on the codebase and how to keep your work well-maintained. Contributing to the mess only hurts yourself, and anyone else who has to inherit the code after you.

Over-Optimizing & Over-Engineering

Software developers often love problem-solving, logical thinking, abstractions, and optimization. These are all generally good things, but they can be overused and used prematurely.

As with most theoretical concepts, someone will always find a case where a specific approach works. In my experience, there is no “right” answer that fits all projects. How you go about engineering any particular app depends on a combination of scope, budget, timeline, platform, programming language, and project type. There is a fine line between the right amount of optimization and architecting for a project and over/under-engineering. It can be hard to gauge that line, even for experienced developers. I don’t subscribe to any single specific programming philosophy but rather pick and choose the bits that make the most sense to me.

Over-Optimization

Over-optimization is easier to think about for me, personally, I usually think like this:

If it is easy optimization

Just do it and it will become habit

For example:

Using wp_cache_get and wp_cache_set in WordPress for queries that happen multiple times on a page load. It’s an extra couple of minutes to include it and after you do it 100 times it will be second nature.

Caching selected DOM elements in JS and jQuery when possible. You save on repeatedly traversing the DOM, and you aren’t adding any extra dev time.

Avoiding unnecessary looping and variable declarations. Admittedly this comes with practice and experience, but if you take the time to understand computer architecture and programming languages you will start to notice little things that are easy to avoid from the start and give you performance boosts.

If it is moderate to difficult optimization

Think about what the primary scope of the project is, decide if it is really necessary right now, and think about whether it would be difficult to optimize later.

For Example:

If you are creating a database schema, changing it later will be difficult because you will need to migrate and handle legacy data. This is a scenario where optimizing for future scalability is worth considering.

Using full-page caching systems to cache and serve pages on your website. Does the site get enough traffic that this is relevant? Is the website slow enough to warrant this approach? Is there a better way to optimize the site?

Using transients to cache query and API results in WordPress. Are you gaining anything from using a transient? It is still a database hit after all, so it should be boosting performance in some other way.

Profiling custom-built functions or language builtins to determine the absolute optimal implementation. Unless you are troubleshooting a very specific problem or you are handling massive amounts of data, you probably don’t need to go this hard. In average use-cases the tradeoffs will usually go unnoticed.

Over-Engineering

Over-engineering can be more subtle. If you don’t build a robust enough system you can find yourself in a bind down the line when you need to extend a rigid piece of functionality, but if you build a system that is too complicated you and your colleagues will wind up in abstraction hell. My personal rules of thumb are:

Always consider reusability

Should this piece of logic be a function? Can these values be parameters, rather than hard-coded? Should this markup block be a template?

Always avoid magic values

Define hard-coded values as constants, function returns, config values, etc. You will thank yourself later when you need to change the value in 20 different places.

Use early returns

I know. Some people don’t like early returns. Too bad, this is my opinion. Early returns improve readability and prevent unnecessary nesting. Use them when it makes sense, and don’t when it doesn’t.

Favor readability where possible

If you find yourself writing something like if( ( i < j && ( x > y || a < b ) ) || !( k > l && ( m < n || o > p) ) ) you should consider turning your conditionals into descriptive function names.

Build for others/future you

When writing abstractions, functions, classes, etc. Try to stop and think beyond what makes sense to you in the moment, and think about what will make them dead simple to understand at a glance.

Don’t build what you won’t need

Any experienced developer knows that it can be hard to go back and add things that you didn’t plan for, and I think this drives us to over-engineer.

Consider this class hierarchy: Entity->Animal->Human->User->Student for a school database. Overkill, right?

This is how it feels when you have to dig 6 classes deep to figure out what some piece of functionality is supposed to be doing. You can run into similar issues with over-abstraction as well.

It is unlikely that the school database will ever need the level of abstraction captured here, but that doesn’t mean you should go to the other extreme. Imagine 3 disparate classes Student, Teacher, Administration. It is quite likely that these 3 entities will share some common traits.

A better model might be User->Student, User->Teacher, User->Administration or even User->Administration->Teacher, depending on the project needs. Abstraction and polymorphism can make code very nice to work with when used properly, but if you cross the line they can make it exponentially worse to work with.