Christian Maioli M.

đź›  Web development and đź–Ą computer programming

Category: Agile

Being Agile and working smart are not the same thing

The discussion about software methodologies is not over. Agile did not win, nor did any other methodology. I’ve never worked in a team that hit the right spot in team dynamics and was as effective as possible, whether using incremental development or not. I think I’ve figured out what it is that still needs work in that area.

Most discussion about methodologies is about trying to find an ideal one. Should it be more structured? Less structured? How many meetings are we doing? You try a bunch of recipes such as Scrum, Kanban, waterfall; sometimes they work, and sometimes they don’t. It’s like aiming at a moving target.

The problem is not which recipe to use, it’s being aware of each one’s strengths and weaknesses. Teams need to know when structure is beneficial and when it’s not—and realize that achieving good team dynamics is a skill to be honed. You can’t write down a process plan and expect everything to magically be smooth sailing from then on, because project requirements vary in many ways that sometimes aren’t self-evident.

Even throughout a single project, using a single methodology is far from ideal. For example, are short iterations a good fit for a project that needs big design up front, no matter what? And if you use long iterations, what happens later when you realize your design is lacking in a specific area?

Looking over these issues, I’m going to make a case for why:

  • Being Agile and working smart are not always the same thing
  • An adaptive mindset means knowing when your methodology helps/hinders you
  • Office life and daily issues can completely change the effectiveness of your process

How development focus changes throughout a project

In a large project, the teams focus is constantly changing. When development begins, you might split the project into modules, with strong code ownership rules and laser-focused development.

Later on, when most modules are going through a debugging and testing phase, code ownership gets in the way. At that stage it’s more productive for a developer to go through his own short testing/debugging cycle and be able to fix bugs anywhere, and excessive communication would slow her down considerably. Although someone with more knowledge about a module will be more productive when working on that part of the code, on the other hand, more eyes on the same piece of code will lead to less defects.

Perhaps the realization that things change constantly is what brought us to have more granular methodologies like RUP, with it’s bazillion roles and artifacts. A highly dynamic methodology like RUP requires people to learn new habits constantly. It’s hard to even get people to make a stand-up last 15 minutes. I don’t want to know what happens when you make them jump through 10 times as many hoops.

More structure means more concepts to learn, a higher cognitive load, and a harder to apply methodology. If the question is how to make the work flow as smoothly as possible, then jumping through more hoops is not the answer.

Is it right to use Agile everywhere?

Agile is lightweight, easy to implement, and the project’s direction can be adjusted based on completed work, not on speculation or predictions. It can adapt better to changes, but sometimes it prevents you from going where your project needs you to go. Here are a few cases where it’s iterative process doesn’t work so well:

  • You need a big design phase up front. You need precision and don’t care so much about development time. Anything NASA does would fit here.
  • Your business has a high cognitive load. It’s hard for users and/or developers to get it, so you need a lot of well-organized documentation. Enterprise software. Financial software.
  • You’re doing a complete rewrite of your software, so your team already has a clear idea of where to go. Requirements are clear, and there’s not much need for course-correction anymore.

There is a lot of focus right now on getting the kinks of Agile just right, and it’s a productive mission because most teams don’t get it right. On the other hand, to realise the full potential of your team you need to know when to switch things up, and when Agile is setting you back.

Besides those specific cases, it’s a good idea to always keep in mind the disadvantages of Agile:

  • It requires high customer commitment. Some customers are too busy for that.
  • Requires constant and/or automated testing. Some customers don’t want to spend money on that.
  • Huge communication overhead for remote teams. Prepare to spend more time setting up WebEx/Skype and on IT issues than actually writing code.

These problems are responsible for the fact that most companies today are applying some kind of hybrid Agile process rather than staying true to it’s principles. But it’s important to know why Agile exists, what problems it solves, and when to stick to the core Agile principles. This is why I said that achieving good team dynamics is a skill.

When to make changes to your team

Typically a team stays mostly the same throughout a project, but as the project progresses, it’s unlikely that everyone’s strengths are being leveraged at all times. People are good at different things.

Generally speaking, more experienced team members are better leveraged by having them use their experience more than their mechanical skills. The inverse is true for the least experienced.

Put both kinds of developers to solve a restricted coding challenge and the more experienced one might do it slightly faster, but he knows he can entirely avoid writing any code by just calling a specific library that a less experienced developer won’t even know exists. That difference in experience may well be where the “10x programmer” expression comes from.

One of the main focuses in modern software development has been in reducing cycle times, but they are an incomplete measure of team productivity. Getting stuff out faster will let you adjust more often, but that won’t help you if you’re not willing to use the right people for each problem, and taking their specific experiences into account. Familiarity with a problem is a huge leverage point, and modern methodologies throw it under the rug when they turn the whole team into numbers, each one with identical weight.

Adapt your daily routines to your current team

The final aspect you have to look at when you want to be a fully adaptive team is the details of your daily routines. These can make or break your methodology’s effectiveness.

I remember being in an Agile team where the daily standups were contributing almost nothing due to several factors. You might have seen it yourself, when noone has anything to report in the standup for days in a row. The thing that still made standups effective in that case was the fact that my desk was literally 2 meters away from where we would meet every morning. The effort to get there was so small that it didn’t matter.

A poor implementation of a methodology can still work if the specifics are adjusted to make up for the issues.

I’ve also seen the opposite situation, where you have a well applied methodology that is made worthless by the accumulation of small daily issues, like video conferencing software that never works and poor hardware or infrastructure.

The bottom line is that to make your development process work, you have to take a hard look at who you’re working with, how they like to get things done, and what resources you have at your disposal, and then make every effort you can to help your team stay in the zone. Make the working environment comfortable and effortless and the work will flow.

If your team has lots of experience, trust them, and help them enjoy getting things done. Don’t undermine the relationship by requiring them to jump through hoops to get work done.

Note: this post was originally featured on TechBeacon.

How terrible code gets written by perfectly sane people

When I found out I would be working on porting an old Python codebase to Node, I was slightly excited. These kinds of projects always give you more creative freedom than the ordinary code maintenance gig, and something about the challenge of rewriting other people’s code makes it fun as hell.

The excitement significantly faded once I actually got a look at what we were going to work with. Legacy code can be nasty, but I’ve been programming for 15 years and only a couple of times had I seen something like this. The authors had created their own framework, and it was a perfect storm of anti-patterns: no separation of concerns, mixed spaces/tabs for indentation, multiple names for the same concept, variables being overwritten by the exact same data coming from a different yet nearly identical method, magic strings… this mess could only have been the product of a room full of babbling monkeys copying code randomly from Google.

And yet, it was not the code’s dismal quality that piqued my interest and led me to write this article. What I discovered after some months working there, was that the authors were actually an experienced group of senior developers with good technical skills. What could lead a team of competent developers to produce and actually deliver something like this? What I’ve come up is a list. These are some bad habits that even experienced teams can get into which will severely affect your end product, more than any static code checker or development methodology could rescue it from.

Update: Mia Li was kind enough to offer a Chinese translation of this post here.

Giving excessive importance to estimates

An important component of this project was the focus on deadlines, even to the detriment of code quality. If your developers have to focus on delivering rather than on writing good code, they will eventually have to compensate to make you happy. The two ways in which they can do this are over-estimating and over-promising, and they both come with added baggage.

Typically it’s more difficult to over-estimate because the effects are immediately evidenced on project cost, so your developers might choose to over-promise instead, and then skip important work such as thinking about architectural problems, or how to automate tasks, in order to meet an unrealistic deadline. These tasks are often seen as added value, so they get cut without notice. Product quality will decline as you accumulate technical debt, and you’ll find out about it later than you’d really want to, because reorganizing code later in a project costs exponentially more.

As an example, on this project I would find code that was obviously duplicated elsewhere, but it seemed that people were in such a rush to deliver that some developers would not bother to check if someone else had written the same method or SQL query before.

Sometimes estimates can even be deceiving. For example, Agile has a term called “velocity”. The idea is to calculate how fast your team can deliver, and make the necessary changes to go faster. The problem is that it’s not possible to come up with an accurate velocity in the short to mid term. The law of averages says that you can’t look at past performance to gauge how fast you can go right now, because past performance is not a good indicator of future results.

The law of averages is a layman’s term for a belief that the statistical distribution of outcomes among members of a small sample must reflect the distribution of outcomes across the population as a whole.

In truth, a developer can write a large amount of code one day, and she can take three days to write five lines of code after reading documentation and collaborating with teammates. Averaging estimates will not net you valuable information in the short or mid term.

Giving no importance to project knowledge

As your project progresses, your team learns about the business, the concepts behind it and how they connect together. They also learn about the implementation as they write code, because you can never fully visualize how things will turn out and which challenges you will face. Some business fields even have an inherent complexity that takes a long time to digest.

As this was a full rewrite of old code, it was particularly interesting in this regard, because it serves to show whether your team’s management understands project knowledge and how it impacts development. If you’re in a large project and there are modules for which you have no expert, no-one to ask about, that’s a big red flag.  The value in rewriting code rests entirely on taking advantage of what you learned the first time around, so value that knowledge dearly.

If you put a different team to do the rewriting, as was done in my case, you are ignoring all your learning and relying solely on your new team’s skills, which likely won’t make up for the lack of information. A mediocre developer is going to do a better job at rewriting something he wrote himself, and he’ll do it faster, than if you give the job to someone else entirely.

Even hiring is impacted by project knowledge. The more information the project carriers, the longer it will take to bring people up to speed, so not only is domain knowledge more valuable then, it’s also important to focus on hiring good people. If you don’t hire well, you’ll get tied up to a bad team that will go nowhere for months.

Focusing on poor metrics such as “issues closed” or “commits per day”

When a measure becomes a target, it ceases to be a good measure.
– Goodhart’s law

At some point while I was getting up to speed on this project, somebody asked me why another developer was closing issues much faster than me, as if delivering faster is a good thing. As you can imagine, it took me a glance at his code to find four bugs in a single line. Focusing on unreliable metrics such as this will completely derail your project, and cause people as much stress as deadlines.

One metric that few seem to focus on is regression rate of issues. There are bugs such as null pointer exceptions that might show up much later, and if you’re not tracking regressions, it will seem as if bugs keep springing up out of nowhere. In this situation, you’ll never find the source of your problems, because you’re looking in the wrong place.

Ultimately what matters is what is delivered to the client, how happy they are with the product, and how it affects their bottom line, but it takes a lot of self-control to focus on delivered quality and ignore juicy metrics such as commit rate or issues closed.

A good way to know if a metric is useful or not, is to try to understand what personal values it outlines. Concentrate on metrics that advertise good attention to details, good communication skills and good attitude, especially if they require great effort to cheat.

Assuming that good process fixes bad people

Good process is portrayed as a kind of silver bullet in business. In my experience some companies, especially large ones with a poor hiring methodology, end up making their process increasingly strict to put toxic people in line, in turn restricting the creative freedom of the ones carrying the team. Not only that, but you still need people to carry out your process properly in the first place for it to work.

It never ends, and this entire problem can be disregarded by just fixing the hiring. Talent makes up for any other inefficiency in your team. That’s the entire point of favoring smart work over hard work.

Developers can be especially tricky to communicate with. In such a complex codebase, I had to constantly ask others for help, and would not often have people happily set time aside to give me a hand. That does not reflect a good attitude, and in tough tasks things can get especially stressful if you have to ask for help and can only count on a few of your teammates to have both the knowledge and the will to give you a hand.

You need people who can apply common sense and good taste, and who will call you out if your process is holding them back. Every build tool, every static checker and every communication tool has good and bad use cases, and you need people to tell you about it, not to blindly apply something that looked good in a different context months ago.

Ignoring proven practices such as code reviews and unit testing

Staying up to date on modern software development processes might not be enough to put a derailed project back on it’s tracks, but it’s certainly necessary anyway if you want your team to stay competitive. This is where proven practices step in, and there are a few of them. Test-driven development has been shown to reduce defect rates 40% to 90%, with an increase in development time in the 15%-35% range. Code reviews have also been shown to decrease defect rates, in some cases up to 80% over manual testing.

Imagine my dismay when I had to collaborate with a colleague on that legacy project and his screen displayed Notepad in its full glory. Using “search” to find methods might have been rad back in the nineties, but these days, refraining from using tools such as modern IDEs, version control and code inspection will set you back tremendously. They are now  absolutely required for projects of any size.

For an in-depth look at what is proven to work in software development, check out the book Making Software: What Really Works, and Why We Believe It. It has a nice list of valuable and proven practices which have been backed by studies over the years.

Hiring developers with no “people” skills

It’s not that developers can’t talk to other people. I was once a shy developer myself and eventually managed to stand in front of an audience and give a talk just fine.

The problem comes when someone isn’t willing to even try, or becomes annoyed by requests to improve communication. If there’s one thing that can speed up development time more than anything else that I’ve mentioned, it’s improving communication. Particularly when you reduce distances and work on informational proximity, you enable a more fervent and articulate connection in the workplace. It doesn’t matter that the other guy might be ten thousand miles away. One Skype call can turn a long coding marathon into a five-minute fix.

Conclusions

When you enable and encourage working smart by using the best tools, proven techniques and great communication, software development will definitely flow more naturally. What you can’t assume is that just because you’ve signed up to apply Agile or some other tool, that nothing else matters and things will sort themselves out. There’s a synergistic effect in action here which can make a team exponentially more productive if they’re set up right, and terribly slow and sloppy when no attention is paid to the details.

Note: this post was originally featured on TechBeacon.

Being a versatile hacker is becoming more important than knowing frameworks

Back around 2013, the term Full Stack developer started to come up in job descriptions and blog posts. Companies were realizing that hiring developers with expertise in only one language just wasn’t enough anymore. A web developer that can handle a variety of tasks and environments is considerably more useful, and was starting to become the norm.

In spite of this, knowledge about web architecture itself did not become widespread. Many developers have been building websites without having a good grasp of how things work behind the scenes. Web forms, caching, the HTTP protocol, Apache. All of these were secondary good-to-haves.

How e-learning affects the job market

Perhaps as a consequence of the online learning boom that had started a few years earlier, the self-taught web developer knows surprisingly little about the web’s underlying technology. Language-oriented courses cannot cover the complete web stack, and students will end up clueless about what an htaccess file does, or how to restart a Unix daemon, or how the different types of POST encoding work.

What is a full stack developer supposed to know, anyway? Job descriptions frequently mention combinations of frontend and backend technologies such as JavaScript and Node, PHP and jQuery, Angular and Spring, and many others. In reality there is a significant amount of information outside those realms that would improve someone’s ability to build a website, and gone are the days when you could stick with what you know and make a career out of a single technology.

The future of web developmentIf sticking to your guns won’t suffice anymore, then what can we do, and how can we keep up with the exponential multiplication of web libraries? There is so much software being released today, that the number of possible combinations between technologies is increasing very rapidly. This combinatorial explosion will drive software development into a more ad-hoc territory. Your chances of knowing how to integrate two random libraries X and Y are ever diminishing, and any help that googling can provide is diminishing at the same rate. The window is about to close, and a time will soon come when we’ll be required to figure these tough problems out on the spot, every time. Not something for the lazy ones among us.

Hackers: the antifragile programmers

I was introduced to this very interesting concept in an article by programming rockstar John Carmack. It’s described in the following quote from the Antifragile book:

“Just as human bones get stronger when subjected to stress and tension, and rumors or riots intensify when someone tries to repress them, many things in life benefit from stress, disorder, volatility, and turmoil. What Taleb has identified and calls “antifragile” is that category of things that not only gain from chaos but need it in order to survive and flourish.”

This idea reflects the attitude shared by those that used to be called hackers. Today the word has a negative connotation, but in the early days, it referred to a person with a certain attitude towards technology. As defined by the jargon file, a hacker is: “A person who enjoys exploring the details of programmable systems and how to stretch their capabilities, as opposed to most users, who prefer to learn only the minimum necessary.”

Antifragile - Things that gain from disorderThere was a time when looking things up on Stack Overflow whenever you had a problem just wasn’t an option, and many pieces of software had unreadable documentation, if they had any at all. I remember trying to fix a sound card issue as a kid, and reading the card’s manual, only to find assembly code listings there, with interrupt codes and all. That is the environment where hackers thrived, and that’s what we are going back to, sooner or later. If your first instinct when dealing with a complex issue that affects multiple technologies is to start with a Google search, you should reconsider your working habits.

Granted, being too curious can many times lead you down the wrong path, especially in the corporate environment where time is always short. As an example, it can be very enlightening to write test code for the basic use cases when learning about a new library, but coders looking to impress the boss will take the more pragmatic approach of copying the examples from the documentation, fully unaware of how they work. Giving value as a developer requires a certain amount of skill in time management and in setting expectations, as to allow you to seek the knowledge you need and to save the company money in the long term.

Rethinking the roles

How do you find the hackers? You need to find someone with a particular mindset, the particular curiosity and persistence that I’ve described. This has nothing to do with analytical intelligence, or with being able to memorize a particular set of academic algorithms, so whiteboard coding is out, and Fermi estimation problems don’t look too promising, either. Ask a candidate what he likes to do on his spare time, or what fun projects he worked on as a hobby, and you might be onto something. I have met many programmers that don’t write code in their spare time, and that can sometimes be a factor that tells you that they just don’t enjoy it. Look for clues which reveal their personality.

If you’re a developer, you might be worried that you don’t have that kind of drive or curiosity yourself, so what can you do about it?

Here are some pointers:

  • Whenever you have to google some error message or problem, read all the answers. Get as much context as possible on your problem, and do not be satisfied just with having come across a solution.
  • Learn about the technology, but also about the trade-offs that were made during its design and development.
  • Ask yourself what it would take for you to consider yourself a “complete” developer, and write down a path for you to get there.
  • Do what other people don’t like doing, go where they don’t want to go, and often enough you will be enlightened by the experience.

Software development is growing fast. Learning to code is easier than ever, and soon enough we will be in a survival of the fittest environment. But the guy that makes it is not going to be the guy that first learned about the cool new framework. It’s going to be the guy that asked himself what’s new about it, and what’s different this time. If you want to stay up to date with technology stacks, then stop worrying so much about being up to date, and start hacking.

Note: this article first appeared on TechBeacon.

Project delays: why good software estimates are impossible

Rubik's cube world record

Rubik’s cube world record

Have you ever tried to solve a Rubik’s cube and been unable to complete it? I once tried several times during a long bus trip and felt pretty bummed after failing every time. Then I learned that there are kids out there that can do it in seconds! How is that even possible?

Update: Mia Li was kind enough to offer a Chinese translation of this post here.

Unexpected complexity

When you, as a programmer, start a new project, you will often not know full well how to do it, for many reasons. But you are a professional, and you’ve completed similar tasks in the past, so you either try to figure it out, or find someone who can, and ask them how, or just google it.

Very often, you do not know you’ve found yourself in this dilemma until it’s right in front of you.

Here are some examples:

  • You have to re-implement something using a new framework or library
  • A library you’re trying to use doesn’t like the other library that you’ve been using
  • The API you’ve been integrating doesn’t do something you thought it should do
  • The framework you’re using doesn’t like unit tests and the ones it has are actually integration tests
  • You thought a model in this new framework was a singleton like in the other one but it’s not

Estimating unexpected complexity

“Can you estimate these new tasks for me?”, asks your boss. Remember that Rubik’s cube you couldn’t solve? How many hours do you think it would take you if you gave it another shot?

“What? Two days?”, “But our previous developer could do that in a couple of hours! No way it will take you that long.”

There is actually an algorithm you can learn that, with practice, will help you solve a Rubik’s cube very quickly. But you don’t know that right now, and there is no way to know that you are going to find that, and that it won’t really take you two days to figure it out.

Complexity is cumulative

There more tasks a project has, the more likely that you will run into this kind of situation, so if you do not severely overshoot your estimation, there is really no way to be sure that you’ll deliver on time.

Let’s do some crude napkin math to visualize this a bit better. Say that you are an experienced programmer, and you only run into unexpectedly complexity about 5% of the time. If you are starting a new project and split it up into 10 tasks…

1 - (1 - 0.05) ^ 10 = 0.40

Or: there’s a 40% chance that you will run into a complex task, and blow your estimations for this project.

Why are projects always behind schedule?” backs this up with more math and with actual project metrics over 70,000 hours of work, so check that out if you’re interested.

Creative vs mechanical tasks

“The creation of genuinely new software has far more in common with developing a new theory of physics than it does with producing cars or watches on an assembly line.” – T. Bollinger

Einstein trying to estimate a project

Einstein trying to estimate a project

The problem comes down to the difference between tasks which require a lot of thinking, and routine tasks which you already have some practice with.

In “Software has diseconomies of scale“, there’s an interesting argument being made about what this difference has on productivity. For creative tasks, the more of them you have, the more time each one will take, whereas mechanical tasks have the opposite effect, they can often be automated to some extent.

It doesn’t seem possible to estimate these creative tasks with reasonable accuracy at all, and not even proficiency or experience can help you here. Think about it, would it really have made sense to ask Einstein how long he would take to find a unified theory of physics?

The best you can do is estimate based on historical metrics which, in software as in the stock market, is not worth very much.

Further reading

Mathematical limits to software estimation

Powered by WordPress & Theme by Anders Norén