I’m 15 years in and I feel basically the same. I end up making a feature or change, then going back and trying to split it into chunks that are digestible to my colleagues. I’ve got thousands of lines of staged changes that I’m waiting to drip out to people at a digestible pace.
I yearn for the early stage startup where every commit is a big change and my colleagues are used to reviewing this, and I can execute at my actual pace.
It’s really changed the way I think about software in general, I’ve come around to Rich Hickey’s radically simple language Clojure, because types bloat the refactors I’m doing.
I’d love to have more of you where I work, is there some way I can see your work and send some job descriptions and see if you’re interested?
> I end up making a feature or change, then going back and trying to split it into chunks that are digestible to my colleagues.
If you are doing this AFTER you've written the code, it is probably way easier to do it as you go. It's one thing if you have no idea what the code will look like from the beginning -- just go ahead and open the big PR and EXPLAIN WHY. I know that I'm more than happy to review a big PR if I understand why it has to be big.
I will be annoyed if I see a PR that is a mix of refactoring, bug fixes, and new features. You can (and should) have done those all as separate PRs (and tickets). If you need to refactor something, refactor it, and open a PR. It doesn't take that long and there's no need to wait until your huge PR is ready.
Solving creative problems is often iterative, and one things I'm very concerned about when doing engineering management is maintaining momentum and flow. Looking at latency hierarchies is a really good example, you have registers, then cache, then memory, SSD, network etc. and consulting with another human asynchronously is like sending a message to Jupiter (in the best case).
So, with an iterative process, the more times you introduce (at best) hour long delays, you end up sitting on your arse twiddling your thumbs doing nothing, until the response comes back.
The concept of making PRs as you go fails to capture one of the aspects of low-latency problem solving, which is that you catch a problem, you correct it and you revise it locally, without exiting that loop. Which is problematic because not only have you put yourself in a situation where you're waiting for a response, but you've stopped half-way through an unfinished idea.
This comes back to 'is it done', a gut feel that it's an appropriate time to break the loop and incur the latency cost, which for every developer will be different and is something that I have grown to deeply trust and and adjust to for everybody I work with.
What I'm getting at is the iterative problem solving process often can't be neatly dissected into discrete units while it's happening, and after we've reached the 'doneness' point it takes much more work to undo part of your work and re-do it than it took to do originally, so not only do you have the async overhead of every interaction, but you have the cognitive burden of untangling what was previously a cohesive unit of thought - which again is another big time killer
What I mean is, you make your commit, cherry pick it over to the main branch, and open a draft pr. It doesn't break your flow, it doesn't stop anything, and is pretty quick. It also gives you a quick gut-check to see the PR; if you think your team members won't understand "why" it needs to be refactored, then you have one of two problems:
1. your refactoring is probably going in the wrong direction. Team members will be able to help here more than ever. Let them bikeshed, but don't stop working on your main refactor yet. Revist later and integrate their changes.
2. the PR is too small. it will have to be part of a larger PR.
In my experience, people tend to have the first problem, and not the second one, but they think they have the second one. There are many of these "massive refactoring" PRs I've reviewed over the last 20 years where the refactoring makes the code worse, overall. Why? Because refactoring towards a goal (implementing a feature, fixing a bug, etc.) doesn't have the goal refactoring should have: improving code maintainability. So, the refactored code is usually LESS maintainable, but it does what they wanted.
If you make refactor PRs as you go, do you end up merging redactors towards a dead end and then--once you realize it's a dead end--merging even more refractors in the other direction?
I usually wait until I have the big PR done and then merge redactors towards it because then at least I know the road I'm paving has a workable destination.
This is why I design the heckin' huge change at the start, and then cherry pick the actual change (and associated tests) into a ton of smaller PRs, including "refactor here", "make this function + tests", "make this class + tests", "integrate the code + tests", and so on, as many times as necessary to have testable and reviewable units of code.
If I went about and made a ton of changes that all went into dead ends, honestly, I would get pretty demoralized and I think my team would get annoyed, especially if I then went through and rolled back many of those changes as not ending up being necessary.
These same people also want to see your GitHub history filled with deep green come review time. I start to wonder if they think high levels of GitHub activity is a proxy of performance or if it’s a proxy of plying the game the way they insist you play.
I yearn for the early stage startup where every commit is a big change and my colleagues are used to reviewing this, and I can execute at my actual pace.
It’s really changed the way I think about software in general, I’ve come around to Rich Hickey’s radically simple language Clojure, because types bloat the refactors I’m doing.
I’d love to have more of you where I work, is there some way I can see your work and send some job descriptions and see if you’re interested?