Just Use the Tags!

There I was at KubeCon, hanging out with some friends from Nebulaworks. We were talking about deploying container images.

“I always tell people to use the SHA256 digest of containers. That way you know exactly what you’re deploying.” Said one of my buddies.

“Sure, but that’s a pain.” I replied, not thinking it through. “Just use the tags! That’s what everybody does.”

Vitaly Korovin/Shutterstock.com

90 minutes later we’re sitting in a talk about container registries and some of the risky things you can do with them.

“You should always deploy by digest.” Said Jon, one of the presenters. “When you deploy by tag, you’re basically piping curl into bash in production. Don’t do that.”

To show what’s possible, they implemented a chat service using pushes and pulls from a registry. “It’s built on what Jon has advised us to never do: pull by tag.” Jon’s colleague explained. This was just a fun example, but it was also a scary illustration of the scope of what a malicious user could do under the right circumstances.

My friend leaned over and loud-whispered “SHA256!”


“Just use the tags!” was a naive response.

I should have asked more questions. “Everybody deploys by tags, is that just a common mistake? A lot of registries have a feature that makes tags immutable, is that equivalent?” That would have left room for me to learn something.

There’s so much knowledge out there. You’ll inevitably miss something. No matter how much you know, you’re still usually better off asking questions than making statements.

You might also want to check out these articles:

The Real Cost of Walkouts

When workers dislike their work or their work environment enough, sometimes they walk out.

Maybe there’s a bully in the office. Maybe it’s just not a good fit. I’ve known folks who tried software development, hated it, then discovered they love operations. I’ve known others who tried operations, hated it, then discovered they love software development. Maybe it’s something unrecognizably different from these examples.

Photo Smoothies/Shutterstock.com

Whatever the cause, the most productive workers often quit first. A reputation for productivity creates opportunities, so it’s easiest for them to find alternatives. You can’t predict exactly who will leave first, but over the years I’ve noticed it’s often the core contributors.

Word also gets around. Companies have brands with their workforce just like they do with their customers. It’s common for workers to ask around about companies they’re considering. I’ve seen plenty of candidates decline interviews because they got too many negative reports.

Delivering product means hiring and retaining talent that’s able to deliver. The damage done by walkouts makes it hard to do both those things.

If folks are having a bad experience, prioritize working with them to understand and address the cause. If you wait for them to walk and reactively try to hire a replacement, the damage may already be done.

You might also want to check out these articles:

How to Survive Working from Home

In How to Work from Home, I shared lessons I’ve learned about staying productive when teams go remote. Working from home can impact more than just productivity, though. It can make me feel stagnant. Lethargic. Here are lessons I’ve learned about staying energized while I’m working from home.

Lesson #1: Dedicate a space for work.

Position yourself so you see a view that you only see when you’re in that space. It can be as simple as allocating a chair at the table facing the opposite direction as the chair where you eat. Don’t do anything except work in that space. Don’t work from anywhere except that space. The separation has helped me focus when I’m at work and relax when I’m not.

Lesson #2: Practice different personal habits when working.

For me, this means dressing business casual whenever I’m in my work space (ok, most of the time I’m in my work space). The basketball shorts stay in the closet until after dinner. This gives work a crisp feeling that distinguishes it from downtime.

Lesson #3: Don’t sit for long intervals.

It’s easy to sit all the time when you have a desk job. Working at home, it’s even worse. You don’t have to walk to the front to sign that form for HR. There’s no lunch group walking to the restaurant across the street. Nobody stands around in the break room chatting.

At home, I need replacements for the physical activity that office life used to create. I’ll walk a loop of streets near my house before lunch, do some lunges between emails, or stand in the kitchen doing dishes after each meal.

I’ve also noticed that even a tough workout doesn’t substitute for sitting all day. I can’t finish work, run for an hour, then plop on the couch. To stay energized, I have to move and stand throughout the whole day.

Lesson #4: Add color.

This is the most subjective habit I follow. It might not work for everyone. It worked so well for me that I figured it was worth sharing.

I was feeling cooped up this week, so I bought a bouquet of flowers and put them in my living room. I smiled a lot of times that day for no reason. A colorful print on the wall and a blue ink pen for the notepad on my desk have similar effects. Color makes the light feel brighter and the air seem fresher.

Enjoy skipping your commute!

You might also want to check out these articles:

The Channel Bully

This is a true story, but I changed details to respect everyone’s privacy.

A director created a chat channel for non-work conversations. People could share pictures or puzzles or whatever else. It was fun. Then the Channel Bully showed up.

First, someone posted a puzzle. Someone else wrote a possible solution. The Channel Bully attacked. “We already know you think you’re so smart you can solve everything, you don’t need to tell us again.”

Nebojsa Tatomirov/Shutterstock.com

Next, someone shared a funny pun. The Bully struck again. “Please be more careful to use correct English. This sentence needs the Oxford comma.”

Folks stopped using the channel. Around the same time, several quit. The channel attacks were just one symptom of a broader problem.

A few comments like these can destroy a team’s willingness to collaborate. If people sense that a predator is about to pounce, they’ll find a wall to put their back against and stand there on guard. It’s hard to work with your teammates when your back is against a wall on the other side of the room.

Collaboration happens when team members sense that their team protects them. It stops when they sense that their teammates will attack them.

You might also want to check out these articles:

Discuss or Instruct

A colleague asked me to review a presentation he was preparing for a conference. Let’s say it was about optimizing the performance of Car Brand engines. I Googled the conference. It looked like it was for mechanics who worked at performance tuning shops. His presentation opened like this:

Engines have fuel injection systems that regulate the flow of fuel into cylinders. Car Brand engines have an unusually customizable system that enables you to…

At a conference of professional mechanics, the first sentence isn’t necessary. They already know what fuel injection is.

I had basically two options to give this feedback: discuss it as a potential change, or instruct him to make the change.

To discuss the change, I’d say: “Will your audience need the first sentence? Seems like mechanics would already know what fuel injection is.”

To direct him to make the change, I’d say: “Remove the first sentence. Mechanics will already know what fuel injection is.”

I used the discussion approach. I try to do that most of the time. There are two reasons.


First, it leaves room for me to be wrong. What if my search took me to the website for the wrong conference and I misunderstood the audience? If it was an introduction to automotive technology for high school students, maybe even more information about fuel injection was needed (not less).

Second, I’ve found that people tend to experience conversations positively and instructions negatively. The discussion approach makes things positive for the person I’m helping.

There’s a tiny difference in wording between discussing and instructing, but it has had a big positive impact on my work. It leaves me room to make the mistakes that everyone eventually will. It makes it easier for others to receive my feedback.

You might also want to check out these articles:

What Code Review Can’t Do

When developers complete a feature, they (hopefully) submit it to their colleagues for review. GitHub does this with pull requests. Azure Repos have their own flavor. GitLab uses merge requests. There are many tools.

Code review increases the quality of contributions. It enables the team to have your back. Reviewers might catch mistakes you missed or share ideas you didn’t think of.

But, reviewers can’t guarantee code was written as well as they could have written it if they’d written it themselves. It doesn’t substitute for seniority.

That’s easy to see if we scale up to an extreme example. Five interns plus one senior reviewer doesn’t add up to six seniors. They actually add up to zero seniors because the one you have will be so busy with reviews and helping that they won’t get any work done.

Development is dynamic and creative. You stare into an ocean of tooling and a blank screen and invent a way to implement a feature. There are always many approaches. Some work well. Some don’t. Some seem like they will and then don’t. Sometimes you get halfway through your second approach before you finally realize what you should have done. It takes time and a lot of willingness to rework your own work. Reviewers aren’t spending that time and doing those reworks. They’re looking from a distance at something they didn’t write. They can’t catch everything.

A skilled reviewer can raise the quality of the code they review by 10%-15%. That’s huge value! But it’s also only a little bit of the total. Most of the value still comes from the skills of the developer doing the initial implementation.

You might also want to check out these articles:

Make Me a Ticket

Based on many true stories.

“Hey Steve, I got locked out of the time tracking system. I think my password expired. Can you reset it?” William asked.

“You need to make me a ticket first.” Steve answered. He worked in IT.


“A ticket. On the help desk website. I can’t do anything without a ticket.”

William left and found an old link to the help desk website. He tried to create a ticket. It asked him to enter his billing code, department code, date, and request.

He asked around for the codes. Everybody said they just left those blank. He made a face and did the same thing. He put in today’s date and “time tracking password reset” and clicked send. Four days later, he got a reply.

“We can’t always do these on the same day you ask. Tell me when you really need this done by so we can schedule it.”

William hadn’t even needed it that fast. He’d put in today’s date because that’s what he thought was supposed to go in the “date” field. He showed the reply to his boss, who showed it to Steve’s boss, who told Steve to get it done.


The next day, Rose couldn’t log in to the time tracking system…

There are a lot of things wrong with this, but there’s one thing that sums them up. Steve was looking for ways to say “no”. When you’re doing customer service, look for ways to say “yes”.

Steve really was required to have a ticket for everything he worked on, but he could have made one himself. “You need to make me a ticket first,” could have been, “no problem, one second while I make a ticket.”

Steve really couldn’t reset the password on the day William asked, but he could have explained why. “We can’t always do these same-day,” could have been, “I added this to the list, but the time tracking system is old and only gets updated on Wednesdays.” Even if Wednesday turned out to be longer than William wanted to wait, it’s the best Steve can do.

If you show that you tried, your William will often volunteer to find workarounds for whatever you can’t do: “Thanks dude. I’ll keep notes until then.”

Even if you can’t deliver the request, there’s usually a way to say at least a partial “yes”. Saying “yes” whenever you can makes it easier to say “no” when you have to.

You might also want to check out these articles:

Four Guidelines for Valuable Documentation

📃 We’ve written a lot of documentation for a lot of projects. We’ve also read a lot of documentation for a lot of projects and had mixed experiences with what it taught us. Across that work, we’ve found four guidelines that make documentation easy to write and valuable to readers. Hopefully they save you some time and some frustration!

All four come from one principle:

Documentation exists to help users with generic experience learn your specific system.

Generic experience is a prerequisite. Documentation isn’t a substitute for knowing the basics of the tooling your project uses, it’s a quick way for knowledgeable readers to learn the specific ways your project uses those tools.

Don’t Write Click-by-Click Instructions

❌ This is way too much detail:

  1. Go to https://console.aws.amazon.com/cloudwatch/home
  2. Click Log Groups on the left
  3. Type “widgets-dev-async-processor” in the search box
  4. Click the magnifying glass icon
  5. Find the “widgets-dev-async-processor” in the search results
  6. Click “widgets-dev-async-processor”
  7. Click the first stream in the list
  8. Read the log entries

It’s frustratingly tedious for experienced users. Users who are so new that they need this level of detail are unlikely to get much from the logs it helps them find.

This will also go out of date as soon as the CloudWatch UI changes. You won’t always notice when it changes, and even if you do it’s easy to forget to update your docs.

Use simple text directions instead:

Open the widgets-dev-async-processor Log Group in the AWS CloudWatch web console.

That’s easy to read, tells the reader what they need and where to find it, and won’t go out of date until you change how your logs are stored.

Limit Use of Screenshots

🔍 Searches can’t see into images, so anything captured in a screenshot won’t show up in search results. Similarly, readers can’t copy/paste from images.

Also, like click-by-click instructions, screenshots are tedious for experienced readers, they don’t help new users understand the system, and they’re impractical to keep up to date.

Most of the time, simple text directions like the ones given above are more usable.

Duplicated docs always diverge. Here’s a common example:

Infrastructure code and application code live in different repos. Engineers of both need to export AWS credentials into their environment variables. Infra engineers need them to run terraform, app engineers need them to query DynamoDB tables. Trying to make it easy for everybody to find what they need, someone documents the steps in each repo. Later, the way users get their credentials changes. The engineer making that change only works on terraform and rarely uses the app repo. They forget to update its instructions. A new engineer joins the app team, follows those (outdated) instructions, and gets access errors. There’s churn while they diagnose.

It’s better to document the steps in one repo and link 🔗 to those steps from the other. Then, everyone is looking at the same document, not just the same steps. It’s easy to update all docs because there’s only one doc. Readers know they’re looking at the most current doc because there’s only one doc.

This is also true for upstream docs. For example, if it’s already covered in HashiCorp’s excellent terraform documentation, just link to it. A copy will go out of date. Always link to the specific sections of pages that cover the details your readers need. Don’t send them to the header page and force them to search.

Keep a Small Set of Accurate Documentation

If you write too many docs, they’ll eventually rot. You’ll forget to update some. You won’t have time to update others. Users will read those docs and do the wrong thing. Errors are inevitable. It’s better to have a small set of accurate docs than a large set of questionable ones. Only write as many docs as it’s practical to maintain.

Writing docs can be a lot of work. Sometimes they just cause more errors. Hopefully, these guidelines will make your docs easier to write and more valuable to your readers.

Happy documenting!

Need more than just this article? We’re available to consult.

You might also want to check out these related articles:

A Checklist for Submitting Pull Requests

Reviewing code is hard, especially because reviewers tend to inherit some responsibility for problems the code causes later. That can lead to churn while they try to develop confidence that new submissions are ready to merge.

I submit a lot of code for review, so I’ve been through a lot of that churn. Over the years I’ve found a few things that help make it easier for my reviewers to develop confidence in my submissions, so I decided to write a checklist. ✔️

The code I write lives in diverse repos governed by diverse requirements. A lot of the items in my checklist are there to help make sure I don’t mix up the issues I’m working on or the requirements of the repos I’m working in.

This isn’t a guide on writing good code. You can spend a lifetime on that topic. This is a quick checklist I use to avoid common mistakes.

This is written for Pull Requests submitted in git repos hosted on GitHub, but most of its steps are portable to other platforms (e.g. Perforce). It assumes common project features, like a contributing guide. Adjust as needed.

The Checklist

Immediately before submitting:

  1. Reread the issue.
  2. Merge the latest changes from the target branch (e.g. master).
  3. Reread the diff line by line.
  4. Rerun all tests. If the project doesn’t have automated tests, you can still:
    • Run static analysis tools on every file you changed.
    • Manually exercise new functionality.
    • Manually exercise existing functionality to make sure it hasn’t changed.
  5. Check if any documentation needs to be updated to reflect your changes.
  6. Check the rendering of any markup files (e.g. README.md) in the GitHub UI.
    • There are remarkable differences in how markup files render on different platforms, so it’s important to check them in the UI where they’ll live.
  7. Reread the project’s contributing guide.
  8. Write a description that:
    1. Links to the issue it addresses.
    2. Gives a plain English summary of the change.
    3. Explains decisions you had to make. Like:
      • Why you didn’t clean up that one piece of messy code.
      • How you chose the libraries you used.
      • Why you expanded an existing module instead of writing a new one.
      • How you chose the directory and file names you did.
      • Why you put your changes in this repo, instead of that other one.
    4. Lists all the tests you ran. Include relevant output or screenshots from manual tests.

There’s no perfect way to submit code for review. That’s why we still need humans to do it. The creativity and diligence of the engineer doing the work are more important than this checklist. Still, I’ve found that these reminders help me get code through review more easily.


You might also want to check out these related articles:

How to Work from Home

Remote work can be highly productive, but that productivity requires different tools and habits than it does when you’re working together in an office. During years helping remote teams, I learned some lessons about what works when you’re all working from different places. I wrote this list for software development teams, but most of it will apply to other cases.

✔️ Lesson #1: Keep the ticket tracker up to date.

One of the hardest parts of managing a remote team is keeping track of what’s going on. Good managers keep an ear on their team. When the team leaves the office, those ears aren’t enough anymore. The best way to help them is to make the ticket tracker (like Jira or Trello) the source of truth for the team’s work. My rule is this:

If the boss needs to know the status of your work, all they should need to do is refresh the ticket board.

🗣 Lesson #2: Use chat software for discussions, not to communicate requirements.

It’s easy to take everything to your chat app (like Slack or Discord) as soon as you leave your shared office space, but I find that leads to dropped tasks. If it has to get done, it should be a ticket in the ticket tracker or at least an email that copies the project manager and manager of the project. A ping in a chat channel isn’t enough. It’s too easy to scroll past something critical. My rule is this:

If something that was only posted in chat gets missed, the miss belongs to the person who posted it.

👥 Lesson #3: Replace check-in meetings with a tool.

Home workspaces have background noise. Roommates coming home. Kids getting ready for school. When you’re working, a pair of headphones is enough to mask that noise and help you focus. On a call, that noise can be disruptive.

Frequent check-in meetings (like daily scrums) exaggerate competition for your workspace. They’re also repetitive, so they’re easy to replace with a text-based system. I’ve used Status Hero for this. It sends reminders, teammates can tag each other if they need help, and it tracks blockers. It instantly reduces the call schedule.

📫 Lesson #4: Use Inbox Zero.

Dropped emails are one of the biggest problems I’ve faced on remote teams. Things get missed, and chat channels pollute with “hey did you see that email…” messages.

Inbox Zero is a simple system of processing your email instead of just checking it. It’s great at making sure emails don’t get dropped.

If you’re on a thread with an action item, create a ticket on the board and archive the message. If someone sends you an important link, bookmark it and archive the message. Instead of letting messages pile up, process them and then get rid of them.

Enjoy skipping your commute!

You might also want to check out these articles: