• Interns/New grads are not a resource, they are a cost center
  • What layer you build a capability into is crazy important
  • Ubiquity wins, because HTTP/HTML/X509/REST/JSON are just awful technology
  • REST is a mess, just stop and use GRPC and export to REST if you must
  • Kubernetes is to Borg what Frankstein is to the Dali Lama
  • Benchmarks matter
  • Don't refactor a bad code base, rewrite it

Inters/New Grads are not a resource, they are a cost center

There is this false belief in management circles that interns are a free labor pool. They are a cost center.

Much like new grads, they have no idea what they are doing. Yes, they can do a linked list, but that has about zero use. Unlike new grads, any progress you make with them is lost when they leave.

My new quote is: "show me an intern that doesn't double my work". And I'm actually fine with that, if companies would treat interns as students there to learn (especially unpaid internships).

Let's invest in our interns abilities. That's what internships should be about.

I recently had to spend months taking some intern code that was awful and turning it into something useful. Not the interns fault, when had they ever had to write high performance and rock solid code? Never. Like many teams, they were just put on something no one else wanted to/had time to tackle.

I've only once seen an intern project have any long term use. That project was a huge security issue that took years to remove. Just a disaster.

Now look, someone is going to say "hey, this isn't universally true, ....". Of course not. I mean, I think Russ Cox was like 16 working at Bell Labs and a Google intern for Jeff Dean. He was probably better as an intern than I will ever be. If that story is true, he is the exception that proves the rule.

What layer you build a capability into is crazy important

I've thought about this for years, but in the past year I've seen this in so many places, it has changed my design process.

Take CPU design as a good example: RISC design vs CISC. In essence, both chip designs do the same thing.

Note: this is next part is going to be a gross oversimplification.

Pure CISC design buckets different instructions together to make it more convenient for the assembly programmer.

RISC has individual instructions and pushes that type of work into the compiler.

Because of CISC design, things like pipelining are not possible.

Optimizations in the compiler were harder or not possible.

It wasn't that things were not going to get bundled up and optimized in a RISC design, it was where that was going to happen. And that made a huge difference. Intel/AMD has had to spend decades integrating RISC into their CISC chips without breaking backwards compatibility.

HTTP1/2 is another example. Choosing TCP as the layer 4 protocol meant that all the sequencing/transport/negotiation/... was done in the OS. That choice meant that any improvement would have to ported to all OS stacks without breaking TCP (good luck with that).

HTTP3 will use UDP, moving all that logic into the HTTP libraries. Now minor version changes can add new optimizations that can be taken advantage of immediately. QUIC will be used in HTTP3, but HTTP4 will be able to switch that out for something new without complex code changes or give a range of transport options.

Choices on where something goes are difficult to undo once they are in use and can cause skyrocketing costs/complexity. You need to choose wisely.

Ubiquity wins, because HTTP/HTML/X509/REST/JSON are just awful technology

It is unfortunate, but first movers advantage is real. And we truly love to Frankenstein our tech to make it do things it shouldn't, because it is ubiquitous.

HTML wasn't bad, when it was designed to display text and pictures on a page. But now it is a being used for UI design. I have some old Macintosh computers from the 80s that respond faster than UIs built on chromium, and memory use is measured in Kilobytes.

The HTML 5 spec is a joke: you could eliminate half the tags and with CSS you can achieve the same result.

I could go into the REST (tech humor is so bad), but if you look just a little, they aren't great or left out key components.

But they are first movers and now ubiquitous. We keep building on them, because they are there and accepted. And people don't want to learn anything new.

Having something better is rarely good enough, otherwise Chromium based apps wouldn't exist and we'd all be using Plan9.

REST is a mess, just stop and use GRPC and export to REST if you must

I had never actually used REST much until I left Google. I knew we exported APIs in REST (because it was ubiquitous), but rarely did I touch it directly. If I had to talk to something exporting REST, usually I did it through a native API.

Now that I've had more experience with REST, I loathe it. While officially you can transport whatever you want, it is going to be JSON or XML. I'm going to be talking about the most prevalent, JSON (XML is way worse).

JSON provides two advantages over Protocol Buffers:

  • Human readable text
  • Schemaless

Protocol buffers have:

  • Schemas that compile in every language (I only have to write a data structure once)
  • Binary encoded and can convert to JSON (meaning they are smaller and faster)
  • Without special compilers that are hard to use, protocol buffer encoding is at least a magnitude of 10 faster than JSON
  • Support for the []byte, so you don't have to base64 encode a []byte into a string, which is costly (yeah, I know about BSON, but that isn't JSON)

GRPC is significantly faster, works easily over multiple transports.

With REST I must discern things like what HTTP method a REST call will be over (GET/POST/DELETE/OPTION/...). Versioning in REST is no easier than adding new calls or versioned instances in GRPC. People write books on the best practices (that alone should make you run for something that should be easy).

Even better, GRPC can export in REST too. You can have it all (well, except GRPC proto compilers are ridiculously hard to setup).

Kubernetes is to Borg what Frankstein is to the Dali Lama

When I left Google, I was sold on the whole containerized way of running things. Borg is lightyears ahead of every other cluster orchestration project.

Borg doesn't let you do everything. It is designed to run specifically built applications that are containerized. You don't get Docker images with whatever OS stuff you feel like running that day. The OS is always Google's internal OS. You don't get access to whatever binaries you want to install. You don't get go use whatever security you want. Your RPC system is always going to be Stubby (GRPC internal to Google). Your cluster file system is going to be the only one allowed. Period.

Those limits are freeing. You simply need to have resources to run your jobs and deploy them. Your binaries are packaged up and you just need to say what is going to get run.

So naturally, I've used Kubernetes after I left.

Everything about Borg I liked is gone in Kubernentes. It is trying to solve everyone's problem and solves no one's problem.

It is easy to kill your jobs. Its hard to do things like update a single instance. Service meshes???? Really????

Helm? Great, I can kill all my cluster MySQL databases at the flick of my heml config.

Security, what security? Oh, right, the bring my own model that is just crazy hard.

Need it to work with special cloud sidecars (like special identity services)? Well, that's going to be a fun thing.

Upgrades that change the config language so that your jobs won't run anymore. Perfect.....

And btw, love YAML over the Borg config language, NOT!

And that is on all cloud providers. Google has trouble running Kubernetes (https://blog.dave.tf/post/new-kubernetes/), which should tell you something.

Benchmarks matter

If you are releasing something, benchmark it. Especially things such as API clients. You should know how it compares to other language clients.

Not only will this save you some embarrassment, I've been able to find flaws in backend implementations that allowed me to double the performance of major services with < 10 lines of code.

Don't refactor a bad code base, rewrite it

Now, before I get a bunch of hate mail, everything is situational dependent.

That intern's code I was re-writing, I did it in piecemeal. I estimate this cost about 10x the effort it would have taken had I just re-wrote it from scratch.

We did this to help foster understanding between what I was doing and the future maintainers/contributors.

This caused me to mentally lock into certain design choices that I didn't understand and had to rethink later.

Had I approached it from the ground up, I believe the implementation would be better. I think refactoring made it hard to see the forest because of the trees.

And I don't think it fostered better understanding for the future maintainers and other contributors.

Like everything, take these with a grain of salt

These are things to me that are valuable lessons. To you it might be junk.

The world is full of voices, be careful which ones you listen to. Including me.