I’m curious, in data pipelines how are folks validating correctness after? I know some orgs run validations. Aren’t you just going to write the same query in the validation test and the data pipeline?

If you have the same query in both what is the value?

These girls 9 years ago vs now

youtu.be/PDFH9AcPpyA?si…

That last riff is NASTY. They dropped a video about an eclipse on the week of a 18 year total eclipse cycle.

With all the rumours of Windows adding ads to the start menu, I hope someone sane grows a set of balls to stand up and halt that effort & realize the last thing the struggling Windows brand needs is for mass consumer outrage that will lose more $ than the ads make.

I can’t think of a worse use case of AI than this one.

These 3 young sisters from Mexico in The Warning are fucking fire.

The emotions captured in the last minute of this song is special.

youtu.be/s6b_FgQnXL8?si…

We can scale 1mill consistent transactions like never before. We can aggregate 1bill rows in 50ms like never before.

These are a few examples of things that break my brain and challenge my thoughts about systems design.

Is a cluster easier to run or smaller async workflow?

Once you learn consistency doesn’t need to be strictly linearizable across an entire data store across all tenants all records (this is silly for 99.99999999% of use cases) you start to see systems design differently.

I honestly can’t believe I fell for this scam back then.

Where DDD/CQRS/Event Sourcing community REALLY lost me was the common pushed notion so many reads are eventually consistent but event sourcing isn’t done right unless you use an event store that provides strict linearizable guarantees.

The world runs on read committed tho.

Imposter syndrome has become really bad for me. The more senior I get the more nervous I am to share my inner conflicts around systems design.

If there was 1 thing I tried to do on Twitter besides make people laugh it was to debate and learn in public. I want to bring that back

When I started listening to my mentors & orienting my knowledge around computing more than religious books & religious communities pushing ways to build sw my career took off.

I still struggle with the sync async tradeoffs at times. Sometimes you can scale sync. But should you?

One of the primary reasons I started to separate myself from the DDD CQRS/Event Sourcing communities wasn’t because I fundamentally disagreed biz modelling is a fine art, it’s because there was too much cult driven architecture around asynchrony.

So little sound computing IMO

There’s so much cult about asynchrony even in localized code not just distributed systems. Too much async and CPU cache invalidations go through the roof, prefetching and branch predictions suffer too I assume and performance shits the bed.

Now I know there’s a lot of flaws in these truncated half baked tweets. Like yeah a replication is another form of an asynchronous queue. But it gets the job done faster.

Also with the rise of scalable transactions it challenges my mind what and where asynchronous should exist.

It’s a LOT faster to replicate the row and batch update aggregations in a data store that excels at batch updates than to enqueue a background job in a database that’s slow at batch updates and aggregations.

The user has a much higher chance of a perceived instantaneous result

But if you make many of these decisions now your user experience is junk because so much of your experience is stale. But distributed systems must accept staleness they say! But the web is a big cache anyways they say!

So when I see an architect placing a queue for anything that seems slow I wonder many things. Think of a simple user experience where a user checks a flag and a report representing a large number of rows changes. This could be a BG job!

I really struggle to put this into words so give me some patience here.

I love message queues and other asynchronous infra but it’s also never been a better time for stronger transactional guarantees even across distributed systems. We’ve scaled consistency like never before.

I’m not really sure how to express what feels like an unpopular opinion but as time goes on I feel I’m debating more and more about synchronous operations vs asynchronous ops.

Some system designers lean into too much asynchrony IMO. Or the wrong types of asynchronous designs.

A new coroutine library I just released.

github.com/tidwall/neco

Women’s March Madness has been amazing. The girls are playing out of their minds even with all this extra spot light and pressure. The rivalries have been spicy. Kaitlyn Clark is something else.