I've operated petabyte-scale ClickHouse clusters for 5 years
47 points by adastral 4 days ago | 12 comments

zbentley 20 minutes ago
> For loads with over 20k rows/s and people pushing changes, you may need a full-time person to handle the cluster and take a look at the crazy queries people are going to write.

I think this was a benefit of DBA culture in previous eras. Not that the DBAs were specifically necessary to write good queries (often they'd need to work with application teams to guide them towards schemas/behavior that worked well) or to maintain the database (managed DB offerings obsolete a lot of this work), but because they functioned as gatekeepers and rate-limiters of what queries and schemas could exist.

In that mode, DBAs functioned a bit like a human/process version of a thin microservice wrapping database access functionality. A big benefit was that the rate of change of queries/schema changes/access patterns was controlled and had a higher probability of being reviewed and thought about by humans before it went live. This also resulted in an increased end-database-user culture of trying to make existing schemas/query patterns work before jumping straight to bespoke access patterns. That culture's not what you want as e.g. a startup or pro-rapid-big-refactors shop, but it is what you want when your DB reliability needs or query rate/dataset size are high.

I don't think it's a given that a gatekeeper team is worth the overhead and cost; that's situational. I do think that the code version of that team (aforementioned microservice that wraps DB accesses/schema changes and nothing else) is usually not worth the cost. In my experience, that pretty much always reduces reliability and free performance gains that come from using direct DB clients from user code.

reply
bushbaba 14 minutes ago
Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases
reply
zbentley 4 minutes ago
It's one thing to be able to scale database compute/storage; it's another thing to be able to partition it. It's extremely common for bad queries/access patterns to cause noisy-neighbor effects on other simultaneous accesses to the database, to the extreme of knocking the whole database over with timeouts/OOMs/etc.

Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting events as well) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.

reply
tmpz22 5 minutes ago
Instead we're... listen to this... we're going to take a software developer right. Just a normal developer right. We're going to make them be the database expert right. And the cloud expert. And we're going to put them on call. We're going to have them debug linux logs, and optimize our AWS costs. They'll be there for client escalation work. And big sales calls. From time to time we'll even have them do front end work.

And get this. We pay them the exact same.

reply
bradleyy 42 minutes ago
I just wish Amazon would offer it as an RDS DB; it'd make my life so much easier.
reply
fidotron 2 minutes ago
Do any two teams actually operate it in anything like the same way though?

What I saw of it, especially some years ago, was it was highly particular, and everyone had their own odd habits built around running it, ingestion, querying, everything, to the point I suspect there are a non trivial number of companies using it where it is actually the core operational expertise of the company, despite them all appearing to be in totally different domains.

reply
andriy_koval 25 minutes ago
You can use actual CH Cloud on AWS?..
reply
trynotsober 17 minutes ago
When replaying customer queries against the next version, how do you compare results for queries using now() or approximate aggregates? Curious how you separate expected differences from actual regressions.
reply
lucrbvi 2 hours ago
That's a lot of ®, curious how ClickHouse® Inc. is treating the use of its name by others ... Hopes it's not like Oracle with JavaScript
reply
doe88 22 minutes ago
It's defensive language for sure, i don't know how much it adds of protection in reality, but i nonethelesss sympathize with the author if he feels the need to protect himself that way or signaling the risk he takes.
reply
HatchedLake721 20 minutes ago
They sell managed ClickHouse so I suspect it’s a precaution
reply
walthamstow 57 minutes ago
As an aside, I was stuck when turning on the Fulham v Crystal Palace game last week to find that Fulham have ClickHouse on their shirts this year, and Palace have Temporal AI. Talk about my worlds colliding.
reply