I've operated petabyte-scale ClickHouse clusters for 5 years
47 points by adastral 4 days ago | 12 comments
bradleyy 42 minutes ago
I just wish Amazon would offer it as an RDS DB; it'd make my life so much easier.
replyfidotron 2 minutes ago
Do any two teams actually operate it in anything like the same way though?
replyWhat I saw of it, especially some years ago, was it was highly particular, and everyone had their own odd habits built around running it, ingestion, querying, everything, to the point I suspect there are a non trivial number of companies using it where it is actually the core operational expertise of the company, despite them all appearing to be in totally different domains.
trynotsober 17 minutes ago
When replaying customer queries against the next version, how do you compare results for queries using now() or approximate aggregates? Curious how you separate expected differences from actual regressions.
replylucrbvi 2 hours ago
That's a lot of ®, curious how ClickHouse® Inc. is treating the use of its name by others ... Hopes it's not like Oracle with JavaScript
replywalthamstow 57 minutes ago
As an aside, I was stuck when turning on the Fulham v Crystal Palace game last week to find that Fulham have ClickHouse on their shirts this year, and Palace have Temporal AI. Talk about my worlds colliding.
reply
I think this was a benefit of DBA culture in previous eras. Not that the DBAs were specifically necessary to write good queries (often they'd need to work with application teams to guide them towards schemas/behavior that worked well) or to maintain the database (managed DB offerings obsolete a lot of this work), but because they functioned as gatekeepers and rate-limiters of what queries and schemas could exist.
In that mode, DBAs functioned a bit like a human/process version of a thin microservice wrapping database access functionality. A big benefit was that the rate of change of queries/schema changes/access patterns was controlled and had a higher probability of being reviewed and thought about by humans before it went live. This also resulted in an increased end-database-user culture of trying to make existing schemas/query patterns work before jumping straight to bespoke access patterns. That culture's not what you want as e.g. a startup or pro-rapid-big-refactors shop, but it is what you want when your DB reliability needs or query rate/dataset size are high.
I don't think it's a given that a gatekeeper team is worth the overhead and cost; that's situational. I do think that the code version of that team (aforementioned microservice that wraps DB accesses/schema changes and nothing else) is usually not worth the cost. In my experience, that pretty much always reduces reliability and free performance gains that come from using direct DB clients from user code.
Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting events as well) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.
And get this. We pay them the exact same.