Actually Temporal does have a way to avoid determinism called rainbow deployments.
If you're fine with deploying several versions of workers (and are on a reasonably new version) you can just avoid the determinism issue altogether with their k8s controller.
If you do need to have some long workflows, there is an explicit hook for "what happens to existing workflows on version upgrade".
But to be fair - none of the other orchastrators I used (like AirFlow) made me write workflows.IsNewCode/IsOldCode like temporal does. On the other hand AirFlow doesn't even have the capability to do that in the first place (or at least it didn't last I used it).
Well sure - that's essentially the same as wrapping the whole workflow in a version check for each version; copy-paste the whole lot and change the code wherever. It's still surfacing an issue that would otherwise be less visible on a system that did allow a worker on a new version to pull an old half-completed workflow
In the last two years, we built (with a team of 15, now 100) a billion dollar business on top of Temporal that performs business critical applications for fortune 500 companies. We couldn't be happier with temporal.
Determinism sucks, you do have to work hard and make everything idempotent in activities like we would for durable software anyway. The language we used was incorrect (Go) and has a lot of boilerplate compared to alternatives we later investigated (Python and TypeScript). Visibility can be slow and misses information. We needed to write our own APIs to work effectively with Agents for root-cause analysis of failures.
With all the caveats - Temporal is amazing, it feels much better than previous orchestrators I used like Prefect or Airflow. 100% would adopt again.
That is the real truth people are voicing when they say Temporal is heavy. They are really saying: Durable, reliable, distributed workloads are hard and it takes effort to manage! And that is true. I know of no systems that make that genuinely easy. It is a hard discipline. Maybe Temporal makes that harder than it should be, but I have no experience there.
There are no free lunches in this space. I have no idea how good or bad Temporal is since my usage is pretty small and isolated, but software rarely just works and impresses me and Temporal for my local machine orchestrating genuinely did. I think Netflix's conductor is another cool option, but I ended up with Temporal due to license.
Have you looked into DBOS? Same thesis: durable and reliable workflows are hard to manage -- it just doesn't have to be as hard as Temporal makes it be :)
Have not. For my workflows this was fine. Good to keep in mind thoUgh. I don’t plan to manage a truly distributed system with it. Plus my only reason to do so is professional and we rolled our own system here due to our size solutions like DBOS or Temporal would not work well.
- Temporal itself is written in Go and we use Go for our backend so we expected this to be a natural fit.
- Temporal makes writing activities in Go very explicit and boilerplatey
- This in turn makes testing more difficult than it needs to be often
- Temporal doesn't play well with Go's concurrency model at all (all stuff like goroutines needs to go through its special workflows.Go) a lot more often you have to write stuff that "appeases" temporal.
- The whole workflows.ExecuteActivity(...).Get(...) is weird, having futures in a language explcitly designed to avoid that is weird.
- All our compute isn't done on temporal workers anyway, its done (in another AWS account, owned by the customer) in batch compute (aws batch, lambda, ec2, whatever) so our temporal code isn't CPU heavy but is highly concurrent and needs a very high reliability guarantee.
- Compare that to temporal with TypeScript, where it's simple and easy to use the same code inside or outside of temporal. Testing is trivial and the code looks like "regular code".
Interesting. Perhaps the go sdk is in alignment with the go-principle of being explicit.
I’ll take care to review examples of goroutines etc under temporal before calling any shots.
Thanks
Rust is really fun to work with and the compiler is great, just make sure the rewrite takes compile times into account since larger projects often have to be organized in a way that makes compilation reasonably fast.
how long does it take to compile?
@jarredsumner: It's basically the same as in zig using our faster zig compiler. If we were using the upstream zig compiler, rust port would compile faster.
This is at least partially disingenuous. Zig is working on, and has already shipped for some situations, a faster compiler. Bun runs on an outdated version of Zig that doesn't include it.
Single compiles for sure. Where Zig is optimizing compilation is in the incremental compiler, which I've seen compile the compiler itself in an instant after a single line change.
Of course, that kind of speed is probably not interesting to some people if the AI is writing tons of lines of code before they go to the compilation step.
I found making single line changes in Bun’s zig code led to very long compiles compared to doing the same in Rust code. It was a while ago though and maybe I was doing something wrong.
I have had to do this, well over a decade ago now, when working at a place that was a pretty big deal in the node world, and node was still pretty new. They helped us.
I would imagine GH would do the same if its a high enough profile issue.
Yep, we had to do this recently with Renovate, where we had too many releases, and new publishing hit a size limit on the registry, so we needed support to help us unpublish a load of old releases
First minio and then localstack, as an open source maintainer I find that abandoning their community is bad faith. I totally get wanting to monetize but removing the free product entirely feels like such a betrayel.
Luckily, I've been vibing with Devin since this started having it build a cleanbox emulator on top of real s3 tuned for my specific use case. It's a lot less general but it's much faster and easy to add the sort of assertions I need in it. It's no localstack but for my limited use case it works.
It does feel like a betrayal. We live in a world where money is the main thing that matters and it's increasingly hard to come by and you need increasingly more of it (these are all designed policies, not emergent behavior). It makes sense that people don't want to do things for free unless they already have enough money.
Engineers who remained apolitical are now surprised the politics is bad.
Personally I think we should at least discuss changing this system into something that is more sustainable. Money is main thing, because it was decided that private property is more important than people. People like Joseph Stiglitz show that there's no such a thing as an invisible hand, even though he still believes in free market.
Yeah these moves will gain them a year or so but all these companies built on a "takes time to implement library" are all dead in the water. Localstack has nothing fancy, it just takes time to build. And that moat is gone, it's maybe 4 weekends of token quotas I wouldn't use anyway.
You are not misunderstanding anything, I use Go and Rust/TypeScript in my daily work and you are correct - it is the OP that does not understand why people use lockfiles in CI (to prevent minor updates and changes in upstream through verifying a hash signature).
They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages. See the previous discussion here: https://lobste.rs/s/exv2eq/go_sum_is_not_lockfile
> They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages
Thanks for that link.
Based on reading through that whole discussion there just now and my understanding of the different ecosystems, my conclusion is that certainly people there are telling Filippo Valsorda that he is misunderstanding how things work in other languages, but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.
This subthread to me was a seemingly prototypical exchange there:
Someone in that subthread tells Filippo (FiloSottile) that he is misunderstanding cargo behavior, but Filippo then reiterates which behavior he is talking about (add vs. install), Filippo does a simple test to illustrate his point, and some others seem to agree that he is correct in what he originally said.
That said, YMMV, and that overall discussion does certainly seem to have some confusion and people seemingly talking past each other (e.g., some people mixing up "dependents" vs. "dependencies", etc.).
> but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.
I don't get this impression. Rather, as you say, I get the impression that people are talking past each other, a property which also extends to the author, and the overall failure to reach a mutual understanding of terms only contributes to muddying the waters all around. Here's a direct example that's still in the OP:
"The lockfile (e.g. uv.lock, package-lock.json, Cargo.lock) is a relatively recent innovation in some ecosystems, and it lists the actual versions used in the most recent build. It is not really human-readable, and is ignored by dependents, allowing the rapid spread of supply-chain attacks."
At the end there, what the author is talking about has nothing to do with lockfiles specifically, let alone when they are applied or ignored, but rather to do with the difference between minimum-version selection (which Go uses) and max-compatible-version selection.
Here's another one:
"In other ecosystems, package resolution time going down below 1s is celebrated"
This is repeating the mistaken claims that Russ Cox made years ago when he designed Go's current packaging system. Package resolution in e.g. Cargo is almost too fast to measure, even on large dependency trees.
Addy's users have been developers and Google has been very responsive in the past. I was usually able to get a hold of someone from teams I needed from Chrome DevTools and they've assisted open source projects like Node.js where Google doesn't have a stake. He also has a blog, books and often attended conferences to speak to users directly when it aligned with his role. I agree about the general Google criticism but I believe it's unjustified in this particular (admittedly rare) case.
If you're fine with deploying several versions of workers (and are on a reasonably new version) you can just avoid the determinism issue altogether with their k8s controller.
If you do need to have some long workflows, there is an explicit hook for "what happens to existing workflows on version upgrade".
But to be fair - none of the other orchastrators I used (like AirFlow) made me write workflows.IsNewCode/IsOldCode like temporal does. On the other hand AirFlow doesn't even have the capability to do that in the first place (or at least it didn't last I used it).
reply