I don't have strong opinions about these PEPs, but there's some background on how python is governed that the pydantic maintainer omits (as does most of the discussion here).
I know people don't like the specific decisions python has made recently (me too!), but the process for deprecation and implementation once those decisions are made is at least clear and unambiguous, thanks to backwards compatibility policy adopted in PEP 387, i.e. way back in 2009.
"Wait for the warning to appear in at least two major Python versions. It's fine to wait more than two releases."
PEP 563, the one that apparently breaks pydantic, was accepted during Python 3.7. This was before pydantic reached 1.0 and before fastapi even existed. It was eligible for full implementation in 3.9 according to the backward compatibility policy, but the plan right now is for it to be implemented in Python 3.10. Python 3.10 is due to code freeze in three weeks. The maintainers of these projects have had over three years to raise these issues.
pydantic and fastapi both have outstanding documentation that is a credit to their projects and to the python community. One side-effect of this, however, is they have some inexperienced and frankly excitable users, who perhaps aren't aware how python itself is developed. These are the users their maintainers (see also https://twitter.com/tiangolo/status/1382800928982642692) are unleashing on the python core maintainers and release managers literally days before a release that will implement a change they've known about for over three years.
I hope this can be worked out. PEP 649 looks promising. Maybe PEP 563 should wait for python 3.11. And pydantic and fastapi are great! But this is, simply put, a shitty way to treat volunteer open source maintainers at the absolute busiest and most stressful time of the release cycle.
The fact you are warned in advance that something bad is going to happen doesn't make it less bad. Just more manageable.
The elephant in the room is that the core devs, after Python 2.7, have taken the very bad habit of breaking compat. For such a popular and core tech a Python, this is not responsible. It erodes trust. It also raises the cost of producing good software tremendously.
There are so many moving parts in tech now a day you can't spend you time, as an open source dev, tracking everything that break all the time. Nor fixing it. We have limited resources, energy and spare time.
And what if I miss it? It's human after all.
A project of the scale of Python, that is so central, so essential, should not break on me more than once in a decade, at least willingly.
The way the issue is dealt with currently is very short sighted. It's not about the short term cost of having some teams having to pay the price of a rewrite.
It's about how bad it is for the Python community, ecosystem and project on the long run.
Compatibility is great if you're building on solid, or at least good enough foundation. But that's not Python, the foundation is dubious, or even bad. Extensive C API which prevents advanced JIT efforts, tons of dead libraries in standard library.
Also, even having solid foundation and big company backing, Java is also regularly deprecating and removing things.
Full, unquestionable compatibility is good only if you're committee wizard in your tower and don't care about users beyond big corporate at all, aka C++ model.
Oh, and one thing about Python: it's very understaffed. They have like 3 fully paid devs at all? For a project of this impact, it's really bad. Compare this to tens of thousands of man-years spend on JS.
I agree, and I'm not advocating to freeze the python language, nor don't understand how understaffed the python core devs are.
But there are strategies that are in used or have been in used that we could draw inspiration from:
- async/await didn't need to become keywords so fast. True/False were not for 15 years.
- changing the default encoding is behind env var flag.
- changing the way the c api is done is currently made doing the separate HPy project. It's not breaking anything.
So there were instances where people intelligently decided to move the language forward, but stopped and remembered Linus wisdom of "we don't breaking userland". And they did improve things without breaking others.
Guido used to be very strict on that too. But after so many years, we can't expect him to spend all his energy still policing every aspect of the project, especially after he stepped down as a BDFL.
I looked at the psf budget last normal year, iirc it was like five million in income. They choose to spend most profits on outreach and teaching. That’s their decision but then they also complain about being under resourced, leading to compat breaks on minor versions.
I don’t believe it is necessary, compatibility should come before most other activities.
1. 3.x to 3.x+1 is referred to as a major version change in Python. It's the same in Go, where 1.x to 1.x+1 is also a major version change (https://golang.org/doc/devel/release.html#policy). Python does not use semantic versioning, which may be why your expecations do not align with reality. And they do not introduce breaking changes without two major version bumps (i.e. two years) of public warnings about their plans (in that sense they are more conservative than semver).
2. This breaking change has not happened yet.
3. If it does happen, it won't be because the PSF is under-resourced. It will be because a long formal process occurred in back in 2017 to ratify PEP 563, and that processes success started a timer on its incorporation into python 3.9 or later, consistent with Python's deprecation policy. You can argue that this was a bad decision, but "spending most profits on outreach and teaching" is not the reason it happened.
4. The reason resources are a problem is because the original link here is someone encouraging his users to harass the small (too small!) group of uncompensated volunteers who manage python releases, three weeks before the release, behavior for which he has since apologized.
My expectations align with industry practice, and so should yours.
Python is welcome to play games with their version numbers, it doesn't change expectations however. That they made mistakes with unicode at 3.0 doesn't mean there should be breaks every X.Y release instead.
> A project of the scale of Python, that is so central, so essential, should not break on me more than once in a decade, at least willingly.
They have been unambiguous about their breaking changes policy for twelve years: you get at least two major versions of warning (i.e. two to three years), possibly more. And old major versions are supported (bugfixes, security fixes, and occasional non-breaking backports) for five years.
If that's not what you're looking for (which is totally fine!) then python is probably not a good fit for your use case.
> The elephant in the room is that the core devs, after Python 2.7, have taken the very bad habit of breaking compat. For such a popular and core tech a Python, this is not responsible. It erodes trust. It also raises the cost of producing good software tremendously.
You know, the core devs cannot make it right for everyone. For each its-a-breaking-change-complaint there is a similar amount of we-want-shiny-new-things-and-my-teammates-want-to-move-on-to-shiny-new-language-X complaint.
So its clear to me that Python in order to stay relevant needs to evolve, add features, break occasionally with new releases.
There was once the idea of combining a couple of breaking changes in one release (I would call this the 'once-in-a-decade' breaking change you seem to find acceptable), that was Py2->Py3 and it was a shitshow, in part, because too many changes happened at once.
So I am grateful that we now get the occasional breaking change, one change at a time. I think it is the right way if Python aims to be still relevant in 10 years from now.
See my answer to your sibling. It's not an "or" proposition.
The 2 to 3 transition was hard because we did an abrupt cut. We could have had few transitioning versions that supported both worlds, then disabled the old ways but with a switch to put them back on if needed for a while, and allow for a more peaceful transition.
It’s very much an “or” situation from a practical standpoint. Maybe not exclusively or, but it’s unrealistic to say one does not significantly affect the other. Python core dev (CPython or otherwise) is already thin on resource for a long time, and you can’t just keep adding things without cleaning up old cruft. Parties most involved in development of the Python language (CPython devs, the Steering Council, etc.) value sustainability most, so it makes sense they decide to drop old stuff when they see no need of it. On the other side, parties that value stability more (e.g. libraries authors) need to actively lobby their goals to the core devs so they can assess the situation correctly and find the right balance between the two. You can’t just sit there and expect people to understand your needs; you have to tell them, which is what pydantic failed to do in this situation.
Python 3 was necessary for cleaning. Having default new style classes, super() and so on are goodies, not necessities.
It was necessary for the things that are were not easy to do without breaking compat such as unicode, fixing comparison bugs, etc.
Now none of the stuff that broke compat in 3 after the initial release were things that were necessities. They were goodies. You don't need make from __future__ import annotations as default. You don't need to make async/await keyword.
> You don't need make from __future__ import annotations as default. You don't need to make async/await keyword.
By the same logic unicode_literal and print_function should have never been the default? Yeah I guess you can make that case, but I’m not in your camp. Sorry, goodbye.
I don’t agree because it was the Unicode change that made it difficult, every thing else was relatively trivial. No further huge breaks are even being considered. So kind of a moot point anyway.
> The fact you are warned in advance that something bad is going to happen doesn't make it less bad. Just more manageable.
Exactly, it was very manageable. They could have avoided writing a whole library (FastAPI) knowing it was going to break inevitably. They could have raised concerns during three years.
1 - FastAPI was created before the change was announced.
2 - You assume they knew about it. It's a lot of work to create FOSS, it's very easy to miss those things given the huge amount of information you have to check for compat and security. Hell, I certainly don't assume my main dependency, which is a rock solid language no changing the major version and used by millions is going to break my project. I don't read each line of each changelog of each version upgrade of each of my dependency.
3 - Even if they knew about it, they may never have realized it would have broken something before people told them so.
4 - They may even have noticed, 3 years ago, but forgot about it. It was 3 years ago!
It is surprising that they missed this change, though. Given how core type hints are for their projects and how broadly publicised the changes and developments in type hints were. I'd be surprised if there weren't any issues raised against their projects about those. It feels it'd be harder to miss it than it'd be to stumble upon it.
Also, weird for them to forget that there is a change that will break the foundations of their libraries. You'd think they would have that in the back of their minds and, being stakeholders, even being part of the discussions. Like you said, it's FOSS and, most importantly, python is a community project.
Irrelevant here, AIUI, since pydantic had bugs for this since 2018 but only raised issues with core 3 weeks before 3.10 feature freeze.
They knew it was coming, they were able to test it because it waa available behind a __future__ import, they knew it didn’t work for them, and they sat on it until it waa virtually impossible to resolve well.
I must say I've just spent quite some time looking at both PEPs and discussions and I'm strongly in favor of PEP 563 right now... it does improve the performance of typed modules substantially (which is a big downside for me in typed codebases) -- PEP 649 improves the situation but still adds a considerable overhead to the import time.
Also, I don't see what will be made impossible at the pydantic side... from the bug list that's shown at https://github.com/samuelcolvin/pydantic/issues/2678, yes, the code needs to be adapted, but I don't see what's made impossible (maybe at some point the pydantic devs will actually say what the real problems are as so far no real concrete info on that was given).
Heck, they could even probably not even use `typing.get_type_hints()` (which currently apparently does an internal `eval()` and treat the annotation string however they'd like -- maybe more work, but then, absolute freedom to treat it as they see fit).
So, my plead goes the other way around: please don't make all projects which don't require runtime info from annotations pay the price for the few use cases which do use them (but don't make those impossible either -- but my take so far as that this is not the case).
> I hope this can be worked out. PEP 649 looks promising. Maybe PEP 563 should wait for python 3.11. And pydantic and fastapi are great! But this is, simply put, a shitty way to treat volunteer open source maintainers at the absolute busiest and most stressful time of the release cycle.
"Quick update, the Python steering council, and core devs are fully aware of this and supportively helping figure it out. You can read all the conversations here and in the issue, but please restrain from commenting, tagging, or writing to them directly about this, it's already being handled with care. And [my] Tweet is already causing more extra work for them than helping."
"I only heard about the debate about PEP 649 on Wednesday night, and given the extremely narrow window to get my point heard, I banged the drum in every way I could think of. The lesson here is "careful what you wish for", my message has been amplified more than I expected. I'm sorry if that has wasted anyone's time, even more so if it reduces the chance my request gets a positive reception. You're also right that i should have engaged with python-dev long ago about this, I'm sorry I did not. Lesson (hopefully) learnt."
(More detail and explanation in his comment, but that's the bit that's salient to the point I was making in this thread.)
Both authors saw PEP-649 as a solution, with no need to raise alarm bells. If you follow the discussion of the last proposal, you’ll see that it’s acceptance is now in doubt, and it’s inclusion in 3.10 seems unlikely. Both authors have been active in that discussion.
The issue is not that they had no warning, it’s that the proposed solution is being pulled out from under them.
> I know people don't like the specific decisions python has made recently (me too!), but the process for deprecation and implementation once those decisions are made is at least clear and unambiguous, thanks to backwards compatibility policy adopted in PEP 387, i.e. way back in 2009.
> "Wait for the warning to appear in at least two major Python versions. It's fine to wait more than two releases."
> PEP 563, the one that apparently breaks pydantic, was accepted during Python 3.7. [..] It was eligible for full implementation in 3.9 according to the backward compatibility policy
Can you point me to the warning that was raised in Python 3.7, 3.8, and 3.9? I don't find it on a quick look.
> Can you point me to the warning that was raised in Python 3.7, 3.8, and 3.9?
AIUI, No warning was raised because the PEP 484 functionality that was changed was at the time identified as provisional, and not subject to the deprecation policy, which excludes provisional features.
The lack of warning, but not rationale, is explicitly noted in PEP 563.
If you're looking for runtime warnings, me neither. I think the definition of "appear in" is vague. My understanding (not a python core dev, or even someone who follows PEPs closely) is that a PEP has a corresponding python version, but it does not necessarily result in changes to cpython (or other implementations) that issue runtime warnings.
Ah, we disagree on that reading of PEP 387 then. I read it to say the API or behavior being altered must have a deprecation warning for two major releases.
No. I agree with your reading, but the reality is that the PSF doesn't appear to interpret it that way and it's sufficiently vague to allow them to get away with that :(
I can't remember the last time (if ever) I saw a runtime deprecation warning from cpython itself (I see plenty from libraries).
I don't understand PEP 563 (or the python type system and ecosystem of tooling) well enough to know whether it would even be possible for cpython itself to issue a deprecation warning in this case.
In any case, it's clear that the pydantic and fastapi authors were aware of this issue.
Like I say, it's clear the pydantic and fastapi authors were aware of this issue, but for what it's worth I confirmed that no runtime warnings were issued for PEP 563.
The behavior has been available via `from __future__ import annotations` since Python 3.7. It was the very first change mentioned in the 3.7 release notes (alongside an explicit statement that it would become default four years later in 3.10):
I've read through PEP 563 several times now and I just don't find the motivation particularly strong. The PEP itself outline two high level goals:
1. simply forward refs in types
2. reduce the runtime costs of type defs
While I can respect both goals, I haven't found either problematic enough to break backwards compatibility, including functionality used in several mainstream libraries.
Further, I have yet to see the runtime costs quantified. So I'm not even convinced this will have a tangible benefit. This PEP rejects the alternative option to just disable typedefs with opt-in __future__ imports or command line optimization args. Yet, they rejected this because it would break existing functionality for code that relies on runtime typedefs. And the proposed solution also breaks existing functionality.
In terms of forward refs, the deferred evaluation just feels like a kludge. Every Python programmer is already familiar with how Python evaluates module code at runtime and understands how this prevents forward refs. If anything, this kludge adds a special case for typedefs.
I really feel its time for Python 3 to freeze language features. This PEP and the previous walrus operator fiasco feel like people trying to morph Python into something that it isn't with one kludge after another. This will not lead to a robust language, but instead another Perl 5 monstrosity of complexity.
Instead, I feel programmers looking for more robust, performant, and expressive languages should start exploring new languages. Let Python be the best possible Python and other language features can be explored in languages that incorporate those features in a holistic design.
> I really feel its time for Python 3 to freeze language features. This PEP and the previous walrus operator fiasco feel like people trying to morph Python into something that it isn't with one kludge after another. This will not lead to a robust language, but instead another Perl 5 monstrosity of complexity.
I disagree entirely. This is how Python will be left behind. It needs to be able to compete with languages that adopt new features and paradigms.
For example, new Ruby releases got optional typing and pattern matching. Even Java is getting pattern matching[1]. This is where the developer zeitgeist is heading, and if Python doesn't keep up, it'll be replaced with a language that does.
That won’t happen though. When the 2->3 split happened, people swore up and down “I won’t stand for this! You can’t break Python! We’ll fork it and nobody will ever have to do these changes.”
Now, everyone is on Python 3 for new stuff, and companies are trying to transition to 3 or paying for the company (who’s name escapes me) that does security updates to 2.7. It never works out. It’s best for the PSF to try to reduce breaking changes and try to keep everyone together.
2 to 3 was about incompatible changes, and many companies switched to 3 because of eol of python 2.7, or its EOL would have been a minor event. What I thought about was a backwards compatible branch. We would then see if people need more features or if its enough, but I understand I'm only suggesting more(maintenance) work.
Well, this particular change (PEP 563) is IMHO actually striving for improved static typing and better performance, so, I guess they're in sync with that.
-- PEP 649 would undo those performance improvements and possibilities of improved static typing over concerns of libraries that user types at runtime having to adapt (I took a look at the issues reported at pydantic and while the API would need to be adapted when PEP 563 would come into effect, I don't really see the major breakage they're talking about -- not that it doesn't take some work to adapt, but I don't see the reason it'd suddenly become impossible to do, it'd only make their own lives a bit easier over the performance cost of all the libraries that use typing without requiring runtime information).
I agree with your conclusion, but I think the forward refs issue is more annoying than you're giving it credit (not that the fix should break stuff). Type hints happen at a higher level, so things are more strict.
class A:
def x(self) -> B: # doesn't work
return B() # works
class B:
pass
I find this annoying enough that I've been using the future annotations.
Generally speaking that's considered unpythonic though. Default values are evaluated exactly once, at function definition time. So if you do this, then your default B() will always be the exact same instance of the B class -- literally the same object in memory -- every time you call the function. That's rarely what you want; you usually want the default to be a fresh instance of B(). So instead, the pythonic way is for the default to be None, and then to convert those Nones into their actual defaults in the top of the function/method. This then has the added benefit of playing more nicely with subclassing (as well as composition), because instead of needing to bubble the defaults up the call chain, you simply pass down None and let the implementing function worry about resolving it into the actual default.
I make way more small, immutable classes than the average Python developer. I've never had the reference issue with defaults and always have it with hints.
Honestly, I have mixed feelings about this. Pydantic is pretty cool but also kind of a strange bird.
It seems to have been originally conceived to do the job of forms in a web app (serialization/deserialization). But then it also has a double life as a runtime type validation framework. And I don't think it really does either one quite that well. It definitely offers a level of conciseness that is nice compared with traditional form or serializer frameworks. But then at times it seems underpowered or awkward to customize. I think that's because it doesn't seem to know exactly what it wants to be.
I've worked on projects where we used pydantic and it eventually became clear that we were either overusing or misusing it. It always seemed like the abuse of pydantic came out of developers on the team not really understanding what its purpose was. Maybe it's just my personal bias, but I think it's worth bringing a bit of skepticism to this debate and not just accepting the description of things in the issue ticket at face value. In other words, is it possible that pydantic is colliding with the steering council because it's misusing the annotations feature? I don't think I have a clear answer but it's worth asking the question anyway.
I agree, I don't love Pydantic. I much prefer the "stack" of Attrs + Marshmallow + Desert + Apispec. More tools, yes, but they compose nicely and they each do one thing well. Now where have I heard that one before...
I wish FastAPI had chosen that path instead of Pydantic, but here we are. FastAPI itself is a wonderful tool, with wonderful docs.
is it possible that pydantic is colliding with the steering council because it's misusing the annotations feature?
Yes.
There are lots of good reasons why types aren't necessarily a great match with runtime validation.
It gets the whole thing backwards IMO; I want to be erasing types at runtime and verifying them ahead-of-time. If I consume a piece of data that might be the wrong type, I want to explicitly validate the data myself, and return the desired type if and only if my validation passes. This is the Attrs approach and it works much better, even if you have to put in a few extra keystrokes in the handful of situations when you want explicit validation upon every class instantiation.
Moreover, making annotations lazy is not mindless adherence to purity or some kind of intended "fuck you" to the Pydantic ecosystem. PEP 649 clearly states its motivations. Type annotations have always been a static analysis tool first and foremost, and I think it's absolutely wrong to even suggest that CPython should suffer a higher maintenance burden and suboptimal performance to support what is effectively an off-label usage.
To each their own. I dislike Marshmallow and I adore pydantic. Runtime validation that aligns with static types makes sense to my brain, and keeps me from losing my mind in a complex system.
I don't think making the type annotations lazy is a "fuck you" to pydantic, but it is a breaking change on a minor revision.
Well, it is a breaking change on a minor revision, but Python doesn't use semantic versioning so there's no reason that can't happen.
The relevant PEP, https://www.python.org/dev/peps/pep-0387/, states that breaking changes may be made with two minor versions of warning (or one major version). Minor versions are released on a calendar schedule (again, not semantic versioning), so this policy is 2-3 years of warning in practice.
I generally agree with you. Building automatic validation on top of dataclasses seems like a far-superior solution moving forward, given the language's trajectory.
It's easy to say, but pydantic exists, and automatic validation on top of dataclasses does not/there's no well-known, well-tested library? (I'd also prefer something other than pydantic that doesn't coerce types by default, but alas...)
And it's a huge use-case that should've been obvious. I get type annotations are not static typing, but hand-rolling validation doesn't feel like "batteries included". More generally, there are already tools using type annotations, and there'd likely be more if it was robust/easy. Python is generally good at enabling metaprogramming, but type annotations are a bit of a minefield. See even dataclass' own impl, e.g. [0]. And building on top of dataclasses is also tricky.
I'm hopeful it can be resolved. Type annotations are a huge boon to the language, and it's already no small feat to have added them without breaking backwards compatibility.
Dataclass child can't inherit from a parent that has a default value set. How is this not a dealbreaker for most? That's literally the only reason I keep using pydantic instead of dataclasses.
From a personal point of view, FastAPI and Pydantic are some of the greatest additions to the Python ecosystem lately.
Apart from that, FastAPI is currently the top 3 web framework according to the Python developer survey[0]. I think having such a breaking change is a serious issue.
Other people have already mentioned great things about FastAPI (documentation, data validation, good performance, easy to use, etc).
There's also a lot to love about Pydantic alone. For me, the best thing about Pydantic is that I get performant, strongly validated objects. When I create Python apps, I try to make sure functions receive and return Pydantic models. That way, if the function does not fail, I'm 100% sure about the data coming in and out. It also saves me from writing many if statements.
Those Pydantic objects can also be dynamically created without losing the data validation parts. By using it, I feel very "secure" about my types and objects, but keeping the simplicity of Python for my whole app.
Have you written it with an ORM backed app? I’m curious if you put the business logic on the Pydantic models and keep the ORM small, or put the logic on the ORM and treat Pydantic like a serializer. Personally, I’d rather work with Pydantic models than ORM.
I don’t use ORMs. For bigger apps, I use asyncpg and map the results to Pydantic objects. I write raw SQL and keep Pydantic objects as the “source of truth” throughout my app.
We use SQLAlchemy as the ORM layer, but it's very small and most of the work is done with the Pydantic models. It works quite nicely, despite that we loose change tracking, but SQLAlchemy got us covered with merging models.
From the perspective of someone who mostly dabbles in creating APIs for my own use:
- Auto-generated docs and simple front end for your API allow you to examine and interact with your endpoints with essentially no added work or tools
- Feature rich and performant
- Tight pairing with Pydantic makes specifying and validating data models easy
Basically, FastAPI (and pydantic under the hood) just abstracts away a lot of the work you would need to do to build a relatively fully-featured API on your own, and adds quality of life features that one generally couldn’t justify building/setting up for themselves, especially for smaller projects. Plus, the docs are solid and it’s a pretty simple library to get started with. It’s very much been an “it just works” library for me in most cases.
I like that FastAPI has documentation meant to be read from beginning to end. It walks you through not only the how but the why of core design decisions, and even dives into ancillary things like pydantic. This is in sharp contrast to most python packages of modest size where IMHO the bare minimum effort is put into technical docs. Usually you get something using a 10+ year old Sphinx template that hasn't changed for modern devices, and no description of usage is more than a trite paragraph (i.e. the attention span of a developer writing a docstring).
Among other things, having the query and body parameters automatically validated from the type hints defined on the endpoint and having the corresponding HTTP error and error message automatically added to the response is a huge time saving.
Also, the doc is great and the maintainers are very helpful on github.
"In the end I don't think python's core developers and steering council want to make your experience of python worse. They just haven't realised how important this is. (Even Larry hadn't heard of pydantic or FastAPI until yesterday when he emailed me, I guess they don't read the python developer survey wink)"
Yikes should a steering council that doesn't even understand the needs of the general community through common sources like that survey really be the steering council? That's a really strong signal that they're out of touch and can do more harm than good.
Personally given the drama over the last few years regarding BDFL and then some of the peps that have made it in in recent time, as a heavy user I've had various concerns over the direction of the language and this further drives those.
This Steering Council exists to consolidate power among old-boys and their corporations and not to further the language.
They have done very little apart from locking down the mailing lists and issuing authoritarian statements every now and then, mostly without giving any valid reasoning.
I expect Python to go the way of Perl if this continues.
On the contrary, I think they are more concerned with supporting the core language and interpreter than with anything else. Also why is this a Github issue and not being discussed somewhere like the PEP subforum of discuss.python.org?
It’s in multiple places. The main discussion place for python core development is in the mailing lists and there are topics for this issue posted there today too.
I don't think they can afford not to pay attention to the ecosystem regardless of what their concerns are. At this point the ecosystem makes Python more than their decisions, what good they do has harsh diminishing returns but their potential mistakes have, as this issue displays, large consequences.
I absolutely love working in FastAPI and pydantic. So much so that the backend of a new app I just launched in private beta on is written in FastAPI. My startup has traditionally been a flask workshop but FastAPI is just so much more productive due to the documentation and pydantic type checks.
I seriously hope Python developers understand that apart from the superb programming language it is, package ecosystem also is what makes people prefer Python as the language to invest their hard labor into. Small teams need productive libraries like FastAPI to develop systems. Please don't kill FastAPI and pydantic. I don't want to go back to writing flask again after working in FastAPI.
I don't use Python nearly as much as I used to (mostly Go now) but changes like this make me realize how much I appreciate the Go 1 compatibility guarantee. I have Go libraries from 1.6 that I have upgraded without a second thought.
These sort of PEPs (remember the walrus operator?) seem like fundamental language changes that serve to divide the community without much gained.
Although I will say, Go has made it much easier to be involved in the language design process.
Nothing is immune, these are fundamentally people and person problems which can and will affect any language. It's ironic that just this week one of the core maintainers of Perl stepped down because of backlash and harassment from other maintainers over proposed changes (https://news.ycombinator.com/item?id=26779152).
Python is a little younger than Perl but also faced similar problems--Guido had so much backlash against proposals that he stepped down too (https://news.ycombinator.com/item?id=17515492).
Go is younger than either of those but not immune to the same fate--just look at the total divide and amount of controversy created over things like generics or modules in the community.
There's no magic fix for this--if an organization is going to thrive and grow (or even just survive) for 30+ years it has to address those issues sooner rather than later. Burnout, at all levels, is real.
I have developed a quite powerful tool building on top of Python type hints. In a way, I have almost started loving their flexibility as an API designer.
I hope PEP 649 or a similar proposal is approved before 3.10.
I read that and I'm still no clearer about how to do anything about this, though as a user of both pydantic and fastapi (and having mandated their adoption in my org), I'm extremely motivated to do something about this. Can anyone spell out to me precisely what i need to do to be heard?
It seems that this post and the attempts to draw a lot of attention outside of core dev is likely to hurt pydantic's position rather than help it (though probably it will do neither because the core devs / steering council aren't going to ignore an issue out of spite or something).
This probably would have upset me five years ago when I was still invested in the Python ecosystem (I had been using it since 2.1 in various capacities): the decision to break backwards compatibility in Python 3 was a huge factor in my decision to look around for another language to focus on: my experience with Common Lisp and Clojure (see Rich Hickey’s Speculation talk) has really made me value ecosystems where the people managing the ecosystem don’t waste everyone’s time with trivial breaking changes.
I do use and like both fastapi and pydantic but this situiation is something they brought to themselves by adopting experimenral features into their mainline code and sell it as production ready. Python designers are not responsible for time-to-market decisions of other projects. It comes with the "no liability" part of opensource.
Thought it has been there for some time, it was added to core Python on a "provisional" basis (it says so in the docs), meaning that they were not bound to keep backwards compatibility with it. In other words, it's an experimental feature.
It's definitely experimental. No other language has an established system like this, although Ruby seems to have acquired one in 3.0. We still aren't close to uncovering all the unsolved problems yet, let alone solving said problems.
You and I seem to have different definitions of "experimental feature" then. By your definition, I don't think they did anything wrong. dataclasses in the standard library also rely on runtime type annotations. (how they don't have the same issue with 3.10 I don't know). I would hope people would start relying on new language features years after their release.
Data classes don't do runtime type checking, but they do provide dynamic type info through dataclasses.fields(). This function returns a tuple of Field objects, and according to the documentation[1], Field includes the attribute "type":
type: The type of the field.
In Python 3.10, this now returns a string. In my opinion that's a bug (perhaps in the docs).
I verified this using a short test program:
import dataclasses
@dataclasses.dataclass
class Apa:
bepa: int
first_field = dataclasses.fields(Apa)[0]
print(type(first_field.type))
I'll shamelessly piggy-back here, and note that I'm working on a different type of type-safe validation library for Python here: https://github.com/keithasaurus/koda.
I'm nearing an initial release, but am still working on some of the docs, naming, and things like that. If anyone would like to give feedback, I'd be grateful!
It would be a shame if FastAPI is crippled because of PEPs. It truly is a wonderful API and a very nice API to work with. Here’s hoping a solution can be had and that the Python core maintainers “see the light” so to speak.
Ok so then I’m really confused. I thought it said annotations were having to be strings versus objects. If this isn’t the what’s called annotations in other languages, then what is the actual issue that pydantic has? The python type hints are I guess objects, but they’re standalone non-instantiated.. unless pydantic does some crazy stuff. Their main home page doesn’t show anything out of the ordinary here? What’s the actual issue?
Type annotations were meant to be used as linting, and other forms of type-checking done NOT AT RUNTIME. Gradually, some support for using them at runtime popped up, and them annotations starting being [ab]used for all sorts of things.
PEP 563 (the one breaking pydantic here) is actually super important: it allows referencing things defined later. Like, a function on line 1 can take a type on line 10 as parameter. It is super necessary to make type annotations usable for their original purpose.
It breaks pydantic, but also makes annotations usable for their intended purpose. Previously you had to manually opt it on each file, and now it's the default, as planned.
I feel like pydantic and tools built on top of it were built ignoring the direction on which Python was heading. Much like the user in xkcd#1172.
I've been using FastAPI and type annotations for over 2 years now and they have improved even small scripts. I love FastAPI, it is the best Framework I've come across in years. I wish there was a way to vote for PEP 649 instead of letting a small group decide about this alone. Maybe we can create a fork of Python 3.9 and call it different from hereon?
> instead of letting a small group decide about this alone.
"Small group". It's a group that consists of all people willing to spend their spare time (or money) on improving Python and its ecosystem. In fact, since you asked: You can become a member and get voting rights, too -> https://www.python.org/psf/membership/
I've read all the arguments about static typing, but I still can't comprehend how people get themselves into a situation where using the wrong type is a problem. I totally understand it as an optimization if your language supports it, and even for generating docs. I just don't know why people insist on writing code to stop themselves and other developers from doing things.
That said, I really really can't stand breaking backwards compatibility, and it sounds like that is what is happening here.
In a larger codebase, there's two major kinds of value you can get out of types:
- Types act as a contract between component surfaces - if you need to integrate multiple components and have different people working on it, having the compiler enforce the contract is super useful.
- Types act as a refactoring aid - if you need to alter a component, the compiler can assist you in telling you what needs to be changed and you can offload more of that work at compile time rather than at run time.
As pieces move and change, tracking down what else needs to get moved and changed gets more and more difficult and this is where the "passing the wrong type in" issue can occur: having the compiler help you out with that is a huge boon.
Similar to your first point, types also allow an IDE to inform user about what values can be used for a function or class argument.
I can't say that typing bugs occurred very often in released code I wrote before type annotations, but it was not uncommon to encounter them when testing code I had just written (where they usually just caused an exception early on in the program's lifecycle).
If it were a compiled language where I had to wait to run it, I could see that being a big help, but in python I just constantly run the parts of the code I'm editing. If there are any typing issues I catch them almost immediately.
To be fair, I do set return types in my docstrings, and let the IDE help there.
My gripe really stems from using a libraries that check for a type and raise an exception if it doesn't match. Instead of just trying to actually use it and see if that fails. for example If you require a dict because you're going to expect certain keys, but I have a class that has __get_item__ for those keys, it should work fine. If you check the type then you'll fail before you get that far.
Imagine you need to change the type that's passed to a function. You attempt to update all places, but you miss a path that's in an edge case.
Static typing would have caught that for you. By waiting until runtime, you've now just potentially shipped a regression.
As a code base gets large, it's just not possible/practical to run all possible ways the code you're editing can be reached. Unless you happen to be lucky enough to have 100% unit test coverage, which very few people do, and running that test suite is somehow quick enough to be a practical alternative to static typing.
If I need to change a type thats passed to a function that already exists and is heavily used, I would either cast the old type to the new type so both work, or create a new function. If I really cared about cleaning it up I would add a deprecation warning when its done the old way.
The big benefit of type hints things like Pydantic (and some things I've written myself) is that it can provide useful runtime information. If you know that property `obj.foo` is hinted with `List[Foo]`, then that allows you to do something expecting to have a list of `Foo` objects. This is really extraordinary for deserializing untyped data (like a json blob) into appropriate objects at runtime. You can take something like `{"foo": [...]}` and attempt to deserialize each element of the `foo` list into `Foo` objects, rather than just guessing at what the list contains. If you stop being able to understand type hints at runtime, then the only thing they're good for is documentation and static analysis.
If you have to consume functions called call_some_function(), you've got way bigger problems than untyped parameters.
A well designed function/class will have a descriptive name and descriptive parameters, which should give you good hints about their usage. And if it that isn't enough, a quick scan of the code will show you exactly how the function works - a good idea (even in typed languages) if you have any doubts about what you're calling.
The fact that people can program effectively without types is demonstrated by the enormous volume of code in non-typed languages.
So how do I know if a function will return a list or a string? Will it be "get_things_list" ? Or maybe 'get_things', it's got an s, so therefor it's many things? Or at least an iterable thing? Or maybe an object that contains some values? If there are many things can I index into it, or just iterate over it? If I have to guess, will that force me to use a List over an Iterable, even if that's the wrong abstraction?
At some point you have to convey the type or I have no idea wtf you're giving me, and I'd rather have you convey the type in a way my computer can understand than have you convey it through the godawful way that people name things.
That people manage to do so is no endorsement. Humans managed for quite a while without antiseptic but I don't hear anyone endorsing going around with shit on their hands.
Honestly, I've been programming in both typed and untyped languages for ages and it I am asking how you manage this problem, because I find IDEs radically less effective in dynamically typed languages, I find that I have to ask more questions, do more manual work, etc. In typed languages these things become trivial.
A) I think call_some_function() is metasyntactic, not literal. But even if it was...
B) I largely agree but I still don't get how people manage without types, though I get why it occurs. People have different brain styles and hence different programming styles. Personally I dislike dealing with untyped code with "descriptive" names, because it's rarely descriptive enough.
I'd take a strong type system with methods like .call(), .string(), .get(), .issue() with an IDE vs overly-descriptive methods like .frob_the_snozzberries() and .squelch_parrots() any day. It's not composable, it's not predictable - wait was it .frob_snozzberries()? There's no symmetry. That style usually accompanies very dynamic code, since if your interface is very specific, .post() only can mean one thing.
I feel the same when writing a script or small project. Easy right? Types get in the way.
If you feel that way all the time however, it likely means you've never worked on a large or venerable project with multiple developers of varying expertise and skills. Or demanding management. Preventing bugs is exactly what you want to be doing in that situation.
Type checking is like wearing a seat belt, it's not needed a lot of the time, but when it is it'll save yor life.
What do you mean by "algebraic errors"? The problem I (and DDG, and Google) consider that term to mean happens just as often in statically typed languages.
Strongly disagree that use of FastAPI over Starlette is down to marketing.
I regularly use FastAPI instead of Starlette on small work projects and personal projects. The first thing I did when pointed at FastAPI was follow their very prominent links to the Starlette project. A cursory read of the Starlette docs shows that it’s not the same thing at all and not a viable alternative. I’d use other tools (in Python or other languages) before Starlette for my use case.
The way routing works, the Pydantic validation, and the auto-generated docs all mean that I can spin up a simple service, often in a single ~100 line or less Python file, shove it on a server behind Caddy for HTTPS, and literally have a usable, reasonably performant and self documenting API. Often in less than an hour.
Starlette simply does not do this for me. I hate boilerplate, hate template projects/repos, hate personally doing “devops”, managing configs, etc., and generally dislike anything that gets in the way of connecting an idea to the internet. Enterprise ready? Not in that form (though I’m sure FastAPI _can_ be), but useful: hell yes.
FastAPI requires nothing that doesn’t have a purpose, even in throwaway or prototype projects, is pretty quick, and works the way I think. In my eyes the project has added a HUGE amount of value to Starlette for some users and embodies the point of open source, which is about creating and sharing and improving upon useful code (preferably with attribution, which Starlette gets prominently in the FastAPI repo and docs), not recognition or building a “brand” or user base for yourself or your project. It’s not a marketplace.
This seems unnecessarily antagonistic - maybe you’re haven’t fully explored what it offers. FastApi is very transparent that it is a layer built on top of Starlette https://fastapi.tiangolo.com/features/#starlette-features. It also layers in several other features at really useful abstraction levels by bringing in Pydantic, and I think the dependency injection system and OpenApi integration was mostly built by Sebastian.
It’s got fantastic documentation and tutorials. For a small dev or full stack team, and especially someone just starting out, it feels unparalleled in what it gives you out of the box.
Not sure what else they could say, when 95% of FastAPI features come directly from Starlette. Again, the code on github speak for itself.
As for the remaining 5%, that's basically pydantic and the swagger / openAPI stuff: questionably useful and very easily reimplemented if need be.
The way I see it, FastAPI stole much of the recognition Starlette deserved. I find it a bit sad. Starlette is where all the async magic happens.
Although I don't think ripoff is a good word for describing their relation, I definitely think Starlette deserves more attention. As I prefer CBV I tend to lean towards Starlette.
I used starlette and switched to fastapi. For a one person project the bonus free swagger docs in fastapi were what got my attention. If that's also built in to starlette... I guess then the marketing worked on me.
I know people don't like the specific decisions python has made recently (me too!), but the process for deprecation and implementation once those decisions are made is at least clear and unambiguous, thanks to backwards compatibility policy adopted in PEP 387, i.e. way back in 2009.
"Wait for the warning to appear in at least two major Python versions. It's fine to wait more than two releases."
PEP 563, the one that apparently breaks pydantic, was accepted during Python 3.7. This was before pydantic reached 1.0 and before fastapi even existed. It was eligible for full implementation in 3.9 according to the backward compatibility policy, but the plan right now is for it to be implemented in Python 3.10. Python 3.10 is due to code freeze in three weeks. The maintainers of these projects have had over three years to raise these issues.
pydantic and fastapi both have outstanding documentation that is a credit to their projects and to the python community. One side-effect of this, however, is they have some inexperienced and frankly excitable users, who perhaps aren't aware how python itself is developed. These are the users their maintainers (see also https://twitter.com/tiangolo/status/1382800928982642692) are unleashing on the python core maintainers and release managers literally days before a release that will implement a change they've known about for over three years.
I hope this can be worked out. PEP 649 looks promising. Maybe PEP 563 should wait for python 3.11. And pydantic and fastapi are great! But this is, simply put, a shitty way to treat volunteer open source maintainers at the absolute busiest and most stressful time of the release cycle.