You are missing a major driver of this. The main reason fertility rates have been decreasing is because of decreasing infant mortality. The number of children surviving to adulthood has been relatively stable until recently.
And more specifically there is a delay between when technology lowers the infant mortality rate and it becomes culturally normal to have the smaller number of children leading to the same number of adult descendants, so you typically get 1-2 generations with way higher populations than preceding generations and then the rest after that is just this large initial population having a normal number of kids. This means you get massive populations and intense crowding decades after the fact.
