Belly size isn't that important
Doctors may measure belly size as a ballpark to see if the baby's growth is on track. It's far from accurate though. The idea is that after 20weeks the measurement from the pubic bone to the top of the uterus should match the gestation. For example, If you are 26weeks along, it should be about 26cm from your pubic bone to the top of the uterus. As long as it's within a centimeter or two you're considered on track for growth.
Depending on weight gain, position of the baby, fitness levels and many other factors, belly size can vary differently. Doctors just use it as a basic bench mark. If they're worried about the babies size they'll do more exact testing like and ultrasound.