Children with congenital heart defects (CHD) are often short/lightweight relative to peers. Limited growth, particularly height, may reflect energy deficits impacting physical activity. Latent class analyses of growth from birth and Bruce treadmill exercise data retrospectively identified for height, weight, and body mass index z-scores growth trajectories. Linear regression models examined exercise parameters by growth trajectory, adjusting for age/sex/CHD severity. A total of 213 children with CHD (39% female, 12.1 ± 2.9 years) achieved 85.8 ± 10.1% of the predicted peak heart rate. Peak heart rate among children whose height was consistently below average (class 1) was 15.2 ± 4.9 beats/min lower than children with other height trajectories. These children also attained a lower percentage of predicted peak heart rate. Children whose weight (p = 0.03) or body mass index (p < 0.001) z-score increased throughout childhood had significantly lower exercise duration (mean difference 1–2 min) than children whose growth trajectories were stable or declined. Children with above-average weight or an increasing body mass index also used a higher percentage of their heart rate reserve at each submaximal exercise stage. A very low height z-score trajectory is associated with decreased exercise capacity that may increase the risk for morbidities associated with a sedentary lifestyle. Future studies should examine potential mechanisms for the observed height deficits, such as an inadequate energy supply that could impact physical activity participation, congestive heart failure, cyanosis, pubertal stage, supplemental feeding history, or familial growth patterns. Prospective studies examining growth in relation to objective measures of daily physical activity are required.