The spatial growth of small perturbations developing on a fully developed base flow in a duct with two inhomogeneous cross-flow directions is examined. For a laminar mean flow it is shown that optimally configured vortices at an upstream cross-section induce large transient amplification of a disturbance energy norm downstream. Such a linear growth is a likely initial stage of transition in ducts for which the cross-section is square or of moderate aspect ratio, asymptotically stable according to conventional eigen-analyis. With the increase of the cross-sectional aspect ratio the transient growth results of the plane channel flow case are approached. The optimization methodology is then employed to study the transient amplification of large-scale secondary structures developing in a square duct for which the unidirectional base motion has a turbulent-like profile. Under the hypothesis that fine-scale turbulence does not affect the growth of large-scale secondary flows, it is shown that cross-stream vortices are sustained by the mean flow and grow maximally over a streamwise distance of several thousand viscous units of length.