High harmonics can be generated up to very high orders so that by superposing many exactly phase-locked harmonics, i.e. Fourier-limited pulses, the theory promises that few attoseconds long pulses could be generated. Here we show that there is a fundamental lower limit to the shortest pulse duration that can be achieved by superposing harmonics. By revealing the ultrafast dynamics of the electrons participating to the emission process we have observed that groups of consecutive harmonics are emitted at different times, the delay being a linear function of the frequency. We demonstrate that this drift can be optimized to reach 130 as bursts and propose a method to compensate for it, resulting in pulse duration of 70 as.