Lattice control of multiple objects without distortion

One approach would be to use vertex parenting.

Create a mesh containing triangular faces which are unambiguously closest to their corresponding cobblestones. You could create the cobblestones from the mesh before triangulation, (as I did here,) or the other way round: create the mesh from a duplicate of the bottom faces of existing cobblestones. The mesh faces don't have to be connected to one another, in fact, it would be better if they weren't.

Give the mesh Shape-Keys for the intact and torn states:

You have to plan ahead a bit; leave yourself a state of the mesh from which you can parent the cobbles cleanly, or parent before making the shape-keys, if doing so can create the right parenting.

If they aren't already, P make the cobbles separate objects, and, all selected, ShiftAltX, send all their origins to their individual geometry.

From there, with the deforming mesh in a matching state, and active, with all cobbles selected too, in Object Mode, CtrlP parent the cobbles to the mesh, using Vertex (Triangle). The Cobbles will each be parented to the closest vertex-triangle in the mesh.

You are now free to deform the mesh's shape-keys, however you choose to, and the cobbles will follow, without distortion.

Here, with some Rigid Body cut in after the initial separation: