I'm on heavy clay, the garden wasn't cultivated for about 1 years before I came here, but was before that, and the water table is high in wet weather. Dig a large hole and you create a pond, fillit with earth and it becomes a mud wallow, great if you are keeping hippopotamus. It needs to drain. I found one place where someone had put down a lot of shingle, presumably to create drainage, it had sunk and buried itself in the clay underneath, my impression is that sand does the same, silicone is pretty dense.
My solution has been to remove anything resembling top soil, then dig a hole in the clay and fill it with wood, then add compost to the top soil and put it back on top. That creates a bed raised about eight or nine inches and I have been using the clay dug out to make a bank around the edge. There is still a lot of clay in there and as I hoe it creates balls of clay which I hoe to one corner, gather in a bucket, and put in the greenhouse to dry. When I have a bonfire I make sure it can burn good and hot and then add a couple of buckets of clay. It burns to a sort of coarse terracotta that breaks up fairly easily and is much lighter than sand, and mixed with the ash makes a good additive.
The raised beds drain well, and the sodden wood underneath kept them much damper during last year's drought, after five years here I am gradually getting some good quality vegetable beds and a very nice flower bed, I am old enough that I don't go at it hard, little and often, but I am gradually finding other things that help.
For example, there is a 'lady who rides' who lets me collect from her manure heap, I have been collecting manure and chopping it up with poor top soil and am using it to grow potatoes in old compost bags with wood and leaves in the bottom few inches. Having a block of bags stood on it for a few months clears a piece of ground and I have something good to add when I empty them.
Basically I would say, firstly, don't go for a huge amount of work all at once, if it doesn't work it is very discouraging, secondly, having some clay mixed in is a good thing, it hold moisture and nutrients in the small balls of water that collect around the particles, it's having only clay that is a pain.