the top layer isn't actually bad. Down a bit and you get retaining-wall-fill though. And below that actually is concrete.
It's literally a slope fill over concrete. It's not going to get good no matter what. The better answer is hard terracing with gravel (like the rest of the area like it) and planters, but I can't afford that right now.