If you cannot take the cloth parts off, try using Oxy Clean. Mix the powder with warm water and use a clean rag to scrub the material clean. If you have sensitive skin, wear gloves. I do this to clean my kitchen chairs. A good scrub and then a rinse with another rag takes care of it and usually if I catch it fast enough, makes it look new again.
Just want to add that before you clean the whole stroller, do a spot test of washing, rinsing and drying somewhere under view so you can make sure the color does not change.