Hard to tell from the pics, but the material looks like it might be schist (weathered). A very common mineral in schist is garnet. When garnets weather out of the rock it can resemble rusting, as well seen in your second picture. Also, if you'll look at the top of the point in picture one you'll see several small dark dots. These may be garnet. These garnets are the iron end member of the group, called almandine.There's a lot of schist outcrops in the eastern US where you find the metamorphic belt of the Appalachians.
The garnets are typically this size in any of the Appalachians schist I've seen, but I have seen garnet in schist the size of ping pong balls in Alaska!