We never grew up with a lot of money for vacations and expensive trips, but every weekend was an adventure. When I was around 5 years old my parents bought an old pop up camper. To a 5 year old that was the coolest club house ever. My parents would pull it all over the state with their Toyota Camry. If it wasn't for my parents getting us out of the house, and outside we would have missed out on so much.
If you think about the cost of seeing a movie these days, an outdoor adventure might even save you money. Most permeative campsites are $16 a night, full hook ups are $30 at most state campgrounds. I have never seen a trail head with a credit card machine. They say the best things in life are free, well when it comes to our national parks that pretty true.
Rent before you buy. My family would rent a boat every summer. It made buying a boat easier, because we knew it would get used. More people have buyers remorse when buying a boat or RV then a car or house. You just don't know how much use you will get out of it until you do it. If you're renting a boat a couple times a year ten you're most likely going to get your money out of your own boat. If you're renting a cabin or RV on a regular basis then staying in your own RV will be an easy choice.
Unless you're going to be spending all your time with one activity don't spend all your money on it. Most of us outdoorsmen only kayak or rock climb a couple times a year. I would much rather have a decent used kayak, a tent that keeps water out, and a backpack that holds my stuff while backpacking then just a $1,200 kayak. Check the big box stores at the end of the season for over stocked gear. They don't want to hang on to it, so save some money by storing it your garage. Then when spring hits you're already to hit the woods.