This is a question I can actually relate to myself! If your skin is naturally on the drier end of the spectrum, powder —which absorbs moisture from whatever it is put on top of—will often make things worse. Luckily, there are some alternatives we can look at so you are still able to set your makeup.
1. Switch to a hydrating or dewy setting spray. Now that this awesome product exists, you can ditch the powder all together if you choose. Even if you don't use a setting spray (which I myself don't) you might find that if you simply stop using powder, it doesn't negatively effect your makeup as much as you'd think. Drier skin doesn't typically 'move around' makeup as much as oily skin does, so you might stay in place just as well.
2. Switch your powder & technique. I've heard this complaint before, and then found out the person was using a sponge to smooth on a powder foundation. The sponge inevitably puts a much heavier amount of powder onto the skin than you would get from say, dusting on just a bit with a fluffy powder brush, and powder foundations tend to be heavier themselves. Switch to a loose powder, and either dust on the lightest layer possible all over your skin OR only powder in 'problem areas' aka where you notice you get shiny during the day. A final word on powder selection: also avoid 'HD' powders that are made from pure silica (like the one made by Make Up For Ever or LA girl). Silica is a dessicant, meaning it is designed to dry things out, so it will be especially harsh on dry skin.
3. Ramp up your skin care. My finally suggestion is to make sure you have a good skin care regimen in place. There are tons of incredible products that will deliver hydration to your skin and balance it out so it's not so dry to begin with. If your skin is feeling dry and tight even before you get to the makeup phase of your morning, the makeup will only exacerbate these problems. If you need some guidance, check out my tutorial on the entire basics of skin care to get a good routine going. As an easy recommendation, start using a serum with hyaluronic acid in it beneath your moisturizer, and it will give your skin a nice boost.