Captain: agree completely with SMM.
If you are going to go with generic desert and assuming you're not going to build specific terrain boards then I think the key is get a good quality mat which drapes and allows you to place objects underneath which will give you a sense of rolling low hills with various degrees of elevation / depression.
In the desert a height of only a few meters can be critical in combat.
If possible, the bases of your figures (and vehicles, if you base them) should match the color of the mat and blend in.
On top of the mat you will want to add desert scrub: this can be scattered but I prefer to mount mine on bases. I was making my own until BF/FOW came out with their product in 15mm: it is large and perfect for 20mm (BB226). All I did was repaint / mute the colors down.
You really don't need buildings but a few plus but a small oasis with palm trees is a nice touch.
The above is really all you need for a generic set up.
If you are looking at specific battles than a study of maps will be required. One issue with the majority of maps published in books dealing with the desert battles is the lack of elevation / depression noted. It all just seems 'flat' which can be very misleading.
The majority of tank battles were fought on hard rocky plateau's bisected by gebels / wadi's (hills / ravines).
Then you have the effect of escarpments relative to the coastal areas.
The map below will give you an idea:
shows off the U.S. infantry from the Flames of War starter set.
