Gains from physical activity and eating right takes some time to manifest. Consistency is important to help derive desired benefits. Keep a log, take body measurements and over a few months, you may begin to see changes.
Yo-yo dieting and getting off the plan periodically would prevent any benefits and may lead to falling off the wagon. When you look at a fit person, it is not an instantaneous state, it is in fact the culmination of years of consistent adherence to a healthy lifestyle and regular exercise.
If interested, check out my post on how muscles are built and you would realize that when you are not consistent, gains made are lost. This leads to a feeling of not making progress despite efforts and ultimately leads to falling off the wagon.