These holidays are commanded in the Torah.
It was commanded that on the First day of Tishri, there would be a celebration, because that would be the head of the year. Originally, this holiday was not called Rosh Hashanah, and it bore no connection to Yom Kippur. It was merely commanded to be a celebration on that day which included the blowing of the shofar. In modern times, it's the day to begin the process of repentance. It's also a day that is associated with the creation of the universe, and as such, we are to repent with the intention of tikkun olam (making the world a better place).
Some rituals associated with Rosh Hashanah are the blowing of the shofar, eating honey with apples, making round challah, and tashlich. Tashlich is the casting of crumbled bread into flowing waters; this is supposed to represent the casting off of sins.
It was also commanded that on the 10th day of Tishri that there would be a day of atonement. Jews are called to straighten out their affairs with God and to afflict their souls. In modern times, we afflict our souls with fasting. To do that, we eat no food and drink no water.
It is important to realize that Jews believe that God can not forgive sins committed against other people; only the person wronged can do that. Therefore, today's Jews use the ten days between Rosh Hashanah and Yom Kippur to make amends with the people they may have wronged. These days are called the "Days of Awe".