Staying in a machiya (traditional wooden townhouse) or ryokan is the defining Kyoto experience. The Higashiyama area near Kiyomizudera has excellent guesthouses and boutique ryokans within walking distance of major temples. Gion suits those wanting geisha district atmosphere. Downtown (Kawaramachi/Shijo) has the best transport links.
Arashiyama has lovely bamboo-grove-adjacent inns if you want to explore the western hills at dawn. Budget guesthouses cluster around Kyoto Station. Ryokan stays typically include an elaborate kaiseki dinner and traditional breakfast — budget $150-300/person for the full experience.