Vaccinations have emerged as one of the key tools to combat the COVID-19 pandemic, reduce infections and to enable safe re-opening of societies. Vaccinating the entire world population is a challenging undertaking and with demand far exceeding supply in the world, it is expected that topics surrounding vaccinations generate a wide array of discussions. Therefore, in this paper, we collect data from Twitter during the early days of the COVID-19 vaccination program and adopt a linguistic approach to better understand and appreciate peoples’ concerns and opinions with regards to the roll out of the vaccines. We begin by studying the term frequencies (i.e., unigrams and bigrams) and observe discussions around vaccination doses, receiving doses, vaccine supply, scheduling appointments and wearing masks as the vaccination efforts get underway. We then adopt a seeded topic modeling approach to automatically identify the main topics of discussion in the tweets and the main issues being discussed in each topic. We observe that our dataset has nine distinct topics. For example, we observe topics related to vaccine distribution, eligibility, scheduling and COVID variants. We then study the sentiment of the tweets with respect to each of the nine topics and observe that the overall sentiment is negative for most of the topics. We only observe a higher percentage of positive sentiment for topics related to obtaining information and schools. Our research lays the foundation to conduct a more fine-grained analysis of the various issues faced by the people as the pandemic recedes over the course of the next few years.