The impact of radiation therapy on the immune system has recently gained attention particularly in the context of radiation in combination with systematic immunotherapy. Radiation can have both immune-stimulating as well as immune-suppressing effects making the design of clinical trials difficult. Furthermore, different treatment modalities and fractionation regimens impact the interaction between the immune system and radiation. The goal of this work was to develop a mathematical models to allow clinical trial design by considering the time-dependent rates of malignant tumor cells, lymphocytes, and dying tumor cells during radiation therapy. Specifically, the framework describes how recruited lymphocytes inhibit tumor progression. The parameters of the model were in part taken from the literature and in part extracted from blood samples (circulating lymphocytes (CTLs)) collected from hepatocellular carcinoma patients undergoing radiation treatment. Based on the fitted parameters, the model is able to predict the depletion and recovery of circulating lymphocytes in patients as a function of dose distribution and fractionation regimen. The CTL levels dropped on average down to 62% for a 3 fraction regimen as compared to 51% for a 15 fraction regimen.The increased average CTLs number with a small number of fractions indicated potential anti-tumor immunity with hypofractionation.Moreover, the model allows the prediction of realistic local failure rates for a given biologically effective dose. We were able to demonstrate that tumor control is increased for a hypofractionated regime with a short time interval between treatment fractions. In summary, this study introduces the first mathematical model for tumor-immune interactions using clinically extracted radiotherapy patient data. Our model can be applied to design trials aiming at improving the immunogenicity of RT regimens.