Water scarcity remains a major threat to most rural areas in Africa due to heavy reliance on rainfed agriculture. This prompts the need to document and understand the determinants of traditional knowledge of water conservation practices. Our aim here is to document and identify the determinants of people’s knowledge of traditional knowledge of water conservation in South Africa’s Eastern Cape province. A mixed research method was used. A purposive approach was used to collect data, i.e., elderly people were targeted during the face-to-face pre-COVID-19 pandemic interviews on the basis that traditional knowledge increases with age. However, people of different age groups were also included in the study especially when we employed only an online questionnaires during the pandemic. We used a questionnaire to collect all the data, and 93 people responded in total to our questions. We documented 10 traditional water conservation technologies. We identified rainfall variation, water quality issues, and increased water demand as the major challenges linked to these technologies, while easy access and the amount of water harvested are reported as benefits of these technologies. We found that traditional knowledge is not influenced by age (β =  −0.006 ± 0.01, P = 0.64) or gender (β =  −0.16 ± 0.25, P = 0.64) but rather correlates positively with geographic location, irrespective of the starting point of the distance measurement (distance from Port Elizabeth city: β = 0.002 ± 0.0008, P = 0.004; distance from Ngqushwa village: β = 0.0024 ± 0.0009, P = 0.008). Counterintuitively, formally educated people tend to have more traditional knowledge, but this is likely linked to the modern technologies (online surveys and social media platforms) used to collect data during the COVID-19 pandemic. We suggest that traditional knowledge that has sustained life for centuries in rural communities must be integrated into water resource management to address water scarcity issues in rural Africa.