The emergence of a plethora of wearables and sensing technologies has enabled non-intrusive  digitization of our daily physical activities. Emerging applications utilize such data to make inferences about our physiological and health states, provide health diagnosis, and contribute to wellbeing improvements. The common approach for such applications is to collect data, either using mobile applications or special hardware, e.g., wearables, and store them on a third party storage provider. This results in many unconnected data silos of self-quantification data. Researchers and industry, advocate for a common personal storage space, to conquer the myriad of small chunks of data, deemed to be lost/forgotten in the long term. The benefits of such co-located personal data are tremendous, specifically with regards to personalized medicine, treatment, and health care. However, the centralized storage of data exacerbates the privacy and security concerns that the IoT ecosystem is facing today. In this position paper, we advocate the necessity of privacy and security guarantees for the paradigm of co-located storage of personal health data. We envision two core security functionalities: true end-to-end encryption, such that only encrypted data is stored in the cloud and secure sharing of encrypted data, without disclosing data owner's secret keys. We discuss the challenges in adopting such an end-to-end encryption paradigm while preserving the cloud's basic processing functionalities over encrypted data and how to cryptographically enforce access control.