Abstract. State of the art works for Point-Of-Interest (POI) classification use either traditional feature extraction methods, or, more recently, deep learning (DL) methods, in order to train classification models on historical data, i.e. POIs already annotated with categories, and then deploy these models to classify new, unannotated POIs. These methods are either inherently limited to learning simple, rather than complex, relationships, or learn embeddings trivially and without taking into account domain knowledge or semantic information, thus yielding disappointing classification results. In this vision paper, we discuss the problem of POI classification, identify limitations in the state of the art and propose novel research directions. We propose a framework for learning meaningful context embeddings for POIs, that incorporate domain knowledge and semantic information extracted from external data sources, and we discuss how the proposed approach can overcome inherent limitations of the state of the art. Specifically, we prescribe the construction of fine-grained embeddings, by refining the consideration of spatial neighborhoods of POIs, assessing and selecting the proper attributes to be used as context and target information in the embedding learning process, as well as by properly enriching these embeddings with external, semantic knowledge. Additionally, we advocate the joint utilization of traditional training features and embedding-derived features for POI classification, and argue towards the usefulness of the proposed approach.