Three-dimensional (3D) genome organization, which determines how the DNA is packaged inside the nucleus, has emerged as a key component of the gene regulation machinery. The availability of high-throughput chromosome conformation datasets, such as Hi-C, across multiple conditions and time points offer a unique opportunity to examine changes in 3D genome organization and link them to phenotypic changes in normal and diseases processes. However, systematic detection of higher-order structural changes across multiple Hi-C datasets is a major challenge. Existing computational methods either do not model higher-order structural units or cannot model dynamics across more than two conditions of interest. We address these limitations with Tree-Guided Integrated Factorization (TGIF), a generalizable multi-task Non-negative Matrix Factorization (NMF) approach that can applied to time series or hierarchically related biological conditions. TGIF can identify large-scale compartmental changes as well as smaller topologically associated domain-level changes. Compared to existing methods, TGIF identifies has fewer false positive TAD boundary changes. Application to two mammalian developmental time courses provides multi-scale characterization of genome dynamics that we validate with enrichment of one-dimensional regulatory signals from histone modifications, accessibility and architectural proteins. Finally, we leverage TGIF boundaries to prioritize sequence variants for multiple phenotypes from the NHGRI GWAS catalog. Taken together, TGIF is a flexible tool to examine 3D genome organization dynamics across disease and developmental processes.