Sex chromosomes are unique genomic regions displaying structural and evolutionary features that distinguish them markedly from autosomes. Although nuclear three dimensional (3D) folding of chromatin structure is important for gene expression regulation and correct developmental programs, very little is known about the 3D architecture of sex chromosomes within the nucleus, and how that impacts their function in sex determination. Here, we determine the sex-specific 3D organization of the model brown alga Ectocarpus chromosomes at 2 kb resolution, by comprehensively mapping long-range chromosomal interactions using Hi-C coupled with Oxford Nanopore long reads. We report that Ectocarpus interphase chromatin exhibits a non-Rabl conformation, with strong contacts among telomeres and among centromeres, which feature centromere-specific LTR retrotransposons. The Ectocarpus chromosomes do not contain large local interactive domains that resemble TADs described in animals, but their 3D genome organization is largely shaped by post-translational modifications of histone proteins that regulate chromatin compaction and mediate transcriptional regulation. We describe the spatial conformation and sub-nuclear positioning of the sex determining region (SDR) within the U and V chromosomes and show that these regions are very insulated and span the centromeres. Moreover, we link sex-specific chromatin dynamics and gene expression levels to the 3D chromatin structure of U and V chromosomes. Finally, we uncover the unique conformation of a large genomic region on chromosome 6 harboring an endogenous viral element (EVE), providing insights regarding the functional significance of the chromatin organisation of latent giant dsDNA virus.