Al-Si castings are relevant alloys particularly in the automotive sector where their high strength-to-weight ratio makes them attractive. Significant improvements of strength, impact, and thermal shock properties can be achieved by microstructure modification through addition of small quantities of certain elements. Two types of modification of the eutectic structure can be induced: (1) refinement accompanied by a plate-to-fibrous transition resulting in a coralline silicon structure or (2) just refinement of the plate-like silicon eutectic phase. Examples of remaining open questions regarding these modifications are: why do different elements cause different modifications? Which atomistic mechanisms are governing the modifications? Over the years, it has been evidenced that the additional elements affect the nucleation and the growth during solidification, although the active mechanisms responsible for the plate-to-fibrous transition remains under debate. In this work, we use atom probe tomography (APT) to analyze and compare four elements modifying the structure: sodium, strontium, europium and ytterbium. EBSD, SEM and TEM are also used as complementary methods for the comparison. The alloys modified by sodium, strontium and europium show a comparable eutectic silicon structure with a high density of precipitates and lattice defects enriched by aluminum and the modifying agent. We found that the composition of these enrichments correspond to ternary phases of these systems and therefore, we propose that clusters of these ternary phases form at the Si/liquid interface and take part in the modification of the eutectic Si growth to form a coralline-like microstructure. In contrast, the modification with ytterbium only results in a refinement of the plate-like structure and no ytterbium segregation could be detected. Based on our results, we propose that ytterbium modifies the sample only at the nucleation level and therefore has a finer structure. The other three elements affect not only the nucleation but also the growth inducing the coralline-like structure. We conclude that, although both effects are relevant in the modification of Al-Si alloys, the changes caused by the additional elements at the growth front are responsible for the advantageous coralline structure.