Cardiorespiratory fitness was found to influence age-related changes of resting state brain network organization. However, the influence on dedifferentiated involvement of wider and more unspecialized brain regions during task completion is barely understood. We analyzed EEG data recorded during rest and different tasks (sensory, motor, cognitive) with dynamic mode decomposition, which accounts for topological characteristics as well as temporal dynamics of brain networks. As a main feature the dominant spatio-temporal EEG pattern was extracted in multiple frequency bands per participant. To deduce a pattern’s stability, we calculated its proportion of total variance among all activation patterns over time for each task. By comparing fit (N = 15) and less fit older adults (N = 16) characterized by their performance on a 6-min walking test, we found signs of a lower task specificity of the obtained network features for the less fit compared to the fit group. This was indicated by fewer significant differences between tasks in the theta and high beta frequency band in the less fit group. Repeated measures ANOVA revealed that a significantly lower proportion of total variance can be explained by the main pattern in high beta frequency range for the less fit compared to the fit group [F(1,29) = 12.572, <i>p</i> = .001, partial $η&$lt;sup>2</sup> = .300]. Our results indicate that the dedifferentiation in task-related brain activation is lower in fit compared to less fit older adults. Thus, our study supports the idea that cardiorespiratory fitness influences task-related brain network organization in different task domains.<h4>Supplementary information</h4>The online version of this article (10.1007/s11571-020-09656-9) contains supplementary material, which is available to authorized users.