Start worrying end of January but even then a lot can change. Genuinely worry the day pitchers and catchers report because by then the majority of your team should already be in place. Moves during spring training generally are fringe additions and not major reshaping moves.
Overall it's ok that they haven't made any big moves yet because it can still be done later but there is the facts of time running out, players coming off the board, etc.
Maybe better said, it's not the lack of moves that's the problem, it's the fear that the lack of moves means that no big moves are coming. Because if we could somehow say "Marlins haven't made any big moves yet but it's guaranteed that a big move is coming on January 8th" then you'd be ok because you know it's coming. So it's moreso about fans being nervous that nothing will get done at all.