Halve two of the peaches and remove the pits. Slice the peaches into wedges. Place the peach wedges in a single layer on a parchment paper-lined baking sheet and freeze until solid, at least 4 hours.
Place the tea bags in the boiling water, cover, and steep for 10 minutes. Remove the tea bags and discard them.
Stir the sugar into the hot tea until dissolved. Allow the mixture to cool slightly.
Meanwhile, peel the remaining two peaches and remove the pits. Roughly chop the peaches into pieces and place them into a blender or food processor. Add in the lemon juice, cover, and blend until smooth.
Strain the peach mixture through a fine sieve into a bowl, pressing gently against the pulp to extract the juice. Discard the pulp or save for another use.
In a pitcher, combine the strained peach juice, tea, cold water, and baking soda (if using). Mix well.
Refrigerate the tea mixture until cold.
Serve in individual glasses over ice with frozen peach wedges, if desired.