Cut lychees in half. Soak goji berries in hot water in a bowl until softened, at least 5 minutes; drain.
Bring 3 cups water to a boil in a saucepan; add agar-agar. Cook, stirring occasionally, until agar-agar is dissolved, about 5 minutes.
Stir sugar into agar-agar mixture and cook until dissolved, about 5 minutes. Remove saucepan from heat and stir reserved lychee juice into agar mixture. Pour mixture into a ring mold and add lychees and goji berries. Refrigerate until firm, at least 3 hours. Cut into wedges.