Problem
The radii of the three black circles are 1, 2, and 3 (drawn roughly to scale). Circles are tangent where they look like it. The point of the problem was to find the radius of the red circle with that information.
The solution is straightforward once all the line segments are drawn in.
Diagram
Solution
(r+2)2 = 32 + (r+1)2 - 2(3)(r+1)(cos B)
r2 + 4r + 4 = 9 + r2 + 2r + 1 - 6(r+1)(cos B)
2r - 6 = -6(r+1)(cos B)
cos B = (6 - 2r) / (6r + 6)
cos B = (3 - r) / (3r + 3)
(3-r)2 = 22 + (r+1)2 - 2(2)(r+1)(cos B)
r2 - 6r + 9 = 4 + r2 + 2r + 1 - 4(r+1)(cos B)
-8r + 4 = -4(r+1)(cos B)
2r - 1 = (r+1)(cos B)
cos B = (2r - 1) / (r+1)
Set the two expressions of "cos B" equal...
(3-r) / (3r + 3) = (2r - 1) / (r+1)
(3-r)(r+1) = (3r + 3)(2r - 1)
3 + 2r - r2 = 6r2 + 3r - 3
0 = 7r2 + r - 6
(7r - 6)(r+1) = 0
r = -1 or 6/7
