Compute $int xy dx +(x+y)dy$ over the curve $Γ$, $Γ$ is the arc $AB$ in the 1st quadrant of the unit circle $x^2+y^2=1$ from $A(1,0)$ to $B(0,1)$.

Your book is correct. First of all, your application of Green's theorem here is incorrect, as indicated in the comment. You can't apply Green's theorem for a curve that's not simple closed. See https://en.wikipedia.org/wiki/Green%27s_theorem. So the correct answer is through the line integral evaluation as follows

$$int_Gamma xydx+(x+y)dy=int_{AB} xydx+(x+y)dy$$ $$=int_{y=0}^1bigg[ysqrt{1-y^2}dbigg(sqrt{1-y^2}bigg)+bigg(sqrt{1-y^2}+ybigg)dybigg]$$ $$=-int_0^1y^2dy+int_0^1sqrt{1-y^2}dy+int_0^1ydy=-frac{1}{3}+frac{pi}{4}+frac{1}{2}=frac{pi}{4}+frac{1}{6}$$