As we know that given in question,
ax + by = 6 ........................ (i)
bx – ay = 2 ........................ (ii)
On squaring the both equation and adding,
a2 x2 + b2 y2 + 2abxy + b2 x2 + a2 y2 – 2abxy = 36 + 4
⇒ x2 (a2 + b2) + y2 (a2 + b2) = 40
⇒ (a2 + b2) (x2 + y2) = 40
As per given question, x2 + y2 = 40 , Put this value in above equation, we will get
⇒ (a2 + b2) × 4 = 40
⇒ a2 + b2 = 10