I am not sure, if you write correct 'mygroup'
in the case of group: user_%companyName%
thus in specific case you want to find f.e. user_AAACompany
don't forget that in mysql underscore character "_" is considered as wildcard - i.e. you should write
LIKE 'user\_AAACompany'
and result query should look like:
SELECT id_customer FROM ps_customer_group WHERE id_group IN (SELECT id_group FROM ps_group_lang WHERE name LIKE 'user\_AAACompany' AND id_lang=5)