Finally beat the stupid thing.
I did it like this:
Turn on the minimap (i use compass normally)
Decrease difficulty to easy (Just for this fight, i play hard otherwise)
Then i would wait in the middle with my ship and see from which direction the monster comes, so i could position my ship.
Then try to pepper it with full broad sides when it gets closer. I never use that stupid board canon.
Anyways, with a bit practise i managed to beat it fairly easy like this, after being beat up 3-4 times before when i tried to "hunt" it down.
I hope that helps anyone
i used CoSMOS. ship hp max is around 200 and it's stored as 4 byte int.