You would first have to medically prove that your mother actually has dementia. You would then have to prove that her dementia is severe enough that it is a disability in the sense that she cannot function mentally the same way that a normal person would.
If you are able to prove that your mother is impaired by a mental and/or physical deficiency,illness,or disability to the extent that she permanently or temporarily lacks sufficient understanding to engage in responsible actions or make rational decisions, then the insurance company would be acting in good faith to invalidate the policy. The insurance policy may even have a clause or some type of verbiage within it's operating procedures or policies that would enable them to invalidate the policy under the circumstances that your mother is in fact disabled by her dementia to the point that she cannot make rational decisions or engage in responsible actions.
All of this would be valid if at the time that she was forced by the family member to sign the policy, she was disable at the time. It would not be valid if she signed the policy and then developed the condition of dementia disabling her mental ability later on.
The family member who forced her to sign the document at the time she was incapacitated would also face some repercussions from the act. The insurance policy must contain in it's verbiage a section that defines this type of action and then asks for the signature to make sure that the person is aware and if they were not aware because they were incapacitated, then the signature would act as proof that she was forced to sign the document unlawfully.